Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

10.5: Show files that Time Machine does not back up System 10.5
Leopard's Time Machine is a huge step forward in backup for the average Joe (or Jane). However, one problem is that applications can tell the system to not back up certain files. This is intended to avoid backing up working files, cache files, and such that don't generally need to be backed up. However, we know that VMware, at least, is using the mechanism to quiet exclude virtual machines from being backed up, and others could come. While this makes sense in some ways, it could easily lead to a disaster for those who aren't aware of it happening.

So, with that background, I played around for a few minutes and I figured out how this flag is stored. Turns out that Apple's doing the right thing here and using the metadata store. This means we can find the files with the Spotlight engine. To do this from the command line, do this:
sudo mdfind "com_apple_backup_excludeItem = 'com.apple.backupd'"
And you'll get a nice list back. If you're not comfortable with the command line, here's a simple line of AppleScript that you can build into a double-clickable application with Script Editor:

set myItems to do shell script "mdfind "com_apple_backup_excludeItem = 'com.apple.backupd'" > ~/Desktop/Files excluded by Time Machine.txt" with administrator privileges

When you run that script, you'll be asked for an admin login (so that directories other than your home folder can be searched), and the results will be deposited into a file called Files excluded by Time Machine.txt on your desktop.

[robg adds: Note that the output of this command will not reflect any files, folders, or volumes that you've excluded from Time Machine using its preferences panel. As best as I can tell from looking at the mdls output, Time Machine uses an internal mechanism, not metadata, to track those items.]
    •    
  • Currently 2.50 / 5
  You rated: 5 / 5 (10 votes cast)
 
[51,345 views]  

10.5: Show files that Time Machine does not back up | 32 comments | Create New Account
Click here to return to the '10.5: Show files that Time Machine does not back up' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
10.5: Show files that Time Machine does not backup
Authored by: ocdinsomniac on Apr 04, '08 07:55:34AM

Time Machine also does not back up my ~/Library/Caches folder, yet this folder is not returned by the command in this hint. I'm skeptical that the results returned by this command are complete.



[ Reply to This | # ]
10.5: Show files that Time Machine does not backup
Authored by: Eric3 on Apr 04, '08 10:27:19AM
Time Machine has its own internal list of files and directories that aren't worth backing up; everything is explained at http://shiftedbits.org/2007/10/31/time-machine-exclusions/.

---
Eric3

[ Reply to This | # ]

10.5: Show files that Time Machine does not backup
Authored by: chrischram on Apr 04, '08 10:00:50PM

That's a very interesting list. I already had several of those items in my personal exclude list, which I guess I can go and simplify now.



[ Reply to This | # ]
10.5: Show files that Time Machine does not backup
Authored by: chocky on Apr 04, '08 08:23:01AM

Here's a quick fix of the AppleScript version (the original post didn't work for me).

do shell script "mdfind \"com_apple_backup_excludeItem = 'com.apple.backupd'\" > ~/Desktop/Files_excluded_by_Time_Machine.txt" with administrator privileges



[ Reply to This | # ]
10.5: Show files that Time Machine does not backup
Authored by: siracusa on Apr 04, '08 08:34:55AM

You can exclude files from Time Machine in two ways: by path or by file. Excluding by file attaches a bit of metadata to the file using the extended attributes API, as described above. Excluding by path registers the path with Time Machine itself (somewhere in a plist that backupd reads, IIRC).

Files excluded via extended attributes remain excluded if you rename or move them. Excluding by path means that any file at that path will be excluded; if you rename or move the file it will no longer be excluded.

(This all applies equally to folders, in case that doesn't go without saying.)



[ Reply to This | # ]
10.5: Show files that Time Machine does not backup
Authored by: tjahns on Apr 04, '08 09:27:54AM

Also all paths listed under various keys in

/System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist

are excluded, perhaps the script could be extended to parse all paths there (and expand them recursively for all users), but I feel that would be a lot of work for little benefit.

Regards, Thomas Jahns



[ Reply to This | # ]
10.5: Show files that Time Machine does not backup
Authored by: Tricon on Apr 04, '08 09:43:45AM

Is "iTunes Music Library.xml" the only file that tracks my ratings? If so, I would hope that TM would back this up. I don't want to have to re-rate my music upon restore. Thoughts?



[ Reply to This | # ]
10.5: Show files that Time Machine does not backup
Authored by: Eric3 on Apr 04, '08 10:30:33AM

That XML file is merely a mirror of the file "iTunes Library" in the same folder, which does get backed up. You've nothing to worry about.

---
Eric3



[ Reply to This | # ]
10.5: Show files that Time Machine does not backup
Authored by: PatrickS on Apr 04, '08 10:45:20AM

Actually I'm more concerned about ~/Music/iTunes/Album Artwork/Local as I don't want to re-add all the manually added artwork if I ever lose the disk with my iTunes collection on it...



[ Reply to This | # ]
10.5: Show files that Time Machine does not backup
Authored by: lowbatteries on Apr 08, '08 07:34:30PM

I believe any albums you manually add get written to the tags in the MP3 File, and those that iTunes downloads automatically go into that folder.



[ Reply to This | # ]
10.5: Show files that Time Machine does not backup
Authored by: JackSloth on Apr 04, '08 10:06:11AM

Here is my twist on the script. I just added a dialog to inform the user.

display dialog "This application will create a text file with a list of things that are not backed up in Time Machine. It will not include those items you specified in Time Machine's options to exclude. Your administrative password is required in order to perform the search." buttons {"Cancel", "Continue"} default button 2
if the button returned of the result is "Cancel" then
-- action for 1st button goes here
quit application
else
-- action for 2nd button goes here

end if

set myItems to do shell script "mdfind \"com_apple_backup_excludeItem = 'com.apple.backupd'\" > ~/Desktop/Files_excluded_by_Time_Machine.txt" with administrator privileges



[ Reply to This | # ]
10.5: Show files that Time Machine does not backup
Authored by: ppp on Apr 04, '08 10:06:59AM
My iTunes Library.xml is also in that list. That's playlists, ratings and lots of info I'd rather not lose. Good to know that I need manually back that up.

Thanks to the other poster for the Standard Exclusions path. This file includes the /Library/Logs and ~/Library/Logs folders, and most (all?) other daemon logs, including Apache's error and access logs (/var/log/apache2).

I guess the thought is that if you're a basic enough user to be using Time Machine, you don't care about your logs? Certainly, I wouldn't be devastated if they were gone, but I wouldn't expect them to be. Perhaps keeping that information could be an option.

Note that "Previous Systems" is also excluded, so if you're lazy about reintegrating your Archive Install components (if any), then you lose those on a restore. I think this is a fine decision, but it's good to know.

For people wanting the whole list:


Paths Excluded
	/.Spotlight-V100
	/.Trashes
	/.fseventsd
	/.hotfiles.btree
	/Backups.backupdb
	/Desktop DB
	/Desktop DF
	/Network/Servers
	/Previous Systems
	/Users/Shared/SC Info
	/Users/Guest
	/dev
	/home
	/net
	/private/var/db/Spotlight
	/private/var/db/Spotlight-V100</string>	
Folders Copied, but Contents Excluded
	/Volumes
	/Network
	/automount
	/.vol
	/tmp
	/cores
	/private/tmp
	/private/Network
	/private/tftpboot
	/private/var/automount
	/private/var/log
	/private/var/folders
	/private/var/log/apache2
	/private/var/log/cups
	/private/var/log/fax
	/private/var/log/ppp
	/private/var/log/sa
	/private/var/log/samba
	/private/var/log/uucp
	/private/var/run
	/private/var/spool
	/private/var/tmp
	/private/var/vm
	/private/var/db/dhcpclient
	/private/var/db/fseventsd
	/Library/Caches
	/Library/Logs
	/System/Library/Caches
	/System/Library/Extensions/Caches
Local User Folders copied, but contents Excluded
	Library/Application Support/MobileSync
	Library/Application Support/SyncServices
	Library/Caches
	Library/Logs
	Library/Mail/Envelope Index
	Library/Mail/AvailableFeeds
	Library/Mirrors
	Library/PubSub/Database
	Library/PubSub/Downloads
	Library/PubSub/Feeds
	Library/Safari/Icons.db
	Library/Safari/HistoryIndex.sk


[ Reply to This | # ]
iTunes Library.xml is a copy not the original
Authored by: thebug on Apr 04, '08 02:25:42PM
My iTunes Library.xml is also in that list. That's playlists, ratings and lots of info I'd rather not lose. Good to know that I need manually back that up.
iTunes Library, which is backed up, is the source of playlists, rating and all that other information. Deleting (or moving) the xml file will cause it to be regenerated. It's there to make it easier for other apps to get at that information by parsing a xml file. Apple's saving a bit of room on your backup by avoiding a file that can be easily re-created off the original information.

http://docs.info.apple.com/article.html?artnum=93732

You can also use the xml file to rebuild your Library file if it becomes corrupted.

http://docs.info.apple.com/article.html?artnum=93313

---
Outside of a dog, a book is man's best friend. Inside of a dog, it's too dark to read.

[ Reply to This | # ]

10.5: Show files that Time Machine does not backup
Authored by: johnsawyercjs on Apr 04, '08 06:36:58PM

Personally, I'd prefer if "Previous Systems" was backed up too. It usually doesn't contain much that might be of use, except for non-Apple items in /Library that can usually be reinstalled, but it's nice to be able to look inside it to see what those items are, if one has forgotten. I'm not sure if Archive & Install moves one's personal cron scripts into a newly-installed system, but if it doesn't, they'd be in Previous Systems too.



[ Reply to This | # ]
10.5: Show files that Time Machine does not backup
Authored by: TvE on Apr 05, '08 02:01:09AM

Then you can just rename the folder…



[ Reply to This | # ]
10.5: Show files that Time Machine does not backup
Authored by: mal0rd on Apr 05, '08 02:15:58PM

This doesn't seem to work. It gives me a few files then I get an error:

bad path 41321D00-0000-0000-0000-006B00000000

Any ideas?



[ Reply to This | # ]
10.5: Show files that Time Machine does not backup
Authored by: DistantThunder on Apr 06, '08 05:59:21AM

I had no idea that VMware had excluded itself, but I had experienced the frustration of trying to restore a virtual machine and being perplexed why there were no backups available. Thanks to this hint, now I know why!



[ Reply to This | # ]
Time Machine Updates 1.0 Fixed the problem
Authored by: adamcr on Apr 06, '08 10:11:56AM
This problem has now been fixed with Apple's latest Time Machine and AirPort Updates v1.0. Time Machine will now compact the sparse bundle after it deletes an old backup to make sure it has the right amount of free space. I've posted my logs to show the new behaviour.

Cheers,

Adam

[ Reply to This | # ]
10.5: Show files that Time Machine does not backup
Authored by: gbrandt on Apr 07, '08 06:17:54AM

I notice /Volumes is in the list that is not backed up. I want to include one of my external drives in the backup, does anybody know how to do that?

Gregor



[ Reply to This | # ]
Re: 10.5: Show files that Time Machine does not backup
Authored by: Uncle Asad on Apr 07, '08 07:43:15PM

Because it really grates on me:

"Backup" is an adjective, as in "I have the backup copy in my hand," or sometimes a noun, as in "I stored the backup off-site for safety."

"Back up" is the verb, as in "How many files do you have to back up each hour?" or "Time Machine does not back up those files, but it backs up these files."

;)



[ Reply to This | # ]
Re: 10.5: Show files that Time Machine does not backup
Authored by: lowbatteries on Apr 08, '08 07:39:38PM

Well as long as we're being prescriptive, a semicolon followed by a closing parenthesis makes absolutely no sense. ;)



[ Reply to This | # ]
Re: 10.5: Show files that Time Machine does not backup
Authored by: robg on Apr 21, '08 03:27:57PM

That was a typo on my part; it's fixed now.

-rob.



[ Reply to This | # ]
Re: 10.5: Show files that Time Machine does not backup
Authored by: TXCiclista on Jul 21, '08 11:05:54PM

Since we're being technical, the noun is "back-up," not "backup." Consult OED for more info.



[ Reply to This | # ]
10.5: Show files that Time Machine does not backup
Authored by: lowbatteries on Apr 08, '08 07:33:01PM

I believe any albums you manually add get written to the tags in the MP3 File, and those that iTunes downloads automatically go into that folder.



[ Reply to This | # ]
10.5: Show files that Time Machine does not backup
Authored by: kdz666 on Apr 13, '08 01:20:09PM

Any hint on how to set this metadata on a file (using the command line)?



[ Reply to This | # ]
10.5: Show files that Time Machine does not back up
Authored by: flammable on Apr 27, '08 11:52:12PM

I realize that the iTunes XML file is a copy, but I'd prefer to keep both files in the event of a disaster.

How can I force Time Machine to back it up?



[ Reply to This | # ]
10.5: add external drives to TM backup
Authored by: remac on May 03, '08 05:43:16PM

Exactly what I want to do. I use external drives to store my digital photo and video content so that it doesn't clog up the mac hard drive. I also access these from different macs. I want to add these drives to the backup done by TM. In the TM options I was eventually able to remove them from the 'exclusion' list, but looking in TM they're not backed up? Can anyone give me a clue how to make this work?



[ Reply to This | # ]
10.5: Show files that Time Machine does not back up
Authored by: PopMcGee on Jun 12, '08 10:10:57AM
I realized that time machine also does not back up other files, which are not excluded in any of those lists and not manually excluded. Some of them are folder containing unfinished bittorrent files downloaded using Transmission.

This is also noted here: http://arstechnica.com/journals/apple.ars/2008/01/06/torrent-client-for-for-mac-os-x-transmission-hits-1-0

While I think that this is generally a good idea, I'd love to have a small utility that shows me very clearly what Time Machine backs up and what not.

[ Reply to This | # ]
10.5: Show files that Time Machine does not back up
Authored by: bnetz on Aug 09, '08 02:39:13AM
when i use the command
sudo mdfind "com_apple_backup_excludeItem = 'com.apple.backupd'"

i get this list:
/Users/xy/Music/iTunes/iTunes Music Library.xml
/Users/xy/Music/iTunes/Album Artwork/Local
/Users/xy/Library/iTunes/iPod Software Updates
/Users/xy/Pictures/iPhoto Library/iPod Photo Cache

(no more!)

how can i exclude the other default-folders? i am mostly interested in excluding the spotlight-database from backup, i feel that this generates a lot of traffic/hd-load ... thx in advance

[ Reply to This | # ]
10.5: Show files that Time Machine does not back up
Authored by: benfsmith on Sep 06, '08 04:37:53PM

Does anyone know how to tell which files have been backed up during a particular session? There are often times where TM does a large backup when I little should have changed on my machine. I have excluded the obvious culprits.



[ Reply to This | # ]
10.5: Show files that Time Machine does not back up
Authored by: Bahi on Sep 26, '09 05:10:57AM
Use GrandPerspective and point it to the latest backup on your Time Machine volume (it's easiest to just point to the item called "Latest", which works fine, though it's an alias). Then ask for a filtered search and choose to not show hard links. That way, you get all the stuff contained in the last backup. Point it at any given TM session and perform the same filtered search and you'll find out what was backed up in that particular session.

[ Reply to This | # ]
10.5: Show files that Time Machine does not back up
Authored by: ricede on Jul 17, '12 04:02:20AM

Theres also a small app by CharlesSoft called Time Tracker which is very useful for keeping track of what TM backs up. Just open it after a backup & let it load. It also keeps a list of past back ups in the same GUI.

I know it works in Snow Leopard - don't know about Lion though.

Edited on Jul 17, '12 04:04:33AM by ricede



[ Reply to This | # ]