Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

Make Entourage work well with Time Machine - revisited Apps
One problem for people like me who are still using Microsoft Entourage as their email client under OS X 10.5/10.6 has been its huge monolithic database file. Time Machine will try to back up the database file every time it is modified, even though only a few megabytes out of several gigabytes of the file have been changed. This can quickly fill up your backup storage, and if Time Machine is working over a network, each backup can take a long time.

This hint, which is essentially a non-geek version of this previous hint, will enable Time Machine to incrementally back up the Entourage database file. No command line operations or Unix scripting needed here; all will be done in the GUI.

  1. Create a sparse bundle disk image. In the Finder, check the current size of your Entourage database file, which reside in your user's Documents/Microsoft User Data/Office 2008 Identities/Main Identity folder. Open Disk Utility, click New Image in the toolbar, enter the following parameters, and click Create:
    • Save As: Office_2008_Identities (exact name is not critical)
    • Where: Your Documents folder (location is not critical, either)
    • Name: Office_2008_Identities (again, name is not critical)
    • Size: Two to three times the current size of your Database
    • Format: Mac OS Extended (Journaled)
    • Encryption: None
    • Partitions: Single partition - Apple Partition Map
    • Image Format: Sparse bundle disk image
    Verify that the disk image file has been created in the Documents folder, and a new Office_2008_Identities disk image has been mounted on your Desktop.

  2. Move your current data. Quit all Microsoft Office applications; don't open them until everything is finished here. In the Finder, drag copy the entire Office 2008 Identities folder (~/Documents/Microsoft User Data/Office 2008 Identities) onto the new disk image mounted on your Desktop. Copy the folder itself, not just its contents.

    Once the Finder is finished copying the folder, move the original Office 2008 Identities folder out of the enclosing Microsoft User Data folder and onto your Desktop (or another secure location). Keep this original folder as a backup until you verify that everything works.

  3. Create an alias to the database folder. In the Finder, Command-Option-drag the Office 2008 Identities folder from inside the Office_2008_Identities disk image to your Microsoft User Data folder (~/Documents/Microsoft User Data). This will create an alias to the former folder inside the latter. Once the alias is created, eject the Office_2008_Identities disk image.
That's it; now you can fire up Entourage. You will see the Office_2008_Identities disk image automatically re-mounted on the Desktop upon Entourage launch, and the application should work as if nothing has been changed. But now the huge database file will be backed up incrementally by Time Machine.
    •    
  • Currently 3.50 / 5
  You rated: 2 / 5 (20 votes cast)
 
[17,814 views]  

Make Entourage work well with Time Machine - revisited | 24 comments | Create New Account
Click here to return to the 'Make Entourage work well with Time Machine - revisited' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Make Entourage work well with Time Machine - revisited
Authored by: petieg on Oct 29, '09 08:31:26AM

Now is we can only get microsoft to implement this option w/in Entourage automatically that would be great! Thanks for the tip!



[ Reply to This | # ]
Make Entourage work well with Time Machine - revisited
Authored by: cody121 on Oct 29, '09 08:59:27AM

That really is much simpler, Nice!



[ Reply to This | # ]
Make Entourage work well with Time Machine - revisited
Authored by: clusty on Oct 29, '09 12:11:11PM

would this work also with virtualBox image files?
What is the performance penalty of an extra layer in the IO stack?



[ Reply to This | # ]
Make Entourage work well with Time Machine - revisited
Authored by: mike666 on Nov 02, '09 11:52:32PM

I haven't tried it or benchmarked sparsebundles but it should work fine. The only real inconvenience would be if you had a virtual disk that was already split into 2MB files (in which case using a sparsebundle would be pointless) - the more files you have on a large disk image, the longer it takes to mount. I'm guessing performance shouldn't be too much of an issue though since, once mounted, the OS should be using block IO to access it as far as I'm aware.



[ Reply to This | # ]
Is it safe for Time Machine to backup Entourage while it is open?
Authored by: jhersco on Oct 29, '09 04:31:22PM

I read that it is best to quit Entourage when backing it up to avoid corruption of the database file. (See http://blog.entourage.mvps.org/2009/04/quit_all_microsoft_applications_before_backing_up.html#more). That doesn't appear possible or practical given that Time Machine backups take place every hour. Does anyone know how critical it is to quit Entourage before backing up?



[ Reply to This | # ]
Is it safe for Time Machine to backup Entourage while it is open?
Authored by: cody121 on Nov 06, '09 11:07:33AM

"safe" is not how i would describe it, but I would say the risk of total failure is very remote.

On my system, Entourage is hard shutoff (crashes, power outages, force quits) probably a dozen times a year or more. Since I have had office 2004, I have only had entourage complain once about problems, which were fixed when i rebuilt the database (by holding the option key down). All and all I would call it's databasing system fairly robust.

in practice, I rarely ever have a NEED to recover from a backup, because I dont have system failure often. In the event that I do need to recover, odds are even if the backup occurred during a database update I will be fine. If I am not fine, I will rebuild the database, and if that doesn't work, there are hundreds of other points in history I can try.

If I were planning to migrate to a new hard drive using time machine, I would absolutely shut down entourage first, then do a backup, then do the migration. The rest of the time, I feel that the risk of catastrophic failure doesn't even exist next to the possibility that I can have incremental backups of my 4GB mail archive.



[ Reply to This | # ]
Make Entourage work well with Time Machine - revisited
Authored by: pglock on Oct 30, '09 02:20:30AM
I run a numer of mobile accounts on osx server. Would this hint help with syncing entourage database files at login/logout?
---
Too lazy to think of an original signoff...


[ Reply to This | # ]
Make Entourage work well with Time Machine - revisited
Authored by: zsoltmag on Oct 30, '09 04:05:03AM

And will the sparse image bundle grow automatically as the database grows?
I'm asking this because I tried the same trick with VMW Fusion virtual machines, and there it does not grow.



[ Reply to This | # ]
Make Entourage work well with Time Machine - revisited
Authored by: mike666 on Nov 02, '09 11:28:11PM
It should grow whether you use a sparse image or a sparse bundle unless you set the size too small - both types will stop growing when they hit their size limit although you should be able to increase that as needed with the hdiutil resize command.

[ Reply to This | # ]
How should Time Machine be setup?
Authored by: cmcd on Nov 06, '09 01:25:14PM

First, thanks for this - definitely an improvement.

I followed the instructions and have the new Entourage sparsebundle in ~/Documents. My Time Machine backups are picking it up. In Time Machine preferences, the mounted sparsebundle is excluded from Time Machine backups. Is this correct or should the mounted sparsebundle be included.

And, how does a restore work? If I need to recover a previous version, do I just restore the entire sparsebundle file, replacing the current one in ~/Documents? Or can I somehow restore certain files within the sparsebundle file?

Regarding the risk of corruption, I've decided to run Time Machine manually, maybe twice a day, preceded by shutting down all Microsoft apps as recommended here http://blog.entourage.mvps.org/2009/02/backup_entourage.html#more



[ Reply to This | # ]
How should Time Machine be setup?
Authored by: cmcd on Nov 10, '09 02:13:49PM

I've followed the instructions and have Entourage Identities within a sparsebundle image file - but, my Time Machine backups show a full backup of the Identities file, not just the incremental changes.

After manually running Time Machine a few times per day - instead of hourly - I mounted the Time Machine backup file from the Time Capsule (where we backup to) and explored the backups via Finder (each Time Machine backup shows its own subfolder within the Time Machine sparsebundle with a date-time stamp in the folder name). As I drilldown into each backup and Show Package Contents for each of the Entourage Identities sparsebundle files within, each shows 830 band files each of 8 mb (totaling about 6gb), which is the size of my Identities file. This is repeated in each of the Time Machine backups at different times that I've checked.

I was expecting to see just a number of modified bands in each backup - not all the bands! What's the deal?



[ Reply to This | # ]
How should Time Machine be setup?
Authored by: cmcd on Nov 10, '09 02:26:03PM

Note: of the 830 8mb band files included in each backup, only a relatively small number show a recent modified date (i.e., same day as the backup) - many haven't been modified in several days. But, all 830 files are included in each of the backups, requiring lots of disk space.



[ Reply to This | # ]
How should Time Machine be setup?
Authored by: everkleer80 on Nov 11, '09 06:32:04AM

The way TM works is it creates hard links to any files that haven't changed since they were last backed up. If you're not familiar with hard links, they are basically file entries pointing to data on your hard drive. Technically I think every file on your drive is a hard link and once another hard link to a preexisting file has been created, they are identical and there is no way to tell them apart. So if, for example, you have an 8MB file on your drive and you create another hard link to that file, the new file will both A. look exactly like the original, B. contain the exact same data (if the data changes then both files will be changed), C. report a size of 8MB, but the data will only be on your drive once so no matter how many hard links there are to it, the total amount of disk space used by all of them combined will be 8MB. I don't know if this is a good explanation (if you need more info you can look it up - Wikipedia comes to mind) but basically the files you are seeing that do not have a 'recent modified date' were not acually backed up - they are just hard links that were created for the previously backed up files and they will always report the same size as their counterparts but are not actually taking up extra disk space because they are pointing to the same data.

I hope this makes sense!



[ Reply to This | # ]
How should Time Machine be setup?
Authored by: cmcd on Nov 12, '09 07:03:26AM

Thanks, everkleer80, I think I do understand. Based on further analysis, it appears like Time Machine (TM) isn't backing up as much as I originally thought, but I'm still not sure it's behaving as I would hope. I used Grand Perspective (great tool!) to analyze the TM sparsebundle backup file on the Time Capsule, which allowed me to filter out hard linked files. It showed that TM was not actually backing up the full 6gb Identities folders every time, but it was backing up about 500mb every time. The Database file is actually about 1.5gb and the 500mb corresponds exactly to the size of my Exchange server mailbox size, which would be one of the mailboxes included with the Database file.

Is it possible that all the sparsebundle bands associated with the Exchange server mailbox within the Database file change when a single mail message is sent or received? That's the only way I can explain why that much data would be backed up each time. I will continue to monitor the backup performance to try to figure this out. Thanks for your help.



[ Reply to This | # ]
How should Time Machine be setup?
Authored by: everkleer80 on Nov 12, '09 10:16:07AM

I've actually never used Entourage or anything so I'm not really sure what's going on here. You're right it sounds like every band in the sparsebundle associated with the mailbox is changing. If I had to guess, I'd say that maybe it's moving data around in the file to limit fragmentation (similar to defragmenting a hard drive.) That sounds pretty unlikely, but it would result in data moving to different bands in the sparsebundle and therefore all the associated bands would end up changed...

Anyway, GrandPerspective is a great program! I also found it when troubleshooting a different TM issue (my issue was space not being free'd up in my TM disk image when I deleted files from the backup - turned out the 'deleted' data was going into a hidden temp file on the disk.) I should have mentioned before, though, that TimeTracker is a cool little program that will show you how much data and which files were backed up in each backup. There are some better similar programs out there but I don't think the others are quite as free as TimeTracker.



[ Reply to This | # ]
How should Time Machine be setup?
Authored by: everkleer80 on Nov 12, '09 10:39:25AM

Just a clarification, (I thought we were supposed to be able to edit our posts now?) when I said:

If I had to guess, I'd say that maybe it's moving data around in the file to limit fragmentation (similar to defragmenting a hard drive.)
I meant maybe Entourage is moving data around in the mailbox file which would result in all associated bands being changed.

For example, if I have a simple text file ("abcdefghijklmnop") split into bands or 5 characters each (band1="abcde", band2="fghij", etc) and I insert the letter A at the beginning of the file, then every band would be change (now band1="Aabcd", band2="efghi", etc.) Whereas if I just added the letter to the end of the file then only the last band would be changed.



[ Reply to This | # ]
How should Time Machine be setup?
Authored by: cmcd on Nov 13, '09 06:45:35AM

I've done additional analysis, with both TimeTracker and Finder/Path Finder. It does appear that Time Machine is just backing up the bands that have been modified since the last TM backup - as expected. The surprise for me is how much space the bands are taking up - again, I'm seeing about 400-500mb per daily backup. I suspect the issue is the 8mb band size. For example, for the few backups I examined, the number of bands was 50-70 - multiply by 8mb per band and I get my number. And 50-70 modified bands may correspond to about the number of emails I receive/send in a day (though it seems a bit high).

So, maybe the answer is to convert the sparsebundle file to use either smaller - or much larger - bands. Since email messages are usually small, maybe use 1mb bands instead of 8mb. Then, 70 bands will only be 70mb (except when I receive emails with large attachments). I think hdiutil can convert sparsebundle band size. Any thoughts? My problem is I'm dealing with 2 sparsebundles: the relatively small Microsoft Identities file, and the huge Time Machine backup file on the Time Capsule.



[ Reply to This | # ]
How should Time Machine be setup?
Authored by: everkleer80 on Nov 13, '09 09:52:18AM

I think you're right on with changing the band size. But it sounds like the messages may be spread throughout the file rather than being grouped together (not sure about this) which means that multiple messages may or may not go into one band depending on how close together they are in the mailbox file. So if you increase the band size to, say, 16 MB, and have 10 new messages that are all < 1MB and are grouped together, then they may all go into the same band and you will only have to backup 16MB, but then again if the messages are spread apart in the file and each one goes into a different band, then you will be backing up 160MB. So I think smaller is definately the way to go here so no matter where in the data your messages are, you will be backing up closer to the amount of data that actually changed (maybe make the bands 1 or 2MB if you can?)

I'm not sure if you're saying there is any sort of problem with your TM sparsebundle or not or if your just commenting on the fact that we're dealing with two different sparsebundles, but anyway I don't think you should do anything to the TM bundle. I'm not sure why exacly TM backs up directly to the drive if it's local, and to a sparsebundle if it's a network drive, but either way I'd leave that one alone and just let TM manage it.



[ Reply to This | # ]
How should Time Machine be setup?
Authored by: cmcd on Nov 16, '09 01:54:14PM
OK, now I'm using a sparsebundle file with 1MB bands instead of the default 8MB. It's soon after the change and I'm still monitoring, but the conversion appears to have the desired result - my backup is now in the 100MB range instead of 500MB range. Maybe 500KB bands would reduce the backup size further, I don't know. I used this command to convert the sparsebundle file:
hdiutil convert ~/Documents/Office_2008_backup.sparsebundle -format UDSB -tgtimagekey sparse-band-size=2048 -o ~/Documents/Office_2008_IdentitiesNEW.sparsebundle
where Office_2008_IdentitiesNEW.sparsebundle is the converted file with the 1MB bands. After the conversion, I created a new alias pointing to the converted file.
Edited on Nov 16, '09 01:55:08PM by cmcd


[ Reply to This | # ]
How should Time Machine be setup?
Authored by: cmcd on Nov 17, '09 01:26:15PM

According to the man page for hdiutil, 1MB bands are the smallest allowed.



[ Reply to This | # ]
How should Time Machine be setup?
Authored by: kcheon on Apr 13, '11 05:05:08AM

I just tried, you don't need to exclude the sparse bundle file in TM. It seems that TM is clever enough to look at the contents and only backup the changed files in the bundle.



[ Reply to This | # ]
How should Time Machine be setup?
Authored by: everkleer80 on Apr 14, '11 02:33:25PM
Actually it's not so much TM's cleverness as it is the cleverness of sparsebundles. Rather than being one large file like a sparseimage, a sparsebundle is actually a collection of relatively small files (one for each band of data) inside a folder that the OS displays as a file (aka a package.) When TM runs, all it sees is the sparsebundle folder and all the band files inside it; since it is only backing up files that have been modified, only the bands that have changed are backed up.

[ Reply to This | # ]
Make Entourage work well with Time Machine - revisited
Authored by: jmeggers on May 18, '10 12:38:36PM

I just found this hint, but when I follow the instructions, it appears the original database is still being used, at least according to the time/date stamp on the file. The really strange part, is it appears the other components in the Identities folder are being updated on the new image, again based on time/date stamp. I'm reluctant to delete the original Identities folder. Is there a way to view what Entourage thinks it should be using? Thanks for any info.

John Meggers
Ashburn, VA



[ Reply to This | # ]
Make Entourage work well with Time Machine - revisited
Authored by: sonorelli on Dec 08, '12 08:29:32AM

Unfortunately, this trick does not work with Mountain Lion 10.8.2. I tried changing the alias to a symbolic link, and when Entourage was launched, it would not automatically open the sparse bundle. If I manually opened the sparse bundle, the symbolic link worked, but Entourage still couldn't find the database, and offered to start a new one.



[ Reply to This | # ]