Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

10.5: Improve networked Time Machine performance System 10.5
After using a 500GB Time Capsule for Time Machine backups for awhile, I noticed increasing duration of the backups. Mainly the prepare and post-processing phases, and transfer of large number of small files, took a lot of time. For the transfer of large files, the slowdown was not so noticeable.

I inspected the sparsebundle disk image on the Time Capsule, and inside the disk image bundle of about 150GB, I found nearly 20,000 band files of 8MB each in a single folder. These 20,000 data pieces contain the data of the backup image. When Time Machine reads, creates or deletes folder structures in the backup image, the operating system must randomly access data distributed over these 20,000 band files. And that causes a remarkable amount of additional administrative accesses to the AppleShare volume of the Time Capsule.

I decided that a better size of the bands of the backup sparsebundle disk image is 1GB, i. e. my 150GB backup would contain about 150 band files in one folder. I converted the Time Machine backup image with the following method, which will also work fine for an AirPort-connected hard drive, not just for a Time Capsule.

[robg adds: Note that the following solution is clearly not Apple-approved, and there may be very good reasons why Apple chose an 8MB band size. If you're going to try this hint, realize that you're potentially endangering your backups. With that said, I've noticed the same performance issues on the USB-connected hard drive on my AirPort Base Station, so I'm going to try this later today to see how it goes. (I use this disk for backups of our laptops, neither of which contain critical can't-lose information, so it's a relatively low-risk test.) Read on for the proposed solution...]

Before you begin, note that this hint requires that the Time Machine disk has enough free space to hold another copy of the backup disk image.
  • In the Time Machine System Preferences pane, turn off Time Machine.
  • To be on the safe side, make a backup of the Time Capsule to a USB disk attached to the Time Capsule using the AirPort utility. (This takes some hours.)
  • Attach the Time Capsule to your Mac via Ethernet because it is much faster than AirPort.
  • Mount the Time Capsule disk in the Finder.
  • Open a Terminal window and change to the root of the Time Capsule disk (named TimeCapsule in this example): cd /Volumes/TimeCapsule
  • In this directory, you'll find the backup sparsebundle disk image, which is a folder with the name and the MAC address of your Mac with the suffix .sparsebundle, e. g.: MyMac_001acb9cb23d.sparsebundle. In this folder, you find a folder named bands that contains the many 8MB band files. You can see them with this command: ls -l MyMac_001acb9cb23d.sparsebundle/bands. Note that it will take a while for Terminal to show the list of bands, as there are so many of them.
  • Convert this disk image with the hdiutil tool to a new one using a 1GB band size:
    hdiutil convert MyMac_001acb9cb23d.sparsebundle -format UDSB -tgtimagekey sparse-band-size=2097152 -o NEW_MyMac_001acb9cb23d.sparsebundle
    This will take quite a while -- hours -- to run. (In case you're curious, parse-band-size is set to 2097152 because 2 * 1024 * 1024 512-Byte-Blocks = 2097152 = 1GB band size.)
  • When the conversion finishes successfully, exchange the two images by renaming them (you can do this in the Finder, too):
    $ mv MyMac_001acb9cb23d.sparsebundle OLD_MyMac_001acb9cb23d.sparsebundle
    $ mv NEW_MyMac_001acb9cb23d.sparsebundle MyMac_001acb9cb23d.sparsebundle
  • Eject the Time Capsule disk and disconnect the Ethernet cable.
  • In the Time Machine System Preferences pane, turn Time Machine back on.
Now notice that the Time Machine backups are (hopefully) running a little bit faster than before. When you are happy with the new backup disk image, later delete the OLD_MyMac_001acb9cb23d.sparsebundle image.

CAUTION: If anything goes wrong, you may destroy all of your Time Machine backups! And there is no guarantee that the converted backup disk image works fine in the future, i. e. when the Time Capsule fills up, or when Apple changes the Time Machine software. Apply this hint on your own risk.
    •    
  • Currently 2.00 / 5
  You rated: 4 / 5 (8 votes cast)
 
[23,809 views]  

10.5: Improve networked Time Machine performance | 8 comments | Create New Account
Click here to return to the '10.5: Improve networked Time Machine performance' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
10.5: Improve networked Time Machine performance
Authored by: critcol on May 13, '08 08:30:58AM

I noticed Time Capsule over G slowed down too. I think the problem was the Adium 1.2.5 beta that I was running 'cause once Adium 1.2.5 final released, the problem went away.

I also did some packet sniffing and it seems that Time Machine spends a lot of time backing up email messages, so since I only run IMAP mailboxes and Time Machine doesn't backup the envelope index, I excluded them.



[ Reply to This | # ]
10.5: Improve networked Time Machine performance
Authored by: dbs on May 13, '08 10:34:17AM

This is a very cool hint. It is also unlikely to break future versions of TimeMachine as they work through the disk image toolkit which actively supports this option. (e.g., this should be transparent to the way TimeMachine uses disk images. Of course if that were completely true they'd work fine on other servers as well, so be careful.)

Another reason this may be of significant benefit is that various servers may not deal well with having thousands or tens-of-thousands of files in a single directory. I have a two-tier backup system where I run TimeMachine to a disk on a 10.5 mac, then have a cron (okay, launchd) job that rsyncs that sparse disk bundle to a mirrored NAS. (Google "sync_timemachine.sh".) The NAS runs samba over linux, and gets really really slow when there are so many files in a single directory. Switching this over to larger files would really help. (Of course the real solution would be for Apple to fix this and use a hierarchical structure so you'd never have more than 256 items in a given directory.)

One reason why Apple may use 8MB chunks by default is that if you have an encrypted home directory and you do it with a sparse disk bundle, then any change you make will only require time machine to backup the 8MB chunk that changed, and not more.



[ Reply to This | # ]
10.5: Improve networked Time Machine performance
Authored by: FSugino on May 13, '08 10:52:19PM

"Now notice that the Time Machine backups are (hopefully) running a little bit faster than before."

Your hint warns of procedures that take multiple hours to run, yet you never quantify what kind of performance improvement you got on your system. Did you save seconds per backup? A minute? Two?

Please give some performance figures so we can better judge whether all the steps are worth the time and effort.



[ Reply to This | # ]
10.5: Improve networked Time Machine performance
Authored by: ghay on May 14, '08 08:01:30AM

You should probably factor in some time to actually test that the backup system is still working after you have enabled.



[ Reply to This | # ]
10.5: Improve networked Time Machine performance
Authored by: rhoerbe on May 14, '08 03:00:55PM

To stay on the safe side and get most of the gain it should be sufficient to increase the size of each piece to 32 or 64 MB. Beyond that, diminishing returns should not yield substantial improvements. Theoretically.



[ Reply to This | # ]
10.5: Improve networked Time Machine performance
Authored by: felix-fi on May 14, '08 10:56:19PM

I may be wrong, but it sounds like a VERY BAD IDEA...

For example, hdiutil compact only reclaims bands which are completely "free"... (see the man page).

" For SPARSEBUNDLE images, completely
unused band files are simply removed."

I can easily imagine a situation where you end up with 500 1g bands which have files and TM thinking that the backup is full...

But I may be wrong... you tell us in a week or two.



[ Reply to This | # ]
10.5: Improve networked Time Machine performance
Authored by: res1233 on Nov 16, '13 06:49:51PM

Actually, this likely isn't the case. The bands likely work much like a sliced up sparse image, which can be compacted just like sparse bundles. The thing about removing bands is just a handy optimization due to the fact that removing files is far faster than deleting parts of files. If I had to guess (although I don't have any actual evidence...) the bands can probably be removed of free sectors just like the sparse images can.



[ Reply to This | # ]
10.5: Improve networked Time Machine performance
Authored by: hamarkus on May 17, '08 04:17:40AM

I also had TM slowdowns (preparing taking up to an hour). I deleted the Spotlight index on the TC volume and I ran DiskWarrior on the mounted TM image. Both steps helped (preparing now is just a few minutes).

However, DiskWarrior took hours, posting messages that things would take longer due to hardware issue. My computer froze several times while DiskWarrior was running. I think in the end I ran DiskWarrior from its CD.

So far, nobody could answer me the question how to check the TC disk for hardware problems.



[ Reply to This | # ]