Several networked and non-networked options for Time Machine backups

Apr 08, '11 07:30:00AM

Contributed by: theosib

This hint collects together what I have learned about solutions for Time Machine backups. There are a number of viable solutions, and I cover several of them.

External USB or Firewire drive

The most basic solution is a directly-attached external USB drive enclosure. This might be okay for desktop machines, but what bothers me about this is that both the Mac and the drive are on the same power circuit. If lighting strikes, then both can get hit at the same time. In a Mac Pro, you can use another internal drive for TM. And there are also external RAID solutions like I mention below. Unless you are particularly disciplined, this is a lousy solution for portable Macs. When I was going this route, I was lucky if I remembered to plug in the backup drive once a week. Maybe other people can do better than that, but it's still inconvenient.

Thus, I prefer networked solutions. However, at the end of this hint, I go into a discussion on why TM is a great design for local backups, but a major source of unreliability for networked backups.

Time Capsule

If you want to do Time Machine backups over a network, then Apple's primary supported configuration is Time Capsule. I'm not happy about that, because Time Capsules have a history of being unreliable. Back in 2009, there was a wave of failures. A web site was set up to track the failures. This site is either abandoned, or there haven't been any failures since early 2010. Maybe TC has gotten better, but the fact of the matter is, it's a single-disk system with inadequate ventilation and cooling. People still report them getting very hot. Mechanical drives are already a ticking time bomb -- letting them get hot accelerates that process significantly. The heat also plays havoc with the other electronics, and many failures have left the drive intact but inaccessible. You could get a replacement unit from Apple, but you could not salvage your data, because removing the drive would void your warranty.

Apple server

Once upon a time, Apple sold a blade server called Xserve. That may have been part of a reasonable TM solution for a corporate environment. But with the demise of any sort of dedicated Apple server platform, there is no longer a good centralizable solution for TM backups for a large number of Macs. In a corporate environment, therefore, you must decide if you want to provide every desktop with its own external backup drive and badger the notebook users to plug it in periodically. Or set up a Mac Pro with some huge drives crammed in it and hope it works reliably as a server. For a home user, if you have a desktop Mac and a notebook, you can connect an external drive to the desktop and backup the notebook wirelessly. You just have to make sure that the desktop is awake whenever you want to backup the notebook, which is only slightly more convenient than just plugging the notebook into its own backup drive.

The most convenient solution is some kind of Network Attached Storage (NAS). Unfortunately, Apple's support for that is limited to their own sanctioned solutions.

Airport Extreme Base Station with external USB drive

Another option is a USB drive attached to an Airport Extreme Base Station (AEBS). This isn't officially supported by Apple, although the firmware supports all the necessary Apple Filing Protocol (AFP) features for Time Machine to work reliably. There, the weak link is the drive itself. There are a lot of external USB drive options, like WD's MyBook, but most of those solutions aren't intended for extended use. If you back up three Macs to one drive, then that drive is going to spin up three times an hour, and spin-ups cause wear and tear to desktop drives. On the other hand, if the drive spins continuously, unless the external enclosure has a fan in it, then it's going to heat up, with the usual consequences.

I would recommend a RAID solution like the LaCie 2big Quadra. (There's a switch on the back to select a RAID 1 configuration.) This actually costs more than a TC of the same capacity, but that's because there are two drives, providing redundancy. If a drive fails, open up the unit and replace it. If the logic board fails, pop the drives in a replacement enclosure. You can connect a drive to the AEBS via USB, and it appears as a Time Machine supporting volume with no hassle. If you're willing to spend the money, I think this may be the best solution overall.

Linux or Windows Server

With Linux, you have a few options for serving files, including NFS, SMB, and AFP. For Windows, you have built-in SMB support, and there are NFS solutions as well.

The most obvious solution for Linux would be to use an AFP server, like Netatalk. AFP is more efficient than SMB and therefore faster over high speed networks. Unfortunately, Netatalk has some serious drawbacks:

I've experimented with trying to use NFS for TM backups, but I couldn't get it to work. For one thing, even if you advertize the share using Zeroconf/Bonjour, MacOS won't see it. It worked with Tiger, but it's been broken ever since, so I think this is intentional.

If you manually mount the NFS share, TM will see it (if you have unsupported volumes enabled) at the time you have it mounted and let you select it for backup, but it is unable automatically mount it later when it's time to do a regularly scheduled backup. (Just 'Making backup disk available...' indefinitely.) If you're on a desktop Mac, you can use automount for NFS, but this isn't a solution for portables. MacOS's support for SMB is buggy and slow, but it's easy to setup and seems to be the only viable alternative to AFP. Therefore, the rest of this hint covers using a SMB server (Linux or Windows) to store Time Machine backups.

In my experience, if you want to use a non-Apple server for TM, and you want the share to mount and unmount on demand, then SMB is your only viable option. If you google this, you'll find warnings (mostly from around 2008) to never use SMB for TM shares. Back then, there was a bug in TM involving mismatches between the free space in the sparsebundle and the free space on the network volume. If your network volume filled, but your sparsebundle had free space, TM would get confused when trying to delete old backups to free space, and it would destroy the whole sparsebundle. This bug has long since been fixed. Now, the sparsebundle is resized automatically with the free space on the network volume, and it even works for SMB.

Enabling support for TM backups to unsupported network volumes

To begin with, SMB is an 'unsupported' network volume type, so you have to tell MacOS to let you see it for Time Machine purposes. Open a terminal, and type this command:

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

Creating the TM sparsebundle

When backing up over a network, Time Machine doesn't store files on the network volume directly. Instead, it encapsulates an HFS+ formatted file system volume in a shallow directory structure called a sparsebundle. Time Machine relies on some tricks, like hard-linked directories, that are unique to locally-mounted HFS+ volumes and not supported by any other system. Therefore, Apple opted to store an HFS+ volume in a container and make that container look like just a bunch of regular files, as far as the network drive is concerned. Unfortunately, you cannot just point TM at the SMB share and tell it to store the backup there. It will attempt to create the sparsebundle, but it will fail with an error due to missing features. Once you have a sparsebundle setup for TM, it will work fine over SMB, but you have to create it somewhere else first.

One way is to first point TM at an AFP share. You might use Netatalk, another Mac, or an AEBS with external drive. Follow these steps:

You can also setup the sparsebundle manually, which you'll find in this hint.

Umounting backup volume on sleep

You're not done. MacOS support for SMB is poor, and anything but an Apple solution won't support the Replay Cache and Lock Stealing features that TM relies on. Storing a filesystem in a sparsebundle is simultaneously a brilliant and a horrible idea. See my rant below. If you put a Mac to sleep during a TM backup to an SMB share, it will disconnect, leaving the volume in an inconsistent state, and odds are, you'll completely destroy the backup volume. The solution is to configure your Mac to automatically unmount the backup volume on sleep. The backup process will notice the unmount request and close down gracefully. You'll be left with a partial backup, which TM will clean up later.

There are a number of web pages that explain how to do this, but this is my favorite one.

Some final rants regarding the horror that is Time Machine backups over network

Apple's Time Machine has the advantage of storing backups in a relatively transparent format. Granted, it's in HFS+ format, which makes it less than transparent to any other platform. But if you have your backup on an external drive, you can just mount the filesystem, and besides using the Time Machine UI, it's very simple to find your files using Finder. One of the tricks Apple pulled to keep incremental backups compact was to allow the use of hard-linked directories. If a subtree on your system has not changed since the last backup, then that subtree will appear in the latest backup via a hard link to the last place it was backed up to. Unfortunately, this trick came back to haunt Apple with the development of Time Capsule.

Since most network filesystems (AFP included) don't support hard-linked directories, Time Machine cannot just store the backup directly on the network volume. Instead, they encapsulate the backup in a virtual disk called a sparsebundle. To the network server, the sparsebundle appears as a directory structure containing a large number of 8MiB files. These 8MiB 'bands' are just segments of the virtual disk, like tracks on a hard disk. The virtual disk is then formatted internally as an HFS+ filesystem, which can then store hard-linked directories with impunity. Although the data is transferred over the network, the HFS+ volume is mounted 'locally' on the Mac, so it can organize file and directory structures independently of network filesystem restrictions. This works well, until you encounter a network problem. If you're in the middle of writing a file to a network share, and the network disconnects, the server can handle it gracefully and leave the filesystem in a consistent state.

On the other hand, if you're in the middle of writing a file to a sparsebundle mounted over a network share, the server has no concept of what is file data and what is metadata, and MacOS may be writing those out of order. Disconnecting while writing to a sparsebundle is like pulling the plug on a hard disk while it's being written to. Even with a journaling filesystem, things can go wrong, resulting in data loss and filesystem corruption. Not surprisingly, corruption of the backup has been a significant problem in the past for Time Capsule users, most commonly when a notebook is put to sleep during a backup and woken up on another network. Most recent versions of AFP have added features to work around some of these problems, but features like TM Lock Stealing and Server Replay Cache are just kludges that work around fundamental design problems.

Apple's desire to make their backup format transparent have lead to reliability problems, and the irony is that unless you're using a Mac, an HFS+ volume inside of a sparsebundle isn't really all that transparent.

[crarko adds: A good summary of the state of affairs, and some good tips about the SMB shares. I've used several of the Apple-type solutions for network Time Machine backups, including Time Capsule, Xserve hosting, and USB drives connected to an Airport Extreme. I never really like any of them, and had similar issues to the author's in working with them. In the end, I've concluded that I'll use Time Machine with a directly connected backup device, and something like SuperDuper for networked volume backups. Feel free to put other favorite methods in the comments.]

Comments (39)


Mac OS X Hints
http://hints.macworld.com/article.php?story=20110406195505167