Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

10.5: One way to use Time Machine and a backup server System 10.5
So I just did my first roll out of Leopard server to a client (one server, six iMacs), and I've run into most of the problems everyone's been having (AFP connection issues, iCal server issues, etc.). Due to the wonderfully large brains that visit this site and the Apple discussion boards, I've got it working together finally. So I felt it was time for me to contribute some of my hopefully good wisdom and share what I did for Time Machine.

Of course I was seduced by the ease of Time Machine (TM for short) and the interface that would let the users recover files simply without harassing IT. But reality set in and I quickly realized that Time Machine does not back up well over the network, even to Leopard Server, which supposedly is made to support it. On top of that, I, like every other sane server admin worth his salt, wants a rotating backup that can be taken off site. For small offices, the best option is usally two external FireWire drives rotated weekly. During Tiger, I used rsync on all the client machines in an office and had them back up directly to the external drive on the office server. When they were swapped out, rsync would keep on backing up as long as the drives and backup destination folder were named the same.

I had hoped that Time Machine would be intelligent to handle this too, but obviously it can't. If the drive is swapped out, the next time TM tries to connect, it fails and I have to manually go to each workstation and tell it to choose the network share I created on the server as a TM destination again. Obviously going around to every client on a weekly basis to do this would suck. So I started my thinking cap and tried to come up with a solution that would give the user the awesome Time Machine interface, but back up over the network to my external drive that gets rotated.

After reading on this site that you could point TM to a partition on your main drive as a backup destination after getting past the "if your drive fails" warning, I figured it out. My clients had new iMacs with 250GB drives, and since they were storing their data on the server, they didn't really need much storage. So using Disk Utility, I added a second partition and shrunk the main partition (thank you Apple for giving us the ability to do this non-destructively in 10.5), so that I had a 150GB main partition and a 100GB backup partition.

Each of the client machines and their hard drives are named in a sequence, ie. clientname01, clientname02 or Apple01, Apple02, etc. So I named the new backup partition with the same number so they would be named differently on the backup server, ie. Backup01 is on clientname01, etc. I went into TM and chose the backup partition as my destination and let it do the first backup. Then I dropped to the Terminal and issued sudo setfile -a V /Volumes/BackupXX, where XX is the number of the backup. This hides the partition from being seen in the Finder. TM still sees it and backs up to it, but users can't accidentally eject it or delete files or otherwise mess it up.

Now, I'm still trying to figure out launchd, so I just made a cron job that runs as root to rsync the backup to the server. 10.5's rsync handles forks and other extended attributes, so you no longer have to download RsyncX. Here's the command I use:
date >> /var/log/rsyncd.log; time rsync -a -H -E --progress --force --delete-after -e ssh "/Volumes/BackupXX/." "root@serveraddress:/Volumes/externaldrive/NetworkBackups/BackupXX" >> /var/log/rsyncd.log
NOTE: rsync option -a is archive mode, -H preserves hard links (crucial in Time Machine), -E preserves extended attributes, --progress echoes the progress, --force forces deletion of dirs even if not empty, --delete-after deletes files on the backup that weren't on the source after the backup operation is finished, and -e ssh makes rsync transmit the data over an ssh pipe for security. Note, too, that XX denotes the matching backup numbers, and serveraddress and externaldrive should be replaced with actual addresses and drive names.

This creates an exact duplicate of the Backup partition in the destination folder on the server. All activity is echoed to /var/log/rsyncd.log so you can monitor it (don't forget to add the rsyncd.log to /etc/newsyslog.conf, so that it gets rotated). All the normal rsync configuration is needed, ie., Remote Login turned on at the destination, rsa keys generated and client keys added to the authorized_keys file on the destination so that a password is not required for ssh, etc.

If your client machine dies and you need to restore from backup, go to the server and share the backup folder. On the client, mount the share. Then tell Time Machine to Browse another Time Machine disk and choose the Backups.backupdb folder on the share.

So this makes Time Machine run quickly because it backs up locally and gives a great user experience to the user, gets the backup onto a server's external backup drive that gets rotated, and allows for simple data recovery.

If there's anyone out there with bigger brains than me that sees a problem with this or can think of ways to tweak it, feel free to add to this. All I hope is that this can help some other server admins out there trying to deal with Time Machine and a sane backup policy.
    •    
  • Currently 2.67 / 5
  You rated: 4 / 5 (6 votes cast)
 
[35,353 views]  

10.5: One way to use Time Machine and a backup server | 14 comments | Create New Account
Click here to return to the '10.5: One way to use Time Machine and a backup server' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
10.5: One way to use Time Machine and a backup server
Authored by: tonyy on Dec 23, '07 11:38:58AM

This is a great hint; thank you for that. I'll definitely try it out.

However, from my standpoint, I'm holding off on deploying Leopard Server in any way or form because it's simply broken in way to many places and if you're not there full time; it's a nightmare waiting to happen.

Once its ready, then TM will work as intended. On a parallel front, I configure the server to use mobile profiles for everyone. This creates a mirror image of every user's profile on the server at all times. Then using PSYNC, I back everything up to external drives that are rotated weekly.



[ Reply to This | # ]
10.5: One way to use Time Machine and a backup server
Authored by: matsw on Dec 24, '07 01:51:19AM

Very good idea. The only possible problem I see with this solution is that rsync may not be able to handle hard links to directories, but only to files. The solution would work anyway but the backups would be very large.



[ Reply to This | # ]
10.5: One way to use Time Machine and a backup server
Authored by: Josh43 on Dec 24, '07 12:27:25PM

Good point:

Can anyone do a test and confirm if leopard's rsync does handle hardlinks to directories? (And leave a comment here? ;) )



[ Reply to This | # ]
10.5: One way to use Time Machine and a backup server
Authored by: ikioi on Dec 24, '07 12:42:11PM

There's only three possible issues that I see.

ISSUE 1:
The Finder in 10.5 provides an extended interface for permissions on files. The old 10.4 "Get Info" window just let users set regular unix owner/group/world read and write permissions. These days (10.5), you can list more than just owner/group/world and what you set in the "Get Info" window is stored as an ACL on the file or folder. In 10.4 ACLs were off by default and almost never got created on a normal machine. in 10.5 they are on by default and likely to end up at least occasionally created by users even if those users have no idea what ACLs even are.

Time Machine (TM) is extremely clever at handling ACLs. When a file or folder with ACLs is backed up in TM, all of the ACLs are shifted by one spot (ACL #0 becomes ACL #1, and ACL #1 becomes ACL #2, etc) and a new ACL is created in slot #0. The new ACL denies permission to delete or modify the file in any way. That's why you can't delete files from your TM backup using Finder.

So all of this is great in TM, but the only question is whether rsync properly copies ACLs. There are a couple of way rsync might not (I'm not sure). It might fail to copy them at all (which means that some permissions set by users in Finder would disappear, and the backups on the server could be unwittingly modifed) or if rsync does copy Apple ACLs, it might do a breadth first copy instead of depth first meaning that a folder might get the "deny all changes" ACL set before the child files or folders are copied.

I don't know if any of those are going to be problems, but there's a chance. The mitigating factors are that TM may use rsync as it's backend anyway, which would mean you're set and even if it doesn't, of the above two problems you might not be too worried about the first one, and you'd probably notice the second one right away because almost nothing would back up to the server.

ISSUE 2:
This is a more serious issue and it simply depends on whether some bugs in 10.4's rsync have been fixed in 10.5. In 10.4, rsync would copy extended attributes and resource forks and such, but if you did rsync -aHE, then any file with a resource fork and more than one hard link would silently be skipped. No error, no notice at all, the file just wouldn't be created on the target. I recall the behavior was more subtle if the file already existed on the target. If you haven't already checked it, it's worth making a file with a resource fork, backing it up with TM, modifying its parent folder, and then backing up the file again with TM (to make sure the file has more than hard link), and the finally, trying to rsync the TM backup to an empty target location and seeing if that file is skipped by rsync.

ISSUE 3:
This one won't kill you but might make the server copy take more space than TM copy. TM uses hard links to folders, but I don't know if the API to do that is publicaly available because hard links to folders are dangerous if used improperly. I know the ln command _cannot_ create hard links to folders in 10.5 . I don't know if rsync can or not. If it can't, then it wouldn't even know to check for hard links, so it would still behave gracefully, just creating lots of duplicate entries for folders and files on the server that TM didn't have to on it's local partition. It means the server drive will be slower on disk lookups, but it shouldn't break anything.

Anyway, that's my 2 cents. I can't think of any other problems with it. Nice idea!



[ Reply to This | # ]
10.5: One way to use Time Machine and a backup server
Authored by: ikioi on Dec 24, '07 12:43:45PM

Woops . Looks like matsw already mentioned my ISSUE 3. Sorry for the duplication. :-)



[ Reply to This | # ]
10.5: One way to use Time Machine and a backup server
Authored by: lukasha on Dec 25, '07 08:18:23AM

Thanks everyone for looking this over. Let me attempt to go over some of the issues brought up.

When testing, I did in fact backup files with forks, modify them, and check afterwards, and everything seemed okay. I also checked sizes on the local machine and the remote backup and they were pretty much the same so I'm assuming that rsync is handling hard links for files and directories okay. 10.5's rsync also seems to be able to handle all extended attributes including ACLs perfectly. I haven't been able to find a lot of chatter about rsync in 10.5 out on the internet, so I had to rely on testing it myself and seeing if it worked, which so far it does.

The other thing to think about is that the rsync is not performing an incremental backup, Time Machine is doing that to the local partition. All the questions about incremental changes to forked files and such are kind of moot, because TM is the one handling all that to the local backup partition. All rsync is doing is mirroring the partition to a folder on the backup server periodically. The way to think of it is like it's a mirror RAID, but just not updated instantaneously. And the only reason you would share out the backup folder on the server is after a client machine's system had died and you've reinstalled the OS and such. Then you mount the share and use Time Machine to recover the user's stuff. That way you don't have to worry about a user messing up the backup.

I hope this answered your questions, and again, if I'm missing something or you can think of improvements, please chime in.



[ Reply to This | # ]
What about files changing?
Authored by: avramd on Jan 16, '08 02:02:59PM

Sorry I'm a little late on this discussion - but if the original poster is watching, I'd like to hear your response.

What happens if files on the client are changing while the rsync is in progress? It seems like it would be possible to end up with a partially corrupt backup of your TM files if those files themselves changed while the rsync was in progress. And I have to imagine that a 150 GB rsync takes quite a while, so it seems that you're really banking a lot on the TM backups themselves not to change.

If this an issue, perhaps a solution would be a little script that puts TM in manual mode during the rsync - but then you have to worry that TM doesn't get properly re-set to automatic mode.



[ Reply to This | # ]
What about files changing?
Authored by: kd4ttc on Jan 19, '08 11:44:15AM

Just set up the cron job to run rsync at a time the user isn't likely to be there. Also, this puts rsync bandwidth usage to an idle time. For example, use a time between 1 AM and 5 AM.



---
Steve Holland



[ Reply to This | # ]
What about files changing?
Authored by: knappis07 on Jan 23, '08 08:46:14AM

I don't think files changing is an issue here since (correct me if I'm wrong) no files or directories will ever change in a TM backup. TM will only add new files or directories (or hard links) and remove expired ones. What I think could happen is that you might only get a partial copy of the latest backup in the event TM is currently doing its work but that should be fixed with the next rsync.



[ Reply to This | # ]
What about files changing?
Authored by: lukasha on Jan 24, '08 08:09:42AM

From what I can tell, as the other person said, Time Machine only adds and deletes and doesn't change any files in the backup. And you should probably schedule your rsync for after hours and since no files should be changing when the user isn't there, TM shouldn't interfere with the rsync. But if someone with more knowledge than me can post the commands to check if TM is currently backing up, wait until it's finished, turn it off, run the rsync command, then turn TM back on, I'll gladly add it. I would like to be sure also that the backups are solid.



[ Reply to This | # ]
10.5: One way to use Time Machine and a backup server
Authored by: knappis07 on Jan 23, '08 08:54:49AM

Great hint. I'm looking for the same functionality but I have trouble making it work.

I have tried a modified version of your hint to mirror my TM backup volume to an external drive.

I used the following code:

sudo rsync -a -H -E -v --progress --force --delete-after /Volumes/TM/ /Volumes/BKP_TM >> ~/Library/Logs/rsyncTM.log

The full TM volume was ≈50GB including about 15 backups and there seem to be something wrong with rsync intepretting the hard links since the file count went extremely high and it finally exited with an error (se below).

Does anyone have a clue whats wrong?

To OP, is it still working for you and have you made any modifications since your first post?

building file list ...

......

37143100 files...

rsync(1107) malloc: *** mmap(size=262144) failed (error code=12)

*** error: can't allocate region

*** set a breakpoint in malloc_error_break to debug

ERROR: out of memory in receive_file_entry

rsync error: error allocating core memory buffers (code 22) at / SourceCache/rsync/rsync-30/rsync/util.c(120)



[ Reply to This | # ]
10.5: One way to use Time Machine and a backup server
Authored by: lukasha on Jan 24, '08 05:58:03AM

I know it seems like a small thing, but can you try your command and change "/Volumes/TM/" to read "/Volumes/TM/." (make sure that trailing period is there)? Let me know if that makes a difference.

Jeff



[ Reply to This | # ]
10.5: One way to use Time Machine and a backup server
Authored by: duga on Feb 16, '08 11:04:43AM
I have been trying something similar to this rsync approach and had the same failure (that's how I stumbled on this tip - looking for a solution). I think this is not a viable approach with the current version of rsync shipping in Leopard. The problem is, as the backup grows (in number of files, not total size), rsync will crash. This happens because it is building a list of all the files in the source, which takes about 100 bytes per file (again, actual file size is irrelevant, just the number of files matters). That means the theoretical maximum on files it can sync is just over 42 million. In reality, that number is smaller (closer to 20 million). The newest version (3.0, in pre-release) seems to have a fix for this (--recursive) as well as better xattr and acl support, but it is not (yet) an out-of-the-box solution. I am testing it now to see if it will at least work locally, but it is not quite ready for prime time even if it does.

[ Reply to This | # ]
10.5: One way to use Time Machine and a backup server
Authored by: Fisslefink on Apr 03, '08 12:12:52AM
Here's a potential workaround for the too-many-files-for-rsync problem:

I believe Time Machine can mount a sparsebundle on the local disk for backups (check me on this). If so, you could create a 100 GB sparsebundle (similar to the original poster's suggestion of a 100 GB partition) for local Time Machine backups. This way, those mllions of files are tidily wrapped up in 8MB 'bands' within the sparse bundle. Rsync is quite capable of doing incremental updates of sparse bundle bands (I have tested this).


Just make sure you have a version of rsync on the server that supports the -E flag (I had to compile from source on my old Red Hat box), or when you restore the sparsebundle, it will be recognized as a folder instead of a volume. If you have this problem, use the 'setfile' tools in the developer package to make Finder see it as a 'bundle' again.



[ Reply to This | # ]