Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

An rsync-based backup solution for easy backups UNIX
I had the need for a simple backup solution for disk-to-disk backups on servers. I wanted it to be administered through a terminal (via ssh), run in the background without user interaction, and email me a report once it did its work. I also wanted it to autorotate my backups, and to use hardlinks to save space and speed up incremental backups. Since I couldn't find something simple that suited my needs, I wrote mlbackup.

mlbackup is a convenient wrapper around rsync 3, which is included in the download. rsync 3 itself is compiled as an universal binary, optimized for PPC and Intel processors in 32-bit and 64-bit incarnations.

There are a ton of features in mlbackup; beyond the autorotation, use of hard links, and email notification noted above, it also includes easy configuration via a text file, backs up all HFS+ metadata, skips certain OS X files (caches, temp files) that don't need to be backed up, and postpones Spotlight indexing until the backup is completed. It can also be used to remotely pull backups with rsync ssh transport. (This requires ssh hostkey authentication to be set up for automated backups.) For this to work, mlbackup needs to be installed on both source and destination Macs, and having the same version installed across Macs is highly recommended!

mlbackup is available for free under the GPLv3, and all sources are included with the download. Bugreports and feedback for mlbackup are welcome; the email address can be found in the README file included in the download. Backups created with mlbackup pass the backup tests in the Backup-Bouncer test suite.

[robg adds: I found this worth mentioning because I've basically cobbled together a series of scripts that do similar things, but each one needs to be customized for the backup task at hand, and I know I don't copy all the proper HFS+ metadata. While I haven't tested mlbackup yet, I intend to -- and given that it's free and open source, I thought others might be interested as well. I have mirrored the r88 download (2MB) here on Mac OS X Hints, in case the original server isn't available at some point in the future. You should check the author's site first for the latest version, however.]
    •    
  • Currently 2.11 / 5
  You rated: 2 / 5 (9 votes cast)
 
[33,776 views]  

An rsync-based backup solution for easy backups | 28 comments | Create New Account
Click here to return to the 'An rsync-based backup solution for easy backups' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
An rsync-based backup solution for easy backups
Authored by: fracai on Sep 26, '08 08:10:33AM
Check out rsnapshot as well.
Perhaps not as many features (all it does is backup and rotate, using a simple config file), but simple to use and easy to modify.

---
i am jack's amusing sig file

[ Reply to This | # ]

Second vote for rsnapshot
Authored by: wireface on Sep 26, '08 08:48:40AM

I second rsnapshot. It also is rsync based, makes use of hard links, does automatic rotation, error chechink, etc. It is easy to set up and has been around for years so it has had lost of testing and updates. I was a big fan of rsnapshot before time machine came along and I still use it on my pre 10.5 macs and other unixes.



[ Reply to This | # ]
An rsync-based backup solution for easy backups
Authored by: bfad on Sep 26, '08 08:32:32AM
While it doesn't include the rsync3 binary, lbackup does everything just mentioned and works with rsync3. (Just make sure you get one that's compiled to do meta-data copying)

[ Reply to This | # ]
An rsync-based backup solution for easy backups
Authored by: lowbatteries on Sep 26, '08 08:33:59AM
Carbon Copy Cloner is an rsync-based backup solution also (with a GUI). It can back up locally (to an attached disk) or over the network/internet to a remotely based Mac, it can even archive modified/deleted items. No rotation feature yet, though.

[ Reply to This | # ]
An rsync-based backup solution for easy backups
Authored by: leamanc on Sep 26, '08 02:02:27PM
I'm pretty sure Carbon Copy Cloner uses ditto instead of rsync. Couldn't find verification one way or the other on Bombich's site, though. Perhaps it uses them both.

[ Reply to This | # ]
An rsync-based backup solution for easy backups
Authored by: pepi on Sep 26, '08 03:32:57PM

CarbonCopyCloner used to use ditto up until version 2.3. From CCC 3.0 on it switched to rsync with the current version 3.1.1 utilizing the power of a similarly patched rsync 3.0.1. (Whereas mlbackup uses rsync 3.0.3 and rsync 3.0.4 is the current rsync version at the moment.)

mlbackup doesn't want to compete with CCC. They both just happen to use the excellent rsync 3 (like many other utilites that have been suggested here). It has a different approach and goal.

Just choose the tool that meets your tasks needs and never trust only a single backup.
Pepi



[ Reply to This | # ]
An rsync-based backup solution for easy backups
Authored by: Anonymous on May 11, '09 05:58:45AM

it uses rsync 3.05(currently)....it is included in the ccc_helper package.



[ Reply to This | # ]
An rsync-based backup solution for easy backups
Authored by: ubrgeek on Sep 26, '08 07:13:52PM

I really like CCC for attached backup devices, but it has real issues working via a wireless connection to a backup machine running something other than OS X. I was trying to get it to play nice between a MacBook Pro and an Ubuntu box and just had no luck. I'm interested in this option.



[ Reply to This | # ]
An rsync-based backup solution for easy backups
Authored by: amaloney on Sep 26, '08 12:12:59PM
I recommend Backuplist, also rsync-based
http://rdutoit.home.comcast.net/~rdutoit/pub/Softw.html

Al Maloney

[ Reply to This | # ]
An rsync-based backup solution for easy backups
Authored by: cayla on Sep 26, '08 12:23:45PM

What exactly are you losing if you don't copy the extended attributes / metadata? (Those are essentially the same thing, right?)

If I just want to be able to recover the actual data in my home directory and I don't care about losing any custom icon data, etc.

Am I hurting myself skipping this step? (eg, no -E on rsync)



[ Reply to This | # ]
An rsync-based backup solution for easy backups
Authored by: billabOng on Sep 26, '08 02:11:37PM

You'll lose more than custom icons (speaking from experience). Use the -E flag (I most commonly use -avE).



[ Reply to This | # ]
An rsync-based backup solution for easy backups
Authored by: cayla on Sep 27, '08 11:50:45AM

Can you give me examples of what more I could lose?

The poster below mentioned losing Safari file bookmarks. (I assume they just mean like bookmarks created when you drag a link to the desktop for example and not the internal Safari ones).

Essentially, all I am worried about restoring is actual data. Songs, pictures, text files, movies, etc. Is there that much extra data actually stored in the resource fork? I though OS X was a step away from this?

It probably has more to do with the pokey NAS I use than anything else, but I have noticed that backups take substantially longer when I use the -E flag. It is almost as if rsync can't tell that the information hasn't changed. (Or maybe it has?) Furthermore, it is strange that is takes so long to move 4K of data regardless. (That is how big the NAS shows the resource as. Eg, ._[filename]).

And like you, I was using -aE. Now I just use an -E unless I can be convinced I am losing something valuable.

Thank you both for the responses so far. I appreciate it.



[ Reply to This | # ]
An rsync-based backup solution for easy backups
Authored by: miggins on Sep 28, '08 08:01:47PM
An rsync-based backup solution for easy backups
Authored by: mael on Sep 26, '08 02:57:45PM

I have a similar mindset. I don't like metadata. There isn't much you'd loose. Honestly.
Unfortunately links you dragged from Safari are stored in the resource fork (metadata) while the data fork (file content) is empty. You know, those "somthing.webloc" files. Yeah. I curse Apple every time I stumble across one of those.
Leaving behind the metadata you'll have perfectly empty file. If you double click, it even starts Safari, but thats about it. No information anymore inside.

Links dragged from Firefox work, but then, I do like Safari quite a lot...



[ Reply to This | # ]
An rsync-based backup solution for easy backups
Authored by: miggins on Sep 28, '08 08:04:56PM

A file's name, type, dates (modified, created, accessed, etc), size, location and permissions are metadata. How can you even have files without some basic metadata?



[ Reply to This | # ]
An rsync-based backup solution for easy backups
Authored by: davidmorr on Sep 28, '08 03:03:45AM
Read this article which, amongst other things, gives a good overview of the metadata and why you would want to keep it:

http://blog.plasticsfuture.org/2006/03/05/the-state-of-backup-and-cloning-tools-under-mac-os-x



[ Reply to This | # ]
An rsync-based backup solution for easy backups
Authored by: cayla on Sep 28, '08 11:25:45AM

This was interesting.

Thank you.



[ Reply to This | # ]
An rsync-based backup solution for easy backups
Authored by: miggins on Sep 28, '08 08:00:13PM
Whatever software you end up using for backup, test it using this tool: http://www.n8gray.org/code/backup-bouncer/. This can give you a quick and easy way to actually know if your backup software can handle all the types of metadata Mac OS X supports.

Personally, I've been using rsync 3.0.3 since it now supports everything Mac OS X has: http://www.bombich.com/mactips/rsync.html.

[ Reply to This | # ]
An rsync-based backup solution for easy backups
Authored by: dashard on Sep 29, '08 08:06:19AM
Wanted to try, since I've lately been knee-deep in a Time Machine-type recreation of a client's backup strategy using rsync. The installer that the author very kindly put together, unfortunately, did not work, giving me an error every time I tried to run it. If you too get this error "Operation could not be completed. (com.apple.installer.pagecontroller error -1.) -- Couldn't open "mlbackup r88.mpkg"." the simple workaround to a working install is to use Pacifist to open the .mpkg file, then once in Pacifist select the main installer and click the "Install" button. Worked for me. Testing now.

[ Reply to This | # ]
An rsync-based backup solution for easy backups
Authored by: pepi on Sep 29, '08 11:50:06PM

Do you have any more information on the problems with my installer package? Like the Logs from installer or anything that your system.log might have captured?

It's the first time I read that someone had a problem with the package so I'm interested if I can do something with the installer package to help, or if it is an isolated problem on your side.
Best regards
Pepi



[ Reply to This | # ]
An rsync-based backup solution for easy backups
Authored by: pepi on Sep 30, '08 01:03:52AM

I've just released mlbackup-r95.

What's new?

  • Added the missing install instructions for the new rsync 3 package installer.
  • Corrected outdated information in the README file regarding ACLs and Finder/Spotlight comments. They are in fact correctly backed up since r88.
  • Added commenting to the sample configuration for those using mlbackup with a broken postfix setup.
  • Fixed a bug that occurred when you used spaces in the set name (MLbackupName) Thanks to Julius Zeidler for reporting this one.
  • rsync 3.0.4 is now included. To see what is new in rsync, please see the rsync 3.0.4 release notes.

Thanks to everybody who has given feedback. I really appreciate it! Best regards Pepi



[ Reply to This | # ]
LBackup
Authored by: lucidsystems on Oct 15, '08 02:14:59AM

You may also be interested in LBackup. Also, a wrapper to rsync, LBackup supports pre and post flight scripts and has built in email reporting.

The LBackup website also has various tools so that if you are managing the backup of many machines or many backup servers which each backup many machines then you will be able to quickly scan though the logs files for errors.

Hope this helps



[ Reply to This | # ]
LBackup
Authored by: lucidsystems on Oct 15, '08 02:18:31AM

Sorry, looks like someone bet me to this post. Sorry for the double up in information.



[ Reply to This | # ]
An rsync-based backup solution for easy backups
Authored by: bed42 on Dec 30, '08 10:22:13PM
For interests sake I've documented my own hobbled-together rsync setup at http://abednarz.net/wp/?p=236

[ Reply to This | # ]
mlbackup has been updated to 115
Authored by: pepi on Feb 04, '09 07:58:44PM
There is a new release of mlbackup available now. Improves exclusions handling, rotation and now can also do online versionchecks. Best regards Pepi

[ Reply to This | # ]
mlbackup-r125-3.0.5 released
Authored by: pepi on Apr 18, '09 06:36:02PM

Yet another update to mlbackup has been released:

Release Notes

Download

This update is recommended for all users of mlbackup!

Best regards

Pepi



[ Reply to This | # ]
mlbackup-r128-3.0.6 released
Authored by: pepi on May 10, '09 03:09:16PM
Scope:

Update to rsync 3.0.6 and improvements to the installer.

This update is recommended for all users of mlbackup .

What's new in mlbackup-r128-3.0.6? Improved
  • The installer will not try to fix the postfix queue system on Mac OS X. It will never touch postfix on Mac OS X Server.
  • Sending of notification email is now more reliable when mlbackup is called from launchd.
Bugs worked around:

rsync 3.0.6 has a bug where creation dates and times are not preserved on intel x86 in 64 bit mode. PPCs and intel 32bit are not affected by this. This is tracked in rsync Bugzilla under the ID: 6276. Because of this bug in rsync, the version bundled with mlbackup comes with only a 32bit binary for intel based Macs to work around this bug. PPC (G3, G4, G5) Macs are not affected by this.

Download mlbackup

Best regards Pepi



[ Reply to This | # ]
An rsync-based backup solution for easy backups
Authored by: pepi on May 29, '09 06:55:33AM
Update to mlbackup-r142-3.0.6 released! Now with Growl notifications! See the Release Notes for more info what's new. Best regards Pepi

[ Reply to This | # ]