Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

Create incremental backups using an rsync shell script UNIX
Remarkably, there are many pitfalls in the tools currently available to back up OS X. I ran into difficulties with several recommended backup solutions, such as the popular Carbon Copy Cloner and others. After poking around the web, I found a shell script solution based on rsync that works fine for me.

Here's my version of it. It keeps two older backups, and uses hard links for all unchanged files to avoid unnecessary duplication. Edit it in the obvious way to put the backups where you want them. Also, you must run this script as root.

[robg adds: I haven't tested this one.]
    •    
  • Currently 3.20 / 5
  You rated: 3 / 5 (5 votes cast)
 
[28,772 views]  

Create incremental backups using an rsync shell script | 18 comments | Create New Account
Click here to return to the 'Create incremental backups using an rsync shell script' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Create incremental backups using an rsync shell script
Authored by: rfitz on Mar 22, '07 08:01:27AM

Does this work with ACL's?



[ Reply to This | # ]
Create incremental backups using an rsync shell script
Authored by: stsmith on Mar 22, '07 12:36:05PM

From man rsync:

-E, --extended-attributes
Apple specific option to copy extended attributes, resource
forks, and ACLs. Requires at least Mac OS X 10.4 or suitably
patched rsync.


[ Reply to This | # ]
Create incremental backups using an rsync shell script
Authored by: stsmith on Mar 22, '07 01:10:02PM
Also see here and here.

[ Reply to This | # ]
Create incremental backups using an rsync shell script
Authored by: bwestmoreland on Mar 22, '07 08:11:51AM

There are numerous issues with RSync in Tiger that make it a non-ideal backup candidate. Support for extended attributes being chiefly among them. There have also been reported issues ACLs, so if you decide to implement this hint, then please be cautious and verify your backups very hard before relying on them to save you.



[ Reply to This | # ]
Create incremental backups using an rsync shell script
Authored by: stsmith on Mar 22, '07 12:51:37PM
Heard about Tiger issues with rsync, at least up until 10.4.4. To the best of my knowledge, I have not experienced any of these difficulties with 10.4.8 or 10.4.9, but as with all backups, caveat emptor. There are patches to rsync 2.6.3 for the previously mentioned difficulties.

[ Reply to This | # ]
Create incremental backups using an rsync shell script
Authored by: rcoconnell on Mar 22, '07 08:43:12AM
I've been running similar to this, and it works just fine for me. The bit about spotlight is a nice twist. I would point out that you can easily automate this with Lingon ( http://lingon.sourceforge.net/ ). Set it up as a User Daemon, and set the username to "root".

<a href="mylink">duh</a>

[ Reply to This | # ]
It's called rsnapshot...
Authored by: dbs on Mar 22, '07 08:47:05AM
http://rsnapshot.org

It does exactly this, but it allows you to specify different generations of snapshots to keep. Works pretty well. It is not nearly as elegant as true filesystem-level snapshots, but quite usable on linux machines. I run rsnapshot on my linux cluster and on my MacBook Pro. The cluster goes over the network to another NAS and my Mac goes to an external firewire drive. No problems, but I don't use ACLs on the mac.

[ Reply to This | # ]
Create incremental backups using an rsync shell script
Authored by: rflorence on Mar 22, '07 08:47:08AM

Unless you modify the source to rsync (instructions are available at several sites on the web) and recompile it, rsync presents several problems as a backup tool: when used with the -E flag to backup attributes it reports an error when it fails to delete the pseudo-file (._filename) complement of an already deleted file, and it changes the modification dates of the pseudo-files on each backup.

The modified rsync used with the --delete option, and with a Spotlight toggle like the one the poster has copied from Slashdot, does a pretty good job and does not report bogus errors or change modification times inappropriately.

- Ronald Florence



[ Reply to This | # ]
Create incremental backups using an rsync shell script
Authored by: stsmith on Mar 22, '07 12:53:36PM
Heard about Tiger issues with rsync, at least up until 10.4.4. To the best of my knowledge, I have not experienced any of these difficulties with 10.4.8 or 10.4.9, but as with all backups, caveat emptor. As you say, there are patches to rsync 2.6.3 for the previously mentioned difficulties.

[ Reply to This | # ]
Excellent!
Authored by: googoo on Mar 22, '07 08:52:13AM

Thank you! I wrote a much less sophisticated script that does the same thing, but yours is much nicer. I keep daily incremental backup folders for previous five weekdays and I have a weekly incremental backup folder for each week since last November (when I started doing this). Since rsync hard links identical files in the incremental backup folders, they are only written once on the hard drive. Having regular incremental backups is a comforting thing.

-Mark



[ Reply to This | # ]
Create incremental backups using an rsync shell script
Authored by: mortimerpa on Mar 22, '07 09:56:54AM
I use rdiff-backup since two years and it works great! it keeps an incremental backup very easily. You can recover the latest version with a simple copy of a directory and have many operators to go back in the versions. I'll recommend it to everyone!

[ Reply to This | # ]
Create incremental backups using an rsync shell script
Authored by: lar3ry on Mar 22, '07 11:14:08AM

I've had only one problem with rdiff-backup. I had an incremental respository that, for some unknown reason, because unusable or corrupted. Long story short: unable to get to an incremental, so I gave rdiff-backup the boot. Sorry, but a backup solution that isn't 100% reliable isn't a backup solution.

rsync has incremental support, using the --backup and --backup-dir options (check out the man page).

I use rsync (even with the so-called "problems" it has on OS X) with --backup and haven't had a single problem, and have been able to get to every incremental that I ever wanted.



[ Reply to This | # ]
Create incremental backups using an rsync shell script
Authored by: stsmith on Mar 22, '07 01:06:14PM
I'd like to see a comparison of rdiff-backup, rsync, and rsnapshot for 10.4.9 -- these don't appear in the excellent blog post Mac Backup Software Harmful.

[ Reply to This | # ]
Create incremental backups using an rsync shell script
Authored by: Ross T on Mar 22, '07 11:52:06AM
I wish to be respectful to the poster, but there seems to be no end of backup scripts offered by users who assure us that "it works for me." To my knowledge, only one user has actually investigated exactly what the various backup solutions do and more important fail to do. May I suggest that anyone proposing a backup solution read these two articles first:
<a>http://blog.plasticsfuture.org/2006/03/05/the-state-of-backup-and-cloning-tools-under-mac-os-x/</a>;
<a>http://blog.plasticsfuture.org/2006/04/23/mac-backup-software-harmful/</a>;
The author's point is that unless you know what your backup system does to BSD flags, file ownership, permissions, resource forks, HFS+ extended attributes, and ACLs, then you can't realy claim that it works.

My nightly script runs on a server to wake up sleeping Macs and tell them to run their local backup script. It incidentally tells them to run their periodic maintenance scripts while they're awake. I used to use rsync but found it so buggy on Tiger that I switched to a modified version of psync. I still have concerns, though, which is why I do not recommend my own script, although, as the saying goes, it works for me.

[ Reply to This | # ]
Create incremental backups using an rsync shell script
Authored by: Ross T on Mar 22, '07 11:54:50AM
Create incremental backups using an rsync shell script
Authored by: stsmith on Mar 22, '07 01:21:06PM

Agreed -- That was the very first link!



[ Reply to This | # ]
Explanation plus networked version and more
Authored by: name99 on Mar 23, '07 02:59:05PM
My wiki has a few pages describing all this in much more detail eg
• how rsync works, and how the hard links work
• where to get a decent rsync that works with extended attributes,
• how to do a network backup,
• a full backup-excludes file, with explanation,
• what to do (most important) if your hard drive dies and you need to restore from backup, etc etc.

http://name99.org/wiki99/index.php/Backup_Hardware


[ Reply to This | # ]
Create incremental backups using an rsync shell script
Authored by: lucidsystems on Sep 06, '11 06:57:31PM

You may also find LBackup of interest. This is a backup wrapper for rsync which has been under development for 10+ years. It is available as a .pkg installer and the website offers a number of video tutorials as well as documentation to get you up and running.

When you install LBackup there are a number of example pre and post actions which make encrypting a backup with disk utility a snap as well as post action scripts to simplify distribution of geographically separated copies of your important backup (within a .sparsebundle image) to one or more remote server(s).

Finally, LBackup is released under the GNU GPL (just like rsync).

[ Reply to This | # ]