Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!


Click here to return to the 'Install rdiff-backup for incremental backups' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Install rdiff-backup for incremental backups
Authored by: lar3ry on Sep 28, '05 08:04:30AM
A couple of years ago, when rdiff-backup was first written, I thought it was a good idea and did all my unattended (nightly) backups using it. (This was on Linux, but it's the same program, even though more recently copies may be a bit different nowadays.)

One day, a minor problem corrupted something and I found that getting back to earlier revisions of a file became impossible. A backup solution that doesn't have the ability to work around media failures was false security! Note: the program may have improved in this capability since then, but my feeling is once bitten twice shy.

I now use rsync exclusively with intelligent "--exclude" names such as "Cache/" and "core.*" etc. In addition, rsync "out of the box" has the capability to create incremental backups (--backup and --backup-dir). The incrementals are the directory structure and flat files containing the old values when they change. Using the correct rsync options to support extended attributes, it makes it easy to restore any version of a file on a granularity of 24 hours. I use the following backup directory naming convention:

/Volumes/Backup/FULL/...restOfPath
and
/Volumes/Backup/INCREMENTAL/$year/$month/$day/...restOfPath

The $year, $month, and $day are variables that hold the numeric values for the current date.

Of course, I now have a LOT of incremental files. When the directories are more than a month old, I use "gzip" to compress them to save space.

find /Volumes/Backup/INCREMENTAL/year/month -type f -print0 | xargs -0 gzip -9

I also use another system to back up the full and incrementals over the LAN to a backup system so that I have machine-independence (it's not 100% secure: a bad fire in my house will render both the main system and backup system useless, but I figure I have worse problems to worry about then). People might want to consider copying the incrementals to DVD or CD-RW discs.

Things to worry about are

  • Files that continue to grow (appended files). On Linux, for instance, and pre-Tiger OS X, mail files contain all mail messages appended to one another. Syslog files are like this also, except they do self-archive nowadays. This is not an effecient method to incrementally back up such files. I think rdiff-backup or even CVS might be more appropriate for those.
  • If you have Spotlight indexing on the backup volume, you might get bogus results. Turning off Spotlight for the backup volume isn't the right answer, either, since you might want to use Spotlight to find older versions of files sometime.
  • This requires scripting (perl, python) to work properly, and such scripts must be customized for your own use.

For people planning on using this technique, here's most of the command line that I use:

rsync -avEz --delete -delete-excluded --force --exclude=[...your own excludes] --ignore-errors --backup --backup-dir=/Volumes/Backup/INCREMENTAL/$year/$month/$day /Volumes/MySystemVolume/ /Volumes/Backup/FULL/

I do all this within a perl script and the command line is generated on the fly, but this is the gist of what's actually happening. When run at an opportune time by cron, it's a simple and rather portable solution.

[ Reply to This | # ]