Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

Install rdiff-backup for incremental backups UNIX
rdiff-backup is a free backup utility. It does incremental backups, so it is fast and bandwidth efficient. It maintains a version history, so you can easily recover a file as it existed yesterday or two months ago. rdiff-backup combines the benefits of a mirrored backup with the benefits of an incremental backup, and it works on your local machine or remote servers.

The program itself is very easy to use, but installing it on Mac OS X can be a bit tricky if you aren't familiar with building source code or installing unstable Fink packages. Instead, read on for an easy way to install it using DarwinPorts.

  1. Download and install DarwinPorts. [5.0MB direct download link]

  2. After running the DarwinPorts installer, add the following to your ~/.profile file:
    export PATH=$PATH:/opt/local/bin
    export MANPATH=$MANPATH:/opt/local/share/man
    export INFOPATH=$INFOPATH:/opt/local/share/info
    
    The lines above assume you are using a bash shell (the default in a new install of Mac OS 10.3 and OS 10.4). If the .profile file doesn't exist, create it. If you are using a C shell (OS 10.2 and some 10.3/10.4 installs that upgraded from 10.2), instead add this to your ~/.cshrc file:
    set path=($path /opt/local/bin)
  3. To load the changes you just added to your .profile, type source ~/.profile in the Terminal.

  4. Update your DarwinPorts installation to the latest version by running this command in the Terminal:
    sudo port -d selfupdate; port sync
  5. Finally, install rdiff-backup by running this command in the Terminal (this takes a few minutes):
    sudo port install py-rdiff-backup
You're all set! I use a command like this to back up my home directory to my external hard drive:
rdiff-backup /Users/MyUserName /Volumes/external_drive/MyUserName
You can also use it to make backups to/from a remove server using ssh. To recover the latest version of a file, you can just copy it back over from the backup directory. To get an older version of a file, for example as it existed 10 days ago, you can do this:
rdiff-backup -r 10D /Volumes/external_drive/MyUserName/file \
/Users/MyUserName/file
[robg adds: I installed DarwinPorts and rdiff-backup on my PowerBook, which went well. The selfupdate and port sync commands, however, failed (message: "invalid gid nobody"). I'm trying to get to the bottom of that, and I'll post an update here if/when I do. Nonetheless, the rdiff-backup install went just fine (though the compile is time-consuming!), and a test backup of my user's home folder seems to have worked perfectly.]
    •    
  • Currently 3.60 / 5
  You rated: 5 / 5 (5 votes cast)
 
[26,955 views]  

Install rdiff-backup for incremental backups | 21 comments | Create New Account
Click here to return to the 'Install rdiff-backup for incremental backups' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Install rdiff-backup for incremental backups
Authored by: rflo on Sep 27, '05 08:22:44AM

Does rdiff-backup handle extended attributes and resource forks, like the -E option in MacOS-10.4's broken rsync, or the --eahfs option in rsyncx? What is the advantage of rdiff-backup over rsyncx?

---
Ronald Florence



[ Reply to This | # ]
Install rdiff-backup for incremental backups
Authored by: BlackPenguin on Sep 27, '05 08:53:03AM
Not sure of the advantage that rdiff-backup provides over rsync, but it does support HFS+ resource forks and extended attributes. Check here: rdiff-backup Features. Toward the bottom of the page it notes that it does support HFS+.

[ Reply to This | # ]
support of resource forks, access control lists etc.
Authored by: moritzh on Sep 27, '05 09:24:23AM

From the website (http://www.nongnu.org/rdiff-backup/index.html):

"Also features such as Mac OS X resource forks, Extended Attributes, and Access Control Lists have only been released about a year ago [from 2005-09-10]. There are no known bugs in these newer features, but they are not as thoroughly tested as the basic functionality."



[ Reply to This | # ]
Re: benefits over rsync
Authored by: keaka on Sep 27, '05 03:19:57PM

rdiff-backup uses librsync, but rdiff-backup has the benefit of maintaining a version history. Every time rdiff-backup runs, it keeps track of changes to files and the date/time of the backup. Thus, you can retrieve the latest backup copy of a file, or you can grab a copy of the file as it existed at some other time in the past.

---

Sanity Software: http://keakaj.com/



[ Reply to This | # ]
Install rdiff-backup for incremental backups
Authored by: xkalibur on Sep 28, '05 02:22:07PM

Maybe I'm missing something, but the version that DarwinPorts installed for me is 0.12.7, which does not have the features you all speak of (resource forks were added in 0.13.0 according to the changelog). So maybe using Fink is a better option (it's unstable package is at 1.0.x I believe). That's what I'm on to try tonight...



[ Reply to This | # ]
Install rdiff-backup for incremental backups
Authored by: xkalibur on Sep 28, '05 02:22:14PM

Maybe I'm missing something, but the version that DarwinPorts installed for me is 0.12.7, which does not have the features you all speak of (resource forks were added in 0.13.0 according to the changelog). So maybe using Fink is a better option (it's unstable package is at 1.0.x I believe). That's what I'm on to try tonight...



[ Reply to This | # ]
Install rdiff-backup for incremental backups
Authored by: bumper314 on Oct 01, '05 11:37:04AM

Someone made a special patch for rdiff-backup to handle resource forks and whatnot prior to version 0.13.0. I'm pretty sure those were compiled into the DarwinPorts packages because I remember reading it somewhere...



[ Reply to This | # ]
Install rdiff-backup for incremental backups
Authored by: bumper314 on Oct 01, '05 12:12:41PM

Actually, I just tested this with my DarwinPorts installed rdiff-backup, and NO, IT DOES NOT DO RESOURCE FORKS.

My test was to take a plain bob.html file, copy a new icon to it, verify that a resource fork existed (ls -l bob.html/rsrc), rdiff-backup it up, then restore to another location and the file did not contain the resource fork anymore....



[ Reply to This | # ]
Install rdiff-backup for incremental backups
Authored by: bumper314 on Oct 01, '05 06:57:56PM

I tried the latest from fink (which turned out to be 0.13.6), but it fails to run with some cryptic errors. Oh well...



[ Reply to This | # ]
Install rdiff-backup for incremental backups
Authored by: ndrake on Sep 27, '05 08:51:22AM

I see the same problem when I try a port selfupdate. Wonder what's up.



[ Reply to This | # ]
Install rdiff-backup for incremental backups
Authored by: club60.org on Sep 27, '05 10:22:21AM
Both sudo port -d selfupdate and sudo sync fail on my system.

The first one says:

Synchronizing from rsync://rsync.opendarwin.org/dpupdate/dports
rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at /SourceCache/rsync/rsync-20/rsync/io.c(359)
...

The second one is less informative:

port sync failed: sync failed doing rsync
I have TCP/UDP port 873 opened. Any clues?

[ Reply to This | # ]
Restore individual files?
Authored by: jecwobble on Sep 27, '05 01:13:53PM

Looking at the features page, I cannot tell for certain whether or not you can easily restore one single document from a point in time or if you can only restore the entire folder/directory. Does anyone know for sure?

In the bullet Preserves all information I noticed this statement:

Whether you restore from the mirror directory or from an earlier incremental backup, rdiff-backup will reproduce your files exactly as they were. Files missing at the time of backup will also be missing after the restore.

This leads me to think that you cannot restore a single file.



[ Reply to This | # ]
Re: Restore individual files?
Authored by: keaka on Sep 27, '05 06:03:40PM
Yes, you can restore single files. For example, I did a backup, then deleted a file, and then did another backup. The file no longer exists in the direct mirror created by rdiff-backup, but it exists in rdiff-backup's version history. Since I know the file existed 30 minutes ago (and I did at least one backup after that while the file existed), I retrieved the file with this command:
rdiff-backup -r 30m /Volumes/external_drive/keaka/Documents/myfile.txt /Users/keaka/Documents/myfile_recovered.txt

I think the bullet you quoted is trying to say that you need to recover a file from a point in time when it actually existed. For example, I deleted the file 10 minutes ago and then performed a backup. So if I try to retrieve the file from 1 minute ago (using rdiff-backup -r 1m), it won't work. But I can retrieve the file from 30 minutes ago.

---

Sanity Software: http://keakaj.com/

[ Reply to This | # ]

Restore individual files?
Authored by: Cameroon on Sep 27, '05 06:06:28PM

I think what that is saying is that if you restore a directory, the directory as it existed at the time of the backup will be recreated.

From the examples:

But the main advantage of rdiff-backup is that it keeps version history. This command restores host.net::/remote-dir/file as it was 10 days ago into a new location /tmp/file.

rdiff-backup -r 10D host.net::/remote-dir/file /tmp/file

Other acceptable time strings include 5m4s (5 minutes and 4 seconds) and 2002-03-05 (March 5th, 2002). For more information, see the TIME FORMATS section of the manual page.

Which leads me to believe just the opposite of your conclusion, as the example explicitly mentions restoring a file.



[ Reply to This | # ]
Don't explicitly set MANPATH
Authored by: jwbales on Sep 27, '05 05:36:31PM

OS X uses /usr/bin/manpath to construct the MANPATH on the fly. If you set the MANPATH environment variable you will not "see" your other man pages unless you explicitly set them in the MANPATH (unless you invoke man -M command).

/usr/bin/manpath will construct the MANPATH based on directives in /usr/share/misc/man.conf and your PATH environment. If PATH contains /opt/local/bin and /opt/local/man exists it will be added to MANPATH.

See the manpath man page and try running manpath -d.

[ Reply to This | # ]
Re: Don't explicitly set MANPATH
Authored by: sjk on Sep 28, '05 09:45:00PM

Your comments are informative and a helpful warning but the "Don't" in the title seems too strong since there can be valid reasons to explicitly set MANPATH instead of modifying /etc/manpath.config.

Maybe you don't want to (or can't) modify manpath.config and would rather set MANPATH than run "man -M path ..." commands. You could use the output from "manpath -d" as the starting point for constructing a customized MANPATH, keeping manpath.config generic so there's no risk of an local customizations you've made to it being clobbered by an Apple update (although quite unlikely for that particular file).



[ Reply to This | # ]
.bash_profile
Authored by: keaka on Sep 27, '05 06:15:43PM
In the original hint I mentioned adding the export commands to your ~/.profile file. It's probably better to add them to your ~/.bash_profile file. On a default install of OS X, I think .bash_profile is sourced by login shells and applied to each Terminal session, not .profile.

---

Sanity Software: http://keakaj.com/

[ Reply to This | # ]

rdiff-backup vs. rsnapshot
Authored by: sapporo on Sep 28, '05 12:21:37AM
So how does rdiff-backup compare to rsnapshot?

[ Reply to This | # ]
rdiff-backup vs. rsnapshot
Authored by: sjk on Sep 28, '05 10:31:52PM

Good question. Looking briefly at both, it appears that rsnapshot is a conveniently configurable front-end for rsync (and similar to what lar3ry described in his post) while rdiff-backup does essentially the same thing with an rdiff command (which uses the same algorithm as rsync) and has more features (e.g. detailed statistics). rdiff-backup installation looks more complicated unless you use DarwinPorts.

Hopefully someone who's familiar with both can explain the key differences and any advantages or disadvantages one has over the other.

Timely and inspiring thread since I'm overdue to implement this type of incremental backup strategy for a subset of data to supplement my SuperDuper! clone backups.



[ Reply to This | # ]
Install rdiff-backup for incremental backups
Authored by: lar3ry on Sep 28, '05 08:04:30AM
A couple of years ago, when rdiff-backup was first written, I thought it was a good idea and did all my unattended (nightly) backups using it. (This was on Linux, but it's the same program, even though more recently copies may be a bit different nowadays.)

One day, a minor problem corrupted something and I found that getting back to earlier revisions of a file became impossible. A backup solution that doesn't have the ability to work around media failures was false security! Note: the program may have improved in this capability since then, but my feeling is once bitten twice shy.

I now use rsync exclusively with intelligent "--exclude" names such as "Cache/" and "core.*" etc. In addition, rsync "out of the box" has the capability to create incremental backups (--backup and --backup-dir). The incrementals are the directory structure and flat files containing the old values when they change. Using the correct rsync options to support extended attributes, it makes it easy to restore any version of a file on a granularity of 24 hours. I use the following backup directory naming convention:

/Volumes/Backup/FULL/...restOfPath
and
/Volumes/Backup/INCREMENTAL/$year/$month/$day/...restOfPath

The $year, $month, and $day are variables that hold the numeric values for the current date.

Of course, I now have a LOT of incremental files. When the directories are more than a month old, I use "gzip" to compress them to save space.

find /Volumes/Backup/INCREMENTAL/year/month -type f -print0 | xargs -0 gzip -9

I also use another system to back up the full and incrementals over the LAN to a backup system so that I have machine-independence (it's not 100% secure: a bad fire in my house will render both the main system and backup system useless, but I figure I have worse problems to worry about then). People might want to consider copying the incrementals to DVD or CD-RW discs.

Things to worry about are

  • Files that continue to grow (appended files). On Linux, for instance, and pre-Tiger OS X, mail files contain all mail messages appended to one another. Syslog files are like this also, except they do self-archive nowadays. This is not an effecient method to incrementally back up such files. I think rdiff-backup or even CVS might be more appropriate for those.
  • If you have Spotlight indexing on the backup volume, you might get bogus results. Turning off Spotlight for the backup volume isn't the right answer, either, since you might want to use Spotlight to find older versions of files sometime.
  • This requires scripting (perl, python) to work properly, and such scripts must be customized for your own use.

For people planning on using this technique, here's most of the command line that I use:

rsync -avEz --delete -delete-excluded --force --exclude=[...your own excludes] --ignore-errors --backup --backup-dir=/Volumes/Backup/INCREMENTAL/$year/$month/$day /Volumes/MySystemVolume/ /Volumes/Backup/FULL/

I do all this within a perl script and the command line is generated on the fly, but this is the gist of what's actually happening. When run at an opportune time by cron, it's a simple and rather portable solution.

[ Reply to This | # ]

DarwinPorts install guide
Authored by: SnowLprd on Feb 06, '06 07:48:50AM
Just posted a handy guide to installing DarwinPorts:

Installing DarwinPorts on Mac OS X

[ Reply to This | # ]