Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

Run automated backups to a UNIX server UNIX
Our office network has a UNIX server which is backed up nightly, and I wanted to take advantage of this to back up the contents of my home directory on my Powerbook when I'm in the office.

I'm sure there are GUI tools that will do this but I wanted it to run in the background, and only when I'm on the office network and when the server isn't too busy. The prerequisites are that you have a UNIX server which you can connect to with SSH without a password (ie you're in the authorized_keys file).

First I created this script, which I named backup, in /usr/local/backup. Within the /usr/local/backup directory, I then created a directory called logs and a file called EXCLUDES.your_username.

The EXCLUDES.your_username file lists the paths you want to exclude from the backup. Mine looks like this, which excludes Cache items, my IMAP mail (it's all on the server already) and the trash:
- /Library/*/Firefox/**/Cache/
- /Library/Safari/Icons/
- /Library/Caches/
- /Library/**/ImapMail/**/*.msf
- /Library/**/ImapMail/**/*.sbd/
- /Library/**/ImapMail/**/INBOX
- /Library/Mail/**/*.imapmbox/
- .Trashes/
- .Trash/
- /Temporary Items/
- .TemporaryItems/
- .Spotlight-V100/
Finally, I ran crontab -e and added the following line to run the backup every weekday at 11am, when I'm probably in the office:
0 11 * * 1-5 /usr/local/backup/backup > /dev/null 2>&1
Logs are stored in /usr/local/backup/logs folder (be sure to create that directory) and are rotated every day. Messages are sent to /var/log/system.log.
    •    
  • Currently 2.40 / 5
  You rated: 3 / 5 (5 votes cast)
 
[15,470 views]  

Run automated backups to a UNIX server | 15 comments | Create New Account
Click here to return to the 'Run automated backups to a UNIX server' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Run automated backups to a UNIX server
Authored by: blalor on Jan 18, '06 07:41:03AM

The Original Poster should check the downloadable script against the original. The rsync command has been wrapped to two lines.



[ Reply to This | # ]
rsnapshot
Authored by: sr105 on Jan 18, '06 08:03:42AM
You may want to check out rsnapshot. It is a python script that wraps around rsync and allows for incremental backups. It will produce directories by default for every 4 hours, 7 days, 4 weeks, and 3 months. Each directory should be an exact copy of your filesystem at that time, but it uses hard links to keep the space requirements down. Meaning only one copy of the file really exists even though it appears in multiple directories. I'm not sure if it has the load feature, though.

Oh, and it will do multiple systems and can also just do portions of them if you like.

[ Reply to This | # ]

rsnapshot
Authored by: lar3ry on Jan 18, '06 11:51:32AM

Actually, rsync DOES do incremental backups. The "--backup" and "--backup-dir" options will dump the old version of any file found in the destination before overwriting it. I use this myself and put the incrementals into directory trees that have the year/month/day prefixed to the actual subdirectory name. rsync is pretty versatile like that...!

I once got burned by another "improvement" on rsync (called rdiff-backup), which did incrementals and compression, but the repository somehow got corrupted, keeping me from getting any of my old data. I stopped using that utility quickly thereafter. (It may be better now, but once bitten, twice shy.)

Using the --backup* options, a file corruption will only corrupt a single file. Disk corruption can still cause problems as well, but it won't corrupt the entire repository (hopefully).

Of course, you DO backup your backups, don't you?



[ Reply to This | # ]
Resource forks require a patch to rsync!
Authored by: reynhout on Jan 18, '06 08:25:12AM

rsync doesn't understand Macintosh resource forks.

Apple patched rsync (in 10.4) to fix this problem for Mac->Mac transfers, but Mac->other UNIXes still doesn't work properly.

I wrote a patch for rsync that solves this problem. Details at:
http://www.quesera.com/reynhout/misc/rsync+hfsmode

PS: to fix the script, put a backslash ('\') at the end of the first line of the rsync command. Also, FWIW, recent versions of rsync default to using ssh as the transport, so the '-e ssh' is unnecessary.



[ Reply to This | # ]
Resource forks require a patch to rsync!
Authored by: wiesmann on Jan 19, '06 06:19:45PM

You might as well use unison, which supports backin-up resource forks to non Mac servers. This program also supports a large variety of options.

http://www.cis.upenn.edu/~bcpierce/unison/



[ Reply to This | # ]
Resource forks require a patch to rsync!
Authored by: reynhout on Jan 19, '06 07:34:19PM

Yeah, Unison is great for some purposes.

But backups are supposed to be perfect replicas of important data, and Unison doesn't handle Finder metadata properly. It certainly could, but the designers haven't chosen to pursue it fully.. Unison is designed primarily to provide different functionality (multidirectional synchronization).

For keeping your home dir or media files, etc synchronized in multiple locations, Unison is a great tool.

For backups, I think it's hard to beat rsync.



[ Reply to This | # ]
Run automated backups to a UNIX server
Authored by: fungus on Jan 18, '06 12:04:19PM

Thank you. This was a well thought out script. I really like the usage of syslog and complete rsync logs too. My only suggestion would be to schedule it using launchd if you have Tiger instead of cron, because cron cannot run jobs it missed while it was asleep, and launchd can.

I'll be implementing this for my powerbook today.

Your method is perfect for machines not always connected to the network like laptops. the rsnapshot hint is much better for always connected machines that never sleep (ie. servers).



[ Reply to This | # ]
Run automated backups to a UNIX server
Authored by: jhintzman on Jan 18, '06 01:51:24PM

I notice you are not using the extended attributes (-E) option to rsync. This means that rsync will not copy resource forks. I've had huge problems with the -E options to rysnc - the program crashes, and all the resource forks get copied all the time.

Anyone else see this behavior with rsync? What's the impact to user files if you skip the extended attributes? How can I tell which files have them and which ones I care about?



[ Reply to This | # ]
rsync -E option?
Authored by: jhintzman on Jan 18, '06 01:56:11PM

You didn't preserve extended attributes (-E) - I have encountered lots of trouble with this. Is -E broken?



[ Reply to This | # ]
rsync -E option?
Authored by: sjk on Jan 20, '06 11:02:50AM

I've read about trouble with "-E" and have some notes about it (inconveniently elsewhere at the moment) but haven't actually tested it on my systems. If the problem is within rsync it probably hasn't been fixed since there hasn't been an update in any 10.4.x release. Other readers here probably know specific details or you could web search for them.



[ Reply to This | # ]
Run automated backups to a UNIX server
Authored by: mikebremford on Jan 19, '06 03:50:43PM
Quite correct that this script won't handle resource forks and extended attributes. For me this isn't too important, as I'm dealing primarily with data like source code, photos, text files and so on. The script was put to the test recently after a hard disk crash and the only thing I found didn't restore correctly was a couple of fonts - it's certainly not perfect but will cover most of the important stuff. If you're backing up to an OS X server you might try replacing the stock rsync with rsyncx, which functions identically but allegedly handles resource forks. I'll let you know how well the next time my disk goes...

[ Reply to This | # ]
Run automated backups to a UNIX server
Authored by: reynhout on Jan 19, '06 07:39:56PM

Well, if you're backing up to an OSX server, you can also use the rsync binary that comes with Tiger.. And the -E switch.

But if you're backing up to a non-OSX UNIX, you'll lose resource forks and Finder metadata unless you use an HFS+ aware rsync.



[ Reply to This | # ]
Run automated backups to a UNIX server
Authored by: mikebremford on Feb 10, '06 06:09:23AM

Latest update - the very excellent website http://www.lartmaker.nl/rsync details how to enable the backup of extended attributes to a Linux fileserver, or indeed any disk that's not HFS+. Using the steps he's outlined I compiled a patched rsync for Linux - so by adding "-E" to the original backup script the resource forks are now saved as well.

For what it's worth I've now lost 3 hard-disks inside 12 months in my Powerbook, so you can take it from me this script works as advertised. Holding out for that 12" Macbook...



[ Reply to This | # ]
There is a patch for the big problems.
Authored by: dasil003 on Mar 08, '06 01:21:55PM
See this mini how-to with a Tiger rsync patch that fixes the most glaring issues. I haven't tested it yet, but I am planning on doing so now.

[ Reply to This | # ]
Using Rendezvous SSH connection for laptop backup
Authored by: rsnyder on Apr 18, '06 06:30:58AM

I have a Powerbook that I want to backup to my PowerMac, but only when I am at the office on a fast connection. I want to automate the process with a crontab entry.

The solution I came up with was to SSH via Rendezvous, rather than IP address. That way when I am not on the local network the SSH command fails, the error is logged, and the backup only happens when I am on the fast local connection.

To do this use ssh myMachinesName.local instead of ssh 123.234.234.123. The connection will only be successful when I am on the same local network as myMachinesName.

I am sure others have thought of this, but I thought it was a pretty cool way to automate connections to other hosts only when you are the same local network. It could be used for any case where you only want to connect when on the local network.



[ Reply to This | # ]