Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

Back up key files via rsync and ssh UNIX
If you find online backup solutions expensive and heavy on your system, you might be pleased to hear there is a cheap, easy way to create a mirror backup your Mac (or other UNIX variant) to a secure online server using a program named rsync. And as of Mac OS X Leopard, rsync will even transfer metadata associated with your files like tags and comments.

This method does not allow you to roll back to previous dated backups like certain backup solutions do (Time Machine, for example). What it does is create a mirrored backup of your Mac on a remote server, so it's best used in conjunction with a local hard drive-based incremental backup solution. Still, if the house burns down, your files will be safe and once you've done the initial backup, rsync is very efficient at keeping your remote backup mirrored with your disk.

In this example, we're going to backup the currently logged-in user's Documents folder.

What you'll need:
  • rsync running on your Mac (it's there on 10.4 and 10.5)
  • An Internet Service Provider (ISP) who allows you to connect via SSH. Examples: 1 2 3.
Procedure:
  1. Set up the remote server
    • You're going to need shell access to your server. If you don't already have it, and you use cPanel to administer your server, then there is often a 'Request Shell Access/Remote Login' option. If there isn't, contact your ISP directly.
    • With shell access set up, create a folder that is outside of the scope of your webserver. You could use FTP to do this, if you prefer. For example, set up a folder called private_BKP at the root level of your server, and put a folder in there called Mac_One_BKP. The path is then /home/private_BKP/Mac_One_BKP, and it's in there that we're going to put the contents of our Documents folder.
  2. Set up your Mac
    • Launch the Terminal application (found in your Utilities folder).
    • Type the following:
      rsync -avz -e ssh /Users/yourUserName/Documents yourLoginName@yourServer.com:'/home/private_BKP/Mac_One_BKP'
      An easy way to do the first part is to type rsync -avz -e ssh, then drag and drop your Documents folder right into the Terminal window. Watch out that your system does not add a slash (/) after the word Documents. If it does, just remove it. Why? The slash will tell rysnc to copy the contents of the Documents folder, rather than copying the whole folder.
    • For the second part, you need to guarantee that yourLoginName@yourServer.com:'/home/private_BKP/Mac_One_BKP' is pointing to the folder you created on your remote server.
    • Before starting the transfer, you may want to exclude certain files and folders from being sent to the server (very private stuff, caches, etc.) To do this, see the section below on Deletes and Excludes.
    • Press Return, and you'll be asked for your password. Provide it, and you should see a long list of files scrolling in the Terminal window. These are the files moving from your machine to the remote server.
Some notes:
  1. If you want to check out what the -avz -e stuff does, take a look at the rsync man page.
  2. Your first backup can take hours, if not days (don't worry -- it'll be much faster the second time around, thanks to rsync's ability to transfer only the parts of files that have changed. Take a break from the computer and have a buy a real copy of this instead.
Deletes and Excludes:

One you've backed up your Mac's Documents folder using the method above, there will come a time when you will want to run the script again -- like when you add new content to your folder! However, you many also have deleted some content from your Documents folder.
  • To have rsync delete files on the remote server that are no longer on your machine, add --delete to the command: rsync -avz --delete -e ssh ...etc.
  • To have rsync exclude the transfer of certain files to the remote server , add --exclude='myFileName' to the command: rsync -avz --delete --exclude='myFileName' -e ssh ....etc.. You can add multiple --exclude statements, or add a list via an --exclude_from file (see the manual).
[robg adds: I use a variation on this technique (combined with password-free SSH connections and cron) to create a redundant offsite backup for certain critical data. I also use it to back up sensitive documents (financials, images of passports, etc.) after first writing them to an encrypted disk image. Some day, I'll write it all up in detail for posting...]
    •    
  • Currently 2.75 / 5
  You rated: 3 / 5 (8 votes cast)
 
[26,078 views]  

Back up key files via rsync and ssh | 12 comments | Create New Account
Click here to return to the 'Back up key files via rsync and ssh' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Back up key files via rsync and ssh
Authored by: macavenger on Jul 01, '08 07:45:30AM
A caveat and a note: While it is true that Tiger and Leopard's RSYNC allows you to save metadata (using the -E switch, which is not actually mentioned in the hint):

-E --extended-attributes copy extended attributes, resource forks

this ONLY works if the remote rsync also supports it- which won't be the case unless you are syncing to another 10.4 or 10.5 box. And I believe 10.3 has rsync as well, although also without the -E switch to preserve extended attributes.

---
Aluminum iMac 20" 2.4 GHz/3GB/300GB HD

[ Reply to This | # ]

Back up key files via rsync and ssh
Authored by: blouis on Jul 04, '08 04:35:32PM
Check this:
http://www.bombich.com/mactips/rsync.html

rsync 3.0.2/3.0.3 with patches works flawlessly.



[ Reply to This | # ]
Back up key files via rsync and ssh
Authored by: adrianm on Jul 01, '08 07:59:22AM

I've been doing this for a while (to dreamhost) and one thing I've noticed is that resource fork for file bla get written as .bla on the server (so dreamhost doesn't have to support metadata - good).

Unfortunately, these 'dot' files don't have a time stamp so rsync will copy it in its entirety each time. Not good if a file has a large resource fork, or if you have thousands of files.

I'm thinking of changing my rsync strategy to use a sparse bundle instead - backup to a sparse bundle and rsync the stripes.

---
~/.sig: not found



[ Reply to This | # ]
Back up key files via rsync and ssh
Authored by: Helge33 on Jul 01, '08 08:55:24AM

Yes I did the same observation, seems to be the only practicable option to accomplish Mac-like rsync sessions.

(Amusing to see a "hint" from an apparent fresh OSX User who has never done a google search on "rsync OSX"... ;-)



[ Reply to This | # ]
Back up key files via rsync and ssh
Authored by: lucidsystems on Jul 21, '08 06:33:11PM

There are some notes and scripts to get this working

There are some notes and example scripts to do on the LBackup website.


[ Reply to This | # ]
Back up key files via rsync and ssh
Authored by: amaloney on Jul 01, '08 09:10:14AM
I recommend Rob duToit's BackupList which uses rsync.
http://homepage.mac.com/robdut/ROBJAN/Backup2.html

Al Maloney

[ Reply to This | # ]
Back up key files via rsync and ssh
Authored by: lowbatteries on Jul 01, '08 09:42:38AM

I recommend using the official rsync and not Apple's. The official rsync version 3+ can also transfer metadata, and has a lot of speed improvements. Install it on your local machine and on your server machine (I use dreamhost also).

I replaced the rsync version on all of my machines using "sudo port install rsync" - you'll need MacPorts - I'm sure there's a hint somewhere here on gettings MacPorts set up.



[ Reply to This | # ]
Back up key files via rsync and ssh
Authored by: n8gray on Jul 01, '08 11:20:28AM
However you do your backups you should verify that your tool of choice passes all the "Critical" and most "Important" tests from the free Backup Bouncer test suite. In particular, Apple's rsync still has problems with the combination of hard links and resource forks.

[ Reply to This | # ]
Back up key files via rsync and ssh
Authored by: leamanc on Jul 01, '08 11:59:50AM

I do this to keep my iTunes and iPhoto libraries in sync amongst three Macs and a Linux box.

It was mentioned above that there may be problem using the -E flag rsync'ing to a machine that doesn't support it, but I have no problems with the files on my Kubuntu 8.10 Linux box. It may be the (K)Ubuntu's rsync supports it, but I see no mention of it in its man page. YMMV.

[ Reply to This | # ]

Back up key files via rsync and ssh
Authored by: n8gray on Jul 01, '08 06:04:33PM

It's likely that nothing in your itunes/iphoto libraries uses extended attributes or resource forks.



[ Reply to This | # ]
Back up key files via rsync and ssh
Authored by: BobHarris on Jul 01, '08 07:11:07PM
If you use rsnapshot, you can have something that is similar to Time Machine. rsnapshot is a Perl script that uses rsync --link-dest=/previous/backup/directory/tree where rsync will create hardlinks to the previous backup if the file is unchanged. This way each backup looks like it is a full backup, but each backup only consumes the space needed for the new/modified files. And since hardlinks are used, you can delete any backup and not affect any other snapshot before or after.
<http://www.rsnapshot.org/>;
<http://www.rsnapshot.org/howto/1.2/rsnapshot-HOWTO.en.html>;

While rsync is not perfect it does allow me to securely (over ssh) backup my Mom's iMac across the internet, and have multiple generations of backup, similar to Time Machine. Then again, I only try to backup Mom's iMac once a day, not every hour :-)

So if you are going to invest your time into an rsync solution, you should take a look at rsnapshot.

[ Reply to This | # ]
DreamHost
Authored by: Pedro Estarque on Jul 02, '08 01:22:28PM
I thought that DreamHost didn't allow backup through rsync.
DreamHost does not currently allow the use of their services for backing up personal files (with exception of the use of Files
Are you using Files Forever or has anything changed in their plans ?

[ Reply to This | # ]