Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

Use rsync to remotely mirror files Network
Did you know rsync is available under Mac OS X 10.2? Cool, now you can remotely mirror directories rather than using rcp or scp. rsync can be tunneled over ssh, making it as secure as scp. I'll not go into that for this hint; you'll need to get the general idea first.

Scenario
As a Solaris admin, I have a hardware failover server, and I need to mirror the web_site directory from the primary to secondary server. I use rsync running as a daemon on the "server" to mirror (synchronize) the secondary client. I do this via a cron job at midnight, so I'm never more than 24 hours behind.

This will allow you to synchronize directories and mirror from another machine as a client. Better than remote copy for me. Here's how to do it.
  1. Create a file called rsync in "/etc/xinet.d/"

    Contents:
    # default: off
    # description: The rsync server is a good
    # addition to an ftp server, as it
    # allows crc check summing etc.
    service rsync
    {
    disable = no
    log_on_failure += USERID
    socket_type = stream
    user = root
    server = /usr/bin/rsync
    server_args = --daemon
    wait = no
    only_from = 10.0.0.0
    }
    # This "only_from = 10.0.0.0" Covers this network.
    # Enter the IP or range of which machine can access this sever.
  2. Add the port in "/etc/services":
    rsync    873/tcp    # rsync
    rsync 873/udp # rsync
    I would place it numerically in the 800 range. (duh) ;-)

  3. You'll need a rsyncd.conf file in /etc. This file with provide your server with mappings to the directory you wish to serve.
    [websites]
    path = /web_sites
    comment = Web Site Directory for OS X
    Other options are available:
      auth users = tridge, susan
    secrets file = /etc/rsyncd.secrets
    The /etc/rsyncd.secrets file would look something like this:
    tridge:mypass
    susan:herpass
    I don't use it. If you do, chmod the file to rw acces for root only. Make as many path mappings as you wish.

  4. Does it work? Test it.

    To see which maps you have set up in rsyncd.conf you'll need to type on the command line:
     % rsync localhost::
    This should return:
    web_sites       Web Site Directory
    For the one we set up.

  5. Start your engines:

    Know you are ready to do some sexy rsync'n. You do not have to set up rsync on the CLIENT machine as you did the server unless you want to (as a daemon). But if you want to "Go the other way" so to speak you will. The command and arguments for me on the CLIENT box is:
     % rsync -og --compress --recursive --times
    --perms --links --delete --verbose
    admin@10.0.0.150::web_sites /web_sites/
    [The rsync command should be one line, not three as shown here].

    I'm preserving owner, groups - compressing the stream, preserving times, permissions, symlinks (if any) - deleting files from the client not on the requesting client - verbose mode - I am admin@10.0.0.150 request my mapped drive in rsyncd.conf [web_sites] to update the web_site directory from the receiving server on /web_sites

    You can exclude your MP3's by adding "--exclude=*.mp3" to the arguments, as many excludes as you wish.

    Note that my directory is also called "/web_sites" on the CLIENT. Yours could be "Library/Webserver/Documents/" in keeping with the OS X default path. Could put it in /some_other_directory/tmp

    You can -HUP xinetd to start the services, or reboot. Whatever you wish to do.
You're good to go! Have fun.

Some reading:[Editor's note: Untested locally, but it seems quite useful!]
    •    
  • Currently 4.50 / 5
  • 1
  • 2
  • 3
  • 4
  • 5
  (2 votes cast)
 
[27,171 views]  

Use rsync to remotely mirror files | 22 comments | Create New Account
Click here to return to the 'Use rsync to remotely mirror files' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
WARNING: DONT DO THAT!!
Authored by: SOX on Oct 23, '02 10:48:07AM

DO NOT USE RSYNC on a mac!!!!! it is not HFS+ aware and it will destroy application resource forks, lose all icons, creator codes and create a massively unpleasant experience.

However, you can use RsyncX which is specifically written to be HFS+ aware. it is available free from macosxlabs.org
http://www.macosxlabs.org/rsyncx/rsyncx.html



[ Reply to This | # ]
WARNING: DONT DO THAT!!
Authored by: 240vac on Oct 23, '02 11:10:48AM

Nonsense - I've been using rsync for a long time to mirror my directories between my Tibook and my unix account at work. With no problems whatsoever.

rsync mirrors all the files, and this of course includes the ones for the resources.



[ Reply to This | # ]
WARNING: DONT DO THAT!!
Authored by: Anonymous on Oct 23, '02 11:24:22AM

Thanks for dispelling this myth 240vac.

Nothing like 240 volts of alternating current to zap the fear.



[ Reply to This | # ]
No You have just been lucky, there is a problem
Authored by: SOX on Oct 23, '02 12:16:54PM

Gee too bad your simply wrong about that 240AC. You've just been lucky you have not encountered a problem. As rule of thumb any flat file can be rsynched with only minor damage.

for a quick harmless example,
1) open word or bbedit and make save a new file called "test"*without* the extension .doc
2) In a terminal window type
rsync test rsync_test
3) open the finder and look at the icons of the two files. or try opening them from the terminal window. you will discover the creator code is missing.

this is harmless example. For a harmful example, try rsynching ALL of your applications then throw caution to the wind and delete the original!. you will discover that a some of them will no longer function. In particular, classic applications tend to be most affected since they often rely on resource forks. The new macOSX generally dont use resource forks and thus tend to just have minor problems like creator codes and icons.



[ Reply to This | # ]
WARNING: DONT DO THAT!!
Authored by: Anonymous on Oct 23, '02 11:21:16AM

I appreciate the info but your warning is a little extreme. I'm sure you agree that the version of rsync can be used under the circumstances I provided. In the case rcp, ftp and scp are also "bad."

I'll look into your suggestion, but since I'm not into moving Mac applications I'll have no use for it.

Your right, but I never suggested this to move anything other than files. We can use this for:

.gzip
.tar
.html
.jpg
.gif
.dmg
.bin
.sit
.zip

and a host of others.



[ Reply to This | # ]
RsyncX has problems
Authored by: gctwnl on May 12, '03 12:15:56PM

The HFS+ implementation on maxosclabs (RsyncX) is broken in that it only works if it is installed on all systems communicating with each other. In other words, you gain HFS+ rsyncing between all the Macs you have installed this on, but it willnot work between the this rsync and the real rsync or between rsyncx serving UFS to HFS+. The code that macosxlabs has generated could be improved (goto statements in C, come on...)

So I would hope that a compatible rsyncx would be developed with support for HFS+ and ordinary UFS.



[ Reply to This | # ]
You CAN do this.
Authored by: Anonymous on Oct 23, '02 11:09:43AM

I concur, since I did not specifically state moving anything with resource forks or mac applications. I use this for files, namely text files supporting a web site, as in the example. This would have no impact. I don't know why anyone would rsync mac applications when there are many other methods of moving or installing them.

This hint is safe under the scenenario I provided and this is provided under the "Unix" catergory. Yes, you can lose icons for files but your Mac can associate the files with the correct application.

Nothing to be afraid of.



[ Reply to This | # ]
You CAN do this, but at some risk
Authored by: SOX on Oct 23, '02 12:30:48PM

For moving plain files rsync works fine as long as you dont mind losing creator codes and icons. So like you say nothing to fear there.

But when moving applications or complex documents with creator codes there is much to fear.

For example one I had to reformat my hard disk, so I made the bad mistake of tarring up all of my files and moving them over a linux machine.
when I untarred not a single one of my applications worked!!! and some of my documents could not be opened by their original apps.

So their is lots to fear since rsync works the same as tar. the good news is that this warning mostly applies to classic apps which heavily use the resource forks. Mac OS X apps have a different structure and thus tend not to have too many problems when they are rsynced.

Note if you dont want to use RsyncX there is an interesting workaround that works great. Create a UFS partion. this can be a real partition or virtual one created by disk copy. drag all of your files into this partion and then use rsync. This will work perfectly because when you drag the files from the HFS+ disk to the UFS disk, the mac dumps the HFS database entryies and resource forks in to a real separate (hidden with a ._ prefix) file that rsync handles correctly as long as you remember to copy it too



[ Reply to This | # ]
Please stop implying this is mostly safe...
Authored by: Basilisk on Oct 23, '02 12:42:13PM

Its only safe assuming one understands the differences between applications that store data in their resource forks and those which don't. It is not a safe general backup strategy, and is not something to recommend unless one understands the limitations.

Yes, using standard rsync is safe if one is only syncing flat data files like text, image, and most sound formats. One loses the creator code, but this can be replaced.

Standard rsync is completely unsafe for any MacOS X application, particularly Carbon applications that run in 9 and X (which almost always use resource forks).

Standard rsync is also unsafe for application files which do store relevant data in the resource fork. This includes some fairly common file formats, including the ever-so-humble SimpleText document (which stores formatting information in a 'styl' resource). This is a trivial example, but arguing that no one uses SimpleText is beside the point. The point is that unless one is sure that your file formats are rsync safe it is better to use RsyncX.

I'm not clear on why folks feel a need to defend rsync here, since no one is arguing that rsync isn't a great idea (I use it everyday). Just use RsyncX an keep yourself safe.



[ Reply to This | # ]
HFS-aware rsync
Authored by: ddulay on Oct 23, '02 11:56:39AM

Macosxlabs.org has been working on a port of rsync that is HFS-aware. Their description of this tool:

"RsyncX is an implementation of rsync with HFS+ support and configuration through a command line (Terminal) or graphical user interface. This release requires some form of SSH to be up and running. For full details on rsync, open a terminal window and enter 'man rsync'. This will introduce you to the theory behind the software. Be sure to read rsyncx_readme as well."
More information on RsyncX is available at http://www.macosxlabs.org/rsyncx/rsyncx.html.
I must confess that I have not played around with RsyncX too much because basic rsync is sufficient for my needs.



[ Reply to This | # ]
Use Deja Vu
Authored by: sms on Oct 23, '02 12:59:00PM

...to do this without typing anything.

You can get it here:
http://www.versiontracker.com/moreinfo.fcgi?id=16206&db=mac

As far as I remember it also uses rysnc (if you only want to backup the changed files), but also works without.

The best things are:
* it can schedule 3 different backup times
* it can be controlled via its own System Preferences Pane
* it is Freeware



[ Reply to This | # ]
What will work for FAT32?
Authored by: mgarabed on Oct 23, '02 01:17:11PM

Interesting discussion, unfortunately the only backup solution I have involves FAT32 partitions over a network. Does RsyncX support backing up to a FAT32 drive? Is there a backup solution that will work?

Naturally, I would want to preserve the resource forks, etc, so that I can just backup a slew of files without worrying about it.



[ Reply to This | # ]
rsync w/o daemon
Authored by: copelanduk on Oct 23, '02 03:10:45PM

You can rsync any two machines (as long as rsync is installed on both machines) without running a daemon on either machine or adding rsync to inetd etc.

For example I use something like this to backup my ~ to one of my debian servers:

rsync -rptvlWe ssh --exclude-from=/Users/USERNAME/Documents/Scripts/rsyncexclude.txt --delete --progress --stats ~/ server:backup/MACHINENAME

my rsyncexclude.txt file looks something like this:

# --- MACHINENAME ---
- com.apple.loginwindow.plist # root owned file
- com.apple.NetInfoManager.plist # root owned file
- .localized
- .FBC* # can always re-index
- .DS* # window positions not critical
- Cache/ # skip mozilla/chimera cache
- *.waf # skip IE stuff
- .Trash* # no trash please

As long as you have sshd running on the destination and use "-e ssh" (assuming ssh is in your path), you'll be able to use rsync over ssh w/o an rsync daemon. All Jaguar installs can be used as destination or source with rsync by just turning on remote login in the sharing panel.

PS. I use rsync to mirror my entire Mac OS X home directory and various other directories with no problems (mostly .txt, .rtf, .doc, .html, .xml, java, .class, .mp3, .mov, .sh, .pl, .pdf .png, .tif, etc) and have used rsync with FreeBSD and Linux for some time now. On Mac OS X there ARE problems; I wouldn't try to keep my /Applications folders on my 3 macs in sync with rsync.

Lets say rsync is not for everyone, and indeed it's probably more suited to 10.2 users with a unix background than a classic background. Nonetheless letting users know how they can use a really powerful tool like rsync is important.



[ Reply to This | # ]
rsync w/o daemon
Authored by: fireproof on Oct 24, '02 11:32:25PM

How do you deal with the password stuff? I can't seem to get --password-file=FILE to work (does it have to be in some special format? I'm using a text file), or the RSYNC_PASSWORD environment variable to work either...

Aside from that your post rocks! Very handy, thanks to both of you for putting this information together.



[ Reply to This | # ]
rsync w/o daemon
Authored by: copelanduk on Oct 25, '02 06:32:44AM
I copied the text below from the rsync man page, this is the section on the different ways rsync can be used:
1. for copying from the local machine to a remote machine using a remote shell program as the trans- port (such as rsh or ssh). This is invoked when the destination path contains a single : separator. 2. for copying from a remote machine to the local machine using a remote shell program. This is invoked when the source contains a : separator. 3. for copying from a remote rsync server to the local machine. This is invoked when the source path con- tains a :: separator or a rsync:// URL. 4. for copying from the local machine to a remote rsync server. This is invoked when the destination path contains a :: separator.
In my case I am using server:~/ (not server:: as in the original hint) and specifying to use ssh with the -e switch so I am using method 1. (to server) and 2. (from server). Whatever method you use you to authenticate with ssh will also apply for the rsync over ssh connection. After you enter the rsync command ssh will make a connection and prompt you for your account password if necessary. In my case I use key exchange so I can just enter the rsync command from my previous post and it will run right away. (I've got several different rsync commands stored as term files in: ~/Library/Application Support/Terminal/*.term So I can easily access them from Terminal.)

[ Reply to This | # ]
rsync w/o daemon
Authored by: fireproof on Oct 28, '02 05:40:11PM
Would you care to write a tip on using key exchange? Sounds useful...

[ Reply to This | # ]
rsync w/o daemon
Authored by: copelanduk on Oct 31, '02 11:21:12PM
By key exchange I meant ssh public/private key exchange, which is covered in these hints: here and here. Otherwise google for ssh-keygen, and ssh-agent.

[ Reply to This | # ]
rsync w/o daemon
Authored by: themax on Oct 30, '02 03:17:38PM

Ok here we go. I took your advice and tried

rsync -rptvlWe ssh --delete --progress --stats ~/rsynctest.txt server:/users/myusername/

I was promprted for my ssh key password which I entered correctly and then...

building file list ...
1 file to consider
51303.1/rsynctest.txt.151303
22 100% 0.00kB/s 0:00:00
rsynctest.txt
37 100% 0.00kB/s 0:00:00
rsynctest.txt/rsrc

Number of files: 1
Number of files transferred: 1
Total file size: 37 bytes
Total transferred file size: 37 bytes
Literal data: 59 bytes
Matched data: 0 bytes
File list size: 32
Total bytes written: 223
Total bytes read: 36

wrote 223 bytes read 36 bytes 47.90 bytes/sec
total size is 37 speedup is 0.14


But when I go to look at the file on the remote machine it has a size of ZERO?

Any ideas??

Thanks
BostonMACOSX



[ Reply to This | # ]
rsync w/o daemon
Authored by: copelanduk on Oct 31, '02 11:46:35PM
Ok I just ran: rsync -rptvlWe ssh --delete --progress --stats ~/rsynctest.txt server:~/ And this is what I see:
rsync: building file list... rsync: 1 files to consider. rsynctest.txt 6947 100% 0.00kB/s 0:00:00 Number of files: 1 Number of files transferred: 1 Total file size: 6947 bytes Total transferred file size: 6947 bytes Literal data: 6947 bytes Matched data: 0 bytes File list size: 32 Total bytes written: 7035 Total bytes read: 36 wrote 7035 bytes read 36 bytes 744.32 bytes/sec total size is 6947 speedup is 0.98
My file is fine on the server. Which leads me to believe that the output you saw was from a file with a resource fork. I see only one "100%", you had two, so I guess your file is copied and then replaced by its resource. Try creating a new text file from the CLI: touch ~/rsynctest2.txt then add some text to the file with vi/pico (or TextEdit.app) or just type: cat ~/Sites/index.html >> ~/rsynctest2.txt and run the rsync command again but with rsynctest2.txt. If it works then I guess it was a resource issue, and you'll have to use RsyncX and be careful about what you use rsync on. If the command still doesn't work with the plain text file, you might have a problem with the server (what OS is it running?) or conflicts between rsync versions etc.

[ Reply to This | # ]
rsync w/o daemon
Authored by: themax on Nov 04, '02 10:28:42AM

Ok the solution was that I only had RSYNC on my Jaguar box and RSYNCX on my 10.1.5 box. Now that I have RSYNCX on both I'm fine.

Thanks for the advice.!!!!!!



[ Reply to This | # ]
rsync w/o daemon
Authored by: themax on Nov 04, '02 10:33:18AM

Ok the solution was that I only had RSYNC on my Jaguar box and RSYNCX on my 10.1.5 box. Now that I have RSYNCX on both I'm fine.

Thanks for the advice.!!!!!!



[ Reply to This | # ]
Bottom Line
Authored by: dr_turgeon on Oct 23, '02 04:51:50PM
Good hints! (withstanding the resource-fork freak-outs ;)

It's simple:

rsync for modernized (single-fork) files like web-sites and most cocoa products

rsyncX for the hold-overs from 9 (including some new from Apple stuff, heh...)


[ Reply to This | # ]