Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

A summary hint of rsync-related information UNIX
I have been trying to make an OS X version of an off-site backup system I currently run on Linux boxes using rsync and an rsync server. In the process, I figured out a number ofthings about OSX rsync that I though would be worthwhile to clarify in a single post.

If you do not know about rsync, it is a great utility for mirroring two mounted volumes or mirroring to a remote "rsync server." You can find the details and useful links at rsync website on samba.org.

Things to be aware of:
  1. 1: For Mac HFS+ volumes, you need a special version to accomodate the resource forks on Mac files. This version is known as "RsyncX" and/or rsync_hfs - please see the detials below for further information.

  2. If you are using rsync via an SSH tunnel to mirror to a remote Mac, make sure both machines are running the same HFS+ aware version of rsync.

  3. There is a posted configuration for an rsync server in the macosxhints archive, but I could not get it to work. Digging a little deeper, it seems that rsync does not run in server mode under OS X. Here is an email I received from the RsyncX team:
    rsync binaries currently do not support the "--daemon" mode of operation in OS X. This is true of any binary release, and not just RsyncX. The rsync maintainers are aware of the problem, but have not found a suitable fix.
Read the rest of the article for more info on rsync...

Information: Here is a summary of what I managed to figure out and confirm on the status of rsync for OS X. I'm pretty sure it is all correct.
  1. Both RsyncX and rsync_hfs are written and/or managed by Kevin Boyd.
  2. RsyncX includes a OS X GUI wrapper, while rsync_hfs is only command line. Installing RsyncX will also install rsync_hfs into /usr/local/bin/rsync.
  3. rsync_hfs may be (and as of July 14th 2003 is) slightly more up to date than RsyncX with respect to the rsync core code.
  4. OS X 10.2 install includes the following rsync versions:
    /usr/bin/rsync
    /usr/bin/rsync.2.3.1
    /usr/bin/rsync.2.5.2
    /usr/bin/rsync.hfs.2.4.6
    
    In this list, /usr/bin/rsync is equivalent to rsync 2.5.5 protocol 26 with HFS+ support, which is the same things that is installed by RsyncX 1.7 in /usr/local/bin/rsync.
  5. If you want to use the most recent rsync_hfs, you have to download and compile it yourself form the OpenDarwin projects.
[robg adds: I've never used rsync, but thought this might be useful to someone out there...]
    •    
  • Currently 1.00 / 5
  You rated: 1 / 5 (2 votes cast)
 
[14,377 views]  

A summary hint of rsync-related information | 12 comments | Create New Account
Click here to return to the 'A summary hint of rsync-related information' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
A summary hint of rsync-related information
Authored by: SOX on Jul 21, '03 11:21:34AM

I've used to use RsyncX. In general rsync is a marvelous product. But I ran into a snag with it and stopped using it. the problem is that there is a maximum file count limit. I'm not quite sure what the limit is. but when I try using it on files systems with lots of files I always get malloc errors. unfortunately the while the program does print the error message it also does not abort and instead runs to completetion and returns. Thus when doing this automatically you will not know it failed unless you check your logs. <p>
for this reason I switched to psync. it is HFS aware too. psync is no where near as full featured as rsync but it has the most useful features. it does not work well unless you can mount the remote disk. and due to root squashing on mounted disks, moving root owner file is a problem. however it does not have the file count limitiation and has never failed to work for me.
<p> I've been thinking about switching to rdiff back but 1) I dont know if its hfs aware and 2) it may use rysnc under the hood. any tips?



[ Reply to This | # ]
A summary hint of rsync-related information
Authored by: olve on Jul 21, '03 01:04:52PM

I agree that psync does a "cleaner" job with OSX and in my experience it is the only option if you're going to a non-HFS volume, but rsync wins out in terms of options.

FYI, I run regular "local" mirror backups of several volumes mounted from remote machines. I use rsync_hfs for afp:// volumes and psync for smb:// volumes. Of course, these volumes must always be mounted for this to work. I would of course prefer an rsync_hfs server where the individual clients contact and add to the backup, but as an alternative, the "-e ssh" option in rsync will work, but you have to allow remote root access for get full backups. With psync, the volumes must be mounted.

I have also noticed the malloc errors in rsync when you have a file for which there is no read permission. Is this perhaps the issue more than max file count? On the Linux side, I regularly rsync 80 GB volumes with lots of file without any problems.

As a side note, it is ironic that Retrospect makes preference files in user directories that are only admin accessible and this foils rsync_hfs backups! (I think this is only in the directory of a user that set up the Retrospect job and may only apply if one also sets Retrospect to required admin authentication for change, but am not sure yet).

Olve



[ Reply to This | # ]
Backup Mac->Linux?
Authored by: awk on Jul 21, '03 01:10:10PM

What's the best way to backup from a Mac to a BSD machine?

At the moment I use Retrospect Express FTP feature but it's not very good, I don't like the interface and it constantly chokes on some Unicode filenames. And FTP is something I'd like to get rid of in general.

Can RsyncX copy HFS files to a Linux machine? I'd think it would be possible by treating the resource fork as a second file. If not I was thinking of trying NFS-mounting the backup dir and using RsyncX to copy from the Mac to the backup dir, though I don't think rsync is optimized for this kind of setup.



[ Reply to This | # ]
Backup Mac->Linux?
Authored by: olve on Jul 22, '03 04:47:27PM

This is exactly what I currently do and my reason for looking into rsync was to get away from Retrospect. Unfortunately, I also confirmed with the rsync_hfs team that you cannot use rsync to backup to linux volumes without losing your resource forks.

I have not tried the follwing scenario fully, but I think it will work pretty well to use psync to backup to an NFS mount from a Linux box or another Mac. Psync does work to this type of volume, and if you set up automounting (see the NFS Manager program), then the remote volume should always be available to psync. You can also make the mountpoint a bit obscure (e.g. /mnt/xxx) to keep the backup volume from appearing on the desktop.

Unfortunately, if you want to psync everything then psync runs as root and it must then have root equivalent access on the NFS mount from the remote machine. Depending on your network and system setup, this may or may not be palatable. At least you can restrict the NFS mounts by IP of client machines. BTW, I think there is also an option in psync to save permission/ownership stuff in a separate file without modifying the files themselves on the destination.

I think this is what I will try next. There is probably some clever read-only trick in NFS to remount the backup directory (or a subdirectory thereof) as read-only so people can restore their data without risk of overwriting the backup. Of course, you will probably have to edit UID and GID in NetInfo to match your OSX users with Linux users so that they can see their backed up files.

Frankly, this would be so much easier if rsync_hfs would just work reliably and in server mode under OSX !!!



[ Reply to This | # ]
A summary hint of rsync-related information
Authored by: bluehz on Jul 21, '03 01:29:38PM
I have tried for over a year to get a good clean backup with various incarnations of rsync (hfs-aware) and have never made a single, error-free backup. Call me a stickler, but as far as I am concerned - a backup is not a valid backup unless it completes without errors. I tried working with the rsync developers but they seemed less than receptive, indicating the problem was most definitely on my end and it worked perfectly every time on their machines. Still I try and try and try with rysnc... for functionality it has EVERYTHING I could possibly want in a backup routine. But at this point in time - its purely a toy I pull out every once in a while when I wanna toy with some shell scripting.

FWIW - I sent a hint in last week (has not made it to the boards yet) about an hfs aware version of tar that I was reading about on macintouch also (I think). Anyway - have not tried it out yet - but its on the table...

http://www.helios.de/news/news03/N_06_03.html

[ Reply to This | # ]

A summary hint of rsync-related information
Authored by: escowles on Jul 21, '03 04:14:55PM

I've also been toying around with tar and rsync (and their hfs-aware counterparts) for a while, now. In the end, I don't want to use some forked version of rsync -- if it's stable, it should be merged into the main rsync tree, if it isn't, I'm not using it for my backups.

In the end, I decided to use vanilla rsync for my main (daily) backups, and live with losing resource forks. Most of my documents and other frequently-changing files don't need resource forks anyway. For the apps and system files that do need resource forks, I make backups to DVD-R monthly or so.

-Esme



[ Reply to This | # ]
A summary hint of rsync-related information
Authored by: Brontojoris on Jul 21, '03 07:51:21PM
I run the following script as a cron job, and it has worked really well for me. Note that I mount the Multimedia share first, also via a cron job, first. The share is mounted using an internet shortcut, as this as proven the most reliable.
#! /bin/sh
# Run a weekly backup of the my websites using Psync
echo "Syncronising all of my website folders with the Multimedia server"
echo "A log of file operations can be found at: /Library/Logs/Psync.log"
/usr/local/bin/psync -d -v /Users/joris/Sites 
/Volumes/Multimedia/Archive/websites/Weekly > /Library/Logs/PsyncWeekly.log 
-- Put command above on one line --
echo "Copying files complete"
echo "Remove Dreamweaver file locks"
chflags -R nouchg /Volumes/Multimedia/Archive/websites/Weekly/
echo "Sync complete."


[ Reply to This | # ]
A summary hint of rsync-related information
Authored by: whoadoggy on Jul 22, '03 05:27:46PM

If you don't mind could you also please include the command or script you are using to mount your network drive? I have been wanting to do this but not yet dug enough to find the answer. Also any comments on why the internet shortcut is most reliable? What else did you try?

Thanx,
whoadoggy



[ Reply to This | # ]
A summary hint of rsync-related information
Authored by: Brontojoris on Jul 22, '03 07:23:37PM
To make a shortcut type the following url into your web browser (I'm using Safari):

afp://username:password@10.10.10.10/VolumeName

Then drag the url to the desktop, rename the shortcut (I renamed it to: "Multimedia File Server") and file away somewhere. Then in my Crontab I call:

/usr/bin/open "/Users/joris/Library/Favorites/Quick Connect/Multimedia File Server"

[ Reply to This | # ]

A summary hint of rsync-related information
Authored by: krishna on Aug 31, '03 02:50:34AM

I've used Carbon Copy Cloner (http://www.macosxhints.com/article.php?story=20030529054605961) to make a verified bootable, (apparently) clean backup of my TiBook G4 MacOSX 10.2 boot drive to an external firewire drive.

I plan to use the external drive as my primary drive for a few weeks to verify that the backup is completely clean. I also plan to use CCC's psync integration to keep my laptop drive in sync with the external drive. I'd like to use rsync_hfs after the first CCC run (since it seems to be 20-30% faster than psync), but I expect to use this setup as a reliable backup mechanism once I verify that it's clean (at which point I'll fully vouch for CCC).



[ Reply to This | # ]
A summary hint of rsync-related information
Authored by: krishna on Aug 31, '03 03:10:42AM

I'm showing my lack of experience in the Mac community by asking this question, and I know Un*x has many quirks that old wonks take for granted, but does it seem odd to anyone that third-party developers need to modify the programs that are released as part of MacOSX (such as rsync) in order to make them work with MacOSX's preferred filesystem (HFS+)?

Considering that Apple is getting almost the entire O/S for free, shouldn't it

* update the utilities to make them work properly,
* or not ship the utilities if they're almost certain to cause unexpected problems (such as loss of resource forks),
* or at the very least, update the man pages to indicate that they'll cause problems with HFS+,
* or at the very very least, remove the rsync man pages to indicate that it's not a documented utility?

Having access to Un*x under the hood of a platform hosting high-quality (Photoshop, Macromedia) and standard (Palm Desktop, Adobe office software, etc) applications makes me very, very happy -- so don't get me wrong. These oversights just seem out of whack for a system that otherwise works so well.



[ Reply to This | # ]
A summary hint of rsync-related information
Authored by: alan-trewartha on Feb 12, '04 12:50:53PM
***LONG POST WARNING ***

I thought I'd just relate the results of my successful experimentation with RsyncX and the RsyncxCD - and this seems like the right place to do it...

I've had a lot of success with spot synchronisations with a module/mount point on a server and, more recently, installing entire system images with RsynxCD (latest version 2 I think).

With the CD I have been able to boot up all "New World" machines (well back to a Grey G4 single 450) and install the same system image and it works perfectly. At 100Mbps it takes about 8 minutes to download the image, bless and reboot. Unfortunately it takes about 5 minutes to boot up from the CD on a Grey G4 450!

I've installed the rsyncX (version 2.6.0) executable in place of the standard /usr/bin/rsync on a standard client system and on a server machine configured to run as a daemon. For the server daemon (simply invoked with the --daemon switch) I used the RsyncX utility/wizard that steps you through the creation of the required rsyncd.conf and rsyncd.secrets file, but basically they are like:

/etc/rsyncd.conf (chown'd by root:wheel and readable by others):

secrets file = /etc/rsyncd.secrets
read inly= yes
list = yes
uid = root
gid = wheel
 
[TestMod]
auth users = rsyncx_usera, rsyncx_userb
path = /Applications

/etc/rsyncd.secrets (chown'd by root:wheel and NOT readable by others):

#rsyncd secrets file
rsyncx_usera:usera_password
rsyncx_userb:userb_password

Now if the client has a password file (root readable only or rsync complains) you can run the following script on a client without any user intervention:

/usr/bin/rsync -a -c -z "rsyncx_usera@server_address::TestMod/" \
/Volumes/Work/NewTestApps --eahfs --showtogo --password-file /Library/Management/rsync_password

This syncs the clients "/Volumes/Work/NewTestApps" folder with the server's "Module"/share point (in the above case the Applications folder). Pretty useful and pretty quick - it took a couple of minutes to take an inventory, and about 40 seconds to upload a large adobe application . However I have occasionally hit file number limits (previously mentioned) - only on v large transfers, and it is solved by a second run.

I haven't found any file limit problems with the technique of pushing/pulling loadsets. I did have to enable root user on the server though. Without root, the copying was imperfect. Here I used scripts from the University of Michigan's macosxlabs pages.

To push a loadset:

rsync --eahfs -qe ssh root@server_address:'`mkdir -p "/loadsets/1/Network" \
"/loadsets/1/private" "/Homes/loadsets/1/private/tmp" "/Homes/loadsets/1/dev" \
 "/loadsets/1/private/var" "/loadsets/1/private/var/run" "/loadsets/1/Volumes"`'

time sudo rsync -a --eahfs --exclude "/private/var/automount*" --exclude "/dev*" \
 --exclude "/private/tmp*" --exclude "/Network*" --exclude "/Volumes*" \
 --exclude "/private/var/run*" -e ssh /.  "root@server_address:/loadsets/1" --showtogo

Pretty much all I added to the scripts at the link above is the crucial --eahfs switch, which adds the all important hfs+ support. (I also added the automount exclusion - don't know if I needed that).

Now booting from the RsyncXCD you can initialise the hard disk and pull down the loadset with:

time rsync --eahfs -a -z -e ssh "root@server_address:/loadsets/1/" \
"/Volumes/OS X" --delete --showtogo

bless -folder "/Volumes/OS X/System/Library/CoreServices" -bootinfo "/Volumes/OS X/usr/standalone/ppc/bootx.bootinfo

set the System Disk in System Preferences and restart the machine. Done. Like I say, about 8 minutes at 100Mbps

[ Reply to This | # ]