Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

10.4: Avoid rsync errors during backup due to Spotlight UNIX
Tiger only hintThe command-line utility rsync synchronizes files and directories, and can be used as a tool for backing up important data to external or remote drives. The version of rsync that comes with Mac OS X 10.4 has the ability to include "extended attributes" of files (such as resource forks) when invoked with the -E switch. However, rsync -E fails when used with a large number of files because the Spotlight indexer gets overwhelmed. The command exits with the error:
rsync: fstat failed: No such file or directory
To avoid this error, you can switch off Spotlight indexing for the duration of the sync process.

Thus, a simple backup script might look like this:
#!/bin/sh

EXCLUDES="--exclude tmp* --exclude *Cache* --exclude .Trash*"

# Function for toggling Spotlight indexing
spotlight_switch()
{
  /usr/bin/mdutil -i $1 /
  /usr/bin/mdutil -i $1 /Volumes/Backup
}

# Syncing /Users to /Volumes/Backups
! test -d /Volumes/Backup && echo "Please mount the backup drive!" && exit
spotlight_switch off
/usr/bin/rsync -avE --delete $EXCLUDES /Users /Volumes/Backup
spotlight_switch on
Since mdutil needs superuser privileges to work, the script has to be executed with sudo. For more information on how to use rsync, take a look at its man page.

[robg adds: This earlier hint contains another rsync backup script, but lacks the information about Spotlight indexing. I haven't tested this one...]
    •    
  • Currently 3.75 / 5
  • 1
  • 2
  • 3
  • 4
  • 5
  (4 votes cast)
 
[32,800 views]  

10.4: Avoid rsync errors during backup due to Spotlight | 16 comments | Create New Account
Click here to return to the '10.4: Avoid rsync errors during backup due to Spotlight' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
10.4: Avoid rsync errors during backup due to Spotlight
Authored by: Gigacorpse on Nov 09, '05 07:50:33AM

An easier way to accomplish this would be to set a target directory with a .noindex extension. For example, instead of:

/usr/bin/rsync -avE --delete $EXCLUDES /Users /Volumes/Backup

you could do:

mkdir /Volumes/Backup/backup.noindex
/usr/bin/rsync -avE --delete $EXCLUDES /Users /Volumes/Backup/backup.noindex

This is also handy when you need a temporary directory to compile an application but you don't want Spotlight to freak out with the temp files a compiler can create.



[ Reply to This | # ]
10.4: Avoid rsync errors during backup due to Spotlight
Authored by: CaptainQueeg on Nov 09, '05 08:39:30AM
I recently started using rsnapsot to do rsync backups from one Tiger box to another. It provides incremental backups without using too much disk space by using hard links. Working great so far, and not difficult to set up.

I'm glad I read your hint, since I didn't know that I needed to send the -E swith to have rsync include resource forks. Since I haven't been using the -E swith, I haven't run into the problem you described... yet.

My question - is there any need to turn off indexing on the source volume? Would turning off indexing on the backup volume (or directory) be sufficient?

[ Reply to This | # ]

10.4: Avoid rsync errors during backup due to Spotlight
Authored by: murali1080 on Nov 09, '05 01:05:58PM

Good hint, though I haven't had this problem. I have disabled indexing on my backup volume totally and permanently, anyway.

Also try rdiff-backup. Its in someways better and more mature than rsnapshot. The stable version is comparable to professional backup tools. I also like rsnapshot but do not use it anymore.



[ Reply to This | # ]
10.4: Avoid rsync errors during backup due to Spotlight
Authored by: ptone on Nov 11, '05 06:31:21AM
See this comment about rdiff not handling resource forks

[ Reply to This | # ]
10.4: Avoid rsync errors during backup due to Spotlight
Authored by: fragerybou on Nov 09, '05 03:22:42PM

I was using a custom made perl script to do essentially this same exact thing on my server. It's running every two hours on user home folders, providing snapshots of them should things go wrong. However, after letting this run for about a week, my server needed to be rebooted, and would not boot subsequently. I am nearly convinced that it's related to the snapshots and vast number of files, because after about two weeks of it being up and running snapshots, again it rebooted, never to come back up.

Be careful of how often you run snapshot backups, and how many files it will end up creating, especially on a boot volume.



[ Reply to This | # ]
excessive snapshots
Authored by: sjk on Nov 10, '05 12:18:44PM

Thanks for that cautionary warning.



[ Reply to This | # ]
10.4: Avoid rsync errors during backup due to Spotlight
Authored by: iwiring on Nov 10, '05 12:53:39PM

FWIW, the -E flag and "extended attributes" refers to metadata, not necessarily resource forks.

---
-dhan
---------------------
Dan Shoop, Internet Architect
iwiring@mac.com



[ Reply to This | # ]
10.4: Avoid rsync errors during backup due to Spotlight
Authored by: macshome on Nov 10, '05 01:15:53PM

Hi Dan,

From the man page.
<snip>
-E, --extended-attributes
Apple specific option to copy extended attributes, resource
forks, and ACLs. Requires at least Mac OS X 10.4 or suitably
patched rsync.
</snip>

Looks like forks to me.

Josh
www.afp548.com

---
http://www.afp548.com
Breaking my server to save yours.



[ Reply to This | # ]
10.4: Avoid rsync errors during backup due to Spotlight
Authored by: murat on Nov 10, '05 05:47:22PM

So I tried using rsync as illustrated in this hint to try to mirror a partition of my PowerBook to an external FireWire drive.

What I noticed is that rsync doesn't seem to preserve most modification dates on the mirror (!). This makes incremental updates impractical, since rsync will assume files with differing mod dates need to be updated.

A Google search turned this up, the upshot of which is that the Tiger version of rsync, while able to deal with resource forks (or more properly, "extended attributes"), does not properly preserve modification dates. In other words, rsync on Tiger has a bug that prevents its use in the way most people find useful.

Has anyone found a workaround?



[ Reply to This | # ]
10.4: Avoid rsync errors during backup due to Spotlight
Authored by: ptone on Nov 11, '05 06:24:14AM

Are you using 10.4.2 or 10.4.3 I've been hoping that this was fixed in .3 but haven't tested it yet.

-P



[ Reply to This | # ]
10.4: Avoid rsync errors during backup due to Spotlight
Authored by: mstoops on Nov 11, '05 06:45:10AM

I just tried this in 10.4.3:

$ rsync -av ~/bin /tmp

It seems to have preserved everything just fine.



[ Reply to This | # ]
10.4: Avoid rsync errors during backup due to Spotlight
Authored by: mstoops on Nov 11, '05 06:53:26AM

...and I just tried it across to a linux server. That too seems to have worked just fine, preserving date stamps (as well as a soft-link or two).



[ Reply to This | # ]
10.4: Avoid rsync errors during backup due to Spotlight
Authored by: murat on Nov 11, '05 02:04:51PM

Yes, if you omit the -E option, it works great. Include the -E option to see what I'm talking about.

I.e. try

rsync -avE ~/bin /tmp




[ Reply to This | # ]
10.4: Avoid rsync errors during backup due to Spotlight
Authored by: murat on Nov 11, '05 01:56:37PM

10.4.3



[ Reply to This | # ]
10.4: Avoid rsync errors during backup due to Spotlight
Authored by: BlackPenguin on Jan 16, '06 08:32:01AM
There are a number of known issues with the version of rsync shipped with OS 10.4.x. I'm hoping that they are resolved in future versions, but until then, I opted to build a patched version of rsync that fixes them. There are detailed instructions as well as patches here: Fixing rsync on Tiger.

In order to use these, you will need to have the Developer Tools installed and have little fear of the command line. The patches have worked well for me, and I currently employ rsync in my backup and synchronization strategy.

-- Nick

[ Reply to This | # ]
10.4: Avoid rsync errors during backup due to Spotlight
Authored by: curtinsmith on Nov 24, '05 05:58:57PM

I thought this might answer my problem but afraid it did not

I get the following error all the time when using rsync. Exmaple below

rsync: delete_one: unlink "/Users/admin/Desktop/test2/._macosxhints - Get the most from X!.webloc" failed: No such file or directory (2)
rsync error: some files could not be transferred (code 23) at /SourceCache/rsync/rsync-20/rsync/main.c(717)

Command : rsync -aE --delete ~/Desktop/test1/ ~/Desktop/test2/

It happens only when using the --delete option

To recreate
1) Make folder called test1 and test 2 on desktop
2) Put some files in both folders. Make sure folder 2 has a metta data file kind that folder 1 does not have? I think this is the error. I created this file by loading a webpage in safari and dragging the name in the title to the desktop that created this file
3) Run command rsync -aE --delete ~/Desktop/test1/ ~/Desktop/test2/

# What should happen is all contents in folder 2 that are not in folder 1 should be removed and contents of folder 1 copied to folder 2

It does this and ls -la shows that this has happened but always gives this error

NOTE# The error only happens if the file that is being deleted is not a plain file

Does anyone know of a solution or is it a bugg in Rsync?

Google search shows this problem but no solution

Using 10.4.3 and rsync version 2.6.3 protocol version 28



[ Reply to This | # ]