Huh? The rsync version of this is actually shorter/simplier (and it copies your changed files as well).

rsync -a /Volumes/LocalUSB/Photos /Volumes/RemoteUSB/Photos

Not really that mind-boggling...

[ Reply to This | # ]

really? What happens if you append or omit the trailing slashes? It can have a drastic effect on your backup result. Just one example for mind-boggling with "simple" unix commands.

[ Reply to This | # ]

Actually, rsync will fail if it hits certain types of files. For example, it can't backup a local iDisk volume because it can't cope with the symbolic links. I'm going to try the cp method the author suggests and hope it works better than rsync.

[ Reply to This | # ]

That's great; I was unaware of the "-n" flag. As a longtime UNIX user, I've been using complex "rsync" command lines.

[ Reply to This | # ]

You can also do cp -nav "/Volumes/LocalUSB/Photos/" "/Volumes/RemoteUSB/Photos/". cp doesn't always do the right thing with links. tar is a little better, you can do something like cd "/Volumes/LocalUSB/Photos" ; tar cf - . | tar xpkvf - -C "/Volumes/RemoteUSB/Photos" . The best option, though, is probably rsync: rsync -av "/Volumes/LocalUSB/Photos/" "/Volumes/RemoteUSB/Photos/". You have to be careful with the trailing slashes in rsync, but other than that it's just as easy as cp.

[ Reply to This | # ]

To use rsync it is actually very simple for local files.

All you need is:
rsync -av /source/path/to/directory/ /destination/path/to/directory/

As long as you leave trailing slashes on both directories it will mirror all the files from the source to the destination and will copy only what isn't already there.

[ Reply to This | # ]

Important limitation: cp has no verification option. Also, modified files won't be copied at all.

[ Reply to This | # ]

i love the cp command.
i use it all the time to copy files from dying hard drives.
a finder copy will abort at the first unreadable file that it finds.
a cp command in terminal will register an Input/output error and move on.
it also copies files that are invisible in the finder.
if there is only one terminal command you ever learn, it should be cp.

[ Reply to This | # ]

You mention rsync as an alternative, and I've got to highly recommend that you use it instead of cp. The syntax isn't much worse than your cp command, and it is a much more robust and reliable program. You do not need more robustness for your purposes, but if one wanted to extend this solution to a different situation, they may hit a wall with cp. I say that it is more reliable because you can set rsync to check that files were correctly copied, I'm not sure if cp can recover from an unlikely copy error.

Also, you can ensure that everything with working properly with the "--dry-run" option (along with "--verbose" to see everything that's happening), so you can perfect the command without worrying about file loss. Further, the "--backup" option will never delete files on the source machine.

[ Reply to This | # ]

Does your method copy updated files? Rsync's syntax is as simple as:

rsync -av --progress /source/ /destination/

a is archive
v is verbose
--progress gives more info on the status of each file copy.

A trailing slash on the /source/ will copy the contents of /source to /destination.. Omitting the slash will copy the entire directory to /destination.

[ Reply to This | # ]

The only problem with your solution is that it won't recognize that a source file has changed, which makes it a bad backup solution for files that may be edited.

the rsync command you are looking for would look like this:

rsync -a -E -v /Volumes/LocalUSB/Photos/ /Volumes/RemoteUSB/Photos/

If you're afraid that you'll blow something up with rsync, remember the "--dry-run" option which will output what the command would do, without actually changing any files.

You might also consider adding the "-P" command which would keep partial files in the event of a failure (not very useful) but also shows the progress of each copy as it happens. I wouldn't use this option in a cron job though

[ Reply to This | # ]

I think cp with these options will overlook changed files, my favorite for stuff like this is rsync. Here's my cheatsheet for rsync, read the man page for it though...

rsync -avhP source destination

a=archive,
v=verbose,
h=human readable,
P=save intermediate Partial files (--partial) and give you Progress (--progress)

A trailing slash on the source changes this behavior to avoid creating an additional directory level at the destination. You can think of a trailing / on a source as meaning "copy the contents of this directory" as opposed to "copy the directory by name", but in both cases the attributes of the containing directory are transferred to the containing directory on the destination. In other words, each of the following commands copies the files in the same way, including their setting of the attributes of /dest/foo:

rsync -avhP /src/foo /dest

copies "foo" itself to /dest directory creating a foo subdirectory

rsync -avhP /src/foo/ /dest/foo

copies "foo/*" to /dest directory implicitely forcing the creation of "foo" at the destination

rsync -avhP /src/foo/ /dest

puts the contents of foo into /dest with no "foo" directory

[ Reply to This | # ]

I prefer rsync.

rsync -avE /source/dir/ /backup/dir/

Maybe it's because I want changed files to be copied instead of just new files.

Throw a --delete in there incase you want to remove stuff that has been removed on the source:

rsync -avE --delete /source/dir/ /backup/dir/

The above is the built in rsync. If you are using the rsync from macports, like me, it would be:

rsync -avX --delete /source/dir/ /backup/dir/

The 10.6 installed version is 2.6.9. The macports version (that I have installed...) is 3.0.6.

Adam.

[ Reply to This | # ]

I'd use rsync instead just in case you modify the originals down the line:

cp -npRv "/Volumes/LocalUSB/Photos/" "/Volumes/RemoteUSB/Photos/"

becomes

rsync --progress -av /Volumes/LocalUSB/Photos /Volumes/RemoteUSB

--progress shows you progress as it runs, -av enables all the options for backups. If you add in --delete it will remove files that you deleted from the source directory. Rsync will also work remotely over ssh if need be. Though you don't necessarily need it, the latest version of rsync is even better and you can install that with macports.

[ Reply to This | # ]

For the Mac OS X bundled rsync 2.6.9 you can use the following line to achieve the same, but with the extra speed of delta copy. I've included the long options to make the commands easier to comprehend.

/usr/bin/rsync --recursive --perms --executability --owner --group --times --progress --extended-attributes "source" "target"

If you have rsync 3 from mlbackup installed you can use a slight variation which is required for the much superior version of rsync included there.

/usr/local/maclemon/bin/rsync --recursive --perms --executability --acls --xattrs --owner --group --times --progress "source" "target"

This will also preserve ACLs and all extended attributes, forks etc.

You can also use mlbackup to achieve the same by creating a simple 7 line (max.) config file and have delta copies, backup rotation, growl and email notifications.

mlbackup was also featured in another hint and is available for free, licensed under GPL.

Full and fair disclosure: I am the author of mlbackup and I did write it for a very similar need, namely to backup files from one disk to another.

[ Reply to This | # ]

First, thank you for mlbackup.
Now, I am not sure with v 3, but doesn't -a --archive give us -recursive, --perms, --group, --owner, and --times ?
Then we only need to add -E in (for --extended-attributes) in 2.6 and I suppose --acls in 3.0?
I am leaving off --progess, since I usually automate the backup process and don't watch it.

[ Reply to This | # ]

According to man rsync:
-a, --archive archive mode; same as -rlptgoD (no -H)

I've just used the long options to make it easier to read and to better give context with the given options for cp.


A correctly patched rsync3 is able to successfully pass all the backup bouncer tests which include ACLs, POSIX permisions, forks, extended-attributes, etc which is not possible with cp or the Apple provided rsync. See the README that comes with mlbackup for a more elaborate explanation on the matter.

rsync 3 also gives you much better speed when counting and evaluating files. This is mostly experienced on data sets with large numbers of files, not necessarily huge amounts of data. rsync 3 is the only way to get an exact copy of your source files without having to unmount the source volume or resort to block copy methods which lack some granularity. :-)

[ Reply to This | # ]

Is cp 'touching' the files as it goes along your disk or does the original timestamp stay in place?

touch = Change file timestamps, change the access and/or modification times of the specified files

[ Reply to This | # ]

Sorry, but you say that will only copy "new" files, but that's not right, as there isn't any date checking, only filename checking. So any files of the same filename but newer date wont be copied over.

[ Reply to This | # ]

This is exactly what rsync is made for and I'd wager it would be faster than cp on this. The rsync command you'd want if you want to keep them exactly in sync is:

rsync -avrze --delete "/Volumes/LocalUSB/Photos/" "/Volumes/RemoteUSB/Photos/"

If you just want to move the new ones, but don't care about deleting off the remote usb files that have been deleted from local usb, you'd just remove the --delete option.

[ Reply to This | # ]

actually, rsync is slower than cp. The fastest way is to use cp combined with 'tar'

[ Reply to This | # ]

Notice the OP's own comment:

"My photos are in RAW format (specifically DNG files) and will never change, so I only need to concern myself with new files."

So he doesn't really care about rsync's ability to include changed files.

[ Reply to This | # ]

Why does this qualify as a hint??
Where is the quality behind selecting what becomes a hint and what not. If there *is* anything, that does *not* become a hint anymore at all.

Jeezus. There was a time when this site was the paramount of all things mac os x hints.
But nowadays...

Maybe I should also start posting hints. Like, you know, I just saw that by clicking on the apple logo in the top left, you can shut down the computer. Awesome, dude! I always thought just pushing the on/off button would do that... Wow. Who would've known. Hope I could help others.

*yawn*

[ Reply to This | # ]

Let's count the number of people who have memorized all of Unix. Let's count the number of people who have not memorized all of Unix. If the difference between the two is greater than zero, then Unix tips may help someone do something they didn't know how to do.

If you don't like a hint, don't use it. If you have a better way of doing something, submit it. If you just want to complain without adding any value, I'm not sure I see the point.

-rob.

[ Reply to This | # ]

I am complaining about the quality of this hint and the quality of hints in general on MacOSXHints.com.
I have voiced my opinion here, at a good example of a cheap "hint" in the hope to find others that think like me.

If this is of no value to you, I'm not surprised. It's your site, you are making money off it, you are certainly interested in a certain volume of hints a day/week/whatever to keep people and advertisers coming. And maybe "cp" was - next to another 1064 commands in /usr/bin - actually a new command to you.

I do agree that it is very hard to draw a line between what counts as a hint and what not. Others have already tried in the past and when "cp" counts as a hint, they've failed. I am most certainly not able to draw the line, as this is and always be subjective. So the "line" should be one the community draws. It is still open how that could happen. But if nobody does anything, things won't get better. If you like that or not.

MacOSXHints.com offers the option to rate hints. This is a very good tool.
So I started to vote for good hints and vote less for trivial "hints". And I am trying to make people aware that not everything they perceive as a hint really is worth blogging about.
I wish more people would speak out, so as to learn their opinion. If the majority really thinks that "cp" is a hint, well, then I would not say anything about it anymore. Right now I am feeling this site gets swamped by trivial posts, and I would like MacOSXHints.com to stay a quality site.

Your formula is way too easy. The reasons for that I don't But I see now why "cp" counts as a "hint" and "cp -H" will be the next big thing.

I am sorry if I said something you don't like. But it will not get better by standing by and shaking the head. You can always delete my posts, send me a pm or move it to some place you seem more appropriate.

[ Reply to This | # ]

How short-sighted. This hint solved a problem for me that I've been wrestling with for a week.

I only wish it would have been posted 7 days ago ;-)

[ Reply to This | # ]

I would agree that this hint is somewhat trivial but dont forget there are many Mac Users who never touch the Unix part of OSX so a simple "cp" or "rsync" can be mind-boggling as man-pages are often confusing and cumbersome to read.

I rather dislike those hints where someone writes a super-duper shell/perl.. script to solve a very specific transformation problem from Programm A to B (but only version 1.xx and not 2..) which only 10 other users out there with exactly the same config.

Or the 555.hint how to change the iTunes/Mail buttons/windows, yawn..

There is simply no way to "censor" the quality of a hint so just read or ignore..

[ Reply to This | # ]

I welcome complaints, but really, they have no value as part of the thread of comments on the hint itself: they just confuse users who are looking for discussion on the hint and various alternatives.

For feedback, please feel free to email me directly (there's a link on every page), or even better, start a public discussion on it on the forum site, forums.macosxhints.com. Feel free to submit "Why macosxhints.com sucks" or whatever you want to call it, and describe why you think it sucks, and what you would do to make it better, and ask others for their thoughts.

*That* would be useful. Posting an off-topic rant about why a given hint sucks won't do a thing to change anything in the long run. First, there's no way I can follow every thread in every hint to see the general opinion, and second, it spreads that opinion out over a large base, so it's hard to tell whether complaints about the hints are an exception or a trend.

I long ago gave up trying to make everyone happy; life's too short for that. I know that any given hint will help some people, not help others, and genuinely piss off some other group of people.

And no, I don't make money based on hint volume. I'm on salary. Never in my time with Macworld have I been told how many hints to publish, or that my ad revenues were down (or up) and that my pay would therefore drop (or increase). So there's no "pressure" to publish a certain number of hints per day. I publish things I think at least some number of people may find interesting -- if you grossly disagree with my approach, another alternative is to launch your own site, focused only on those hints you find hint-worthy.

But really, if you're just going to complain about the hints, please either do it to me directly, or via the forum site. Posting complaints in the hint thread does nothing but make the hint harder to use for those trying to read through it.

As for deleting your account/postings, that's not my style. I've only deleted one account here that I can recall, and that was because the user physically threatened to harm myself and my family. If you use foul language, I may edit your post; if you're a spammer, I'll delete your post. Other than that, I leave things alone.

-rob.

[ Reply to This | # ]

Okay. I see your point. Sorry to have voiced my opinion in the wrong place.
I'll now unsubscribe this thread as I assume everything has been said and insults like "arrogance" or whatever else imbecile members try to phrase won't help this at all.

[ Reply to This | # ]

What arrogance. This hint generated lots of posts so it's of interest to ... well, lots of good folks. Then you attack the site itself. OMG, go find a site that meets all your needs. Oh wait, none exist. Some people just complain, some people try and help. How about you switching sides and try and help someone...anyone.

[ Reply to This | # ]

I'd say the 'cp' command is indeed perfect for that. It's funny how many people don't read the requirements (the dng's will never change) and over-engineer their solutions (and don't check existing comments).

While I agree that rsync is way better for keeping things in sync, being lazy on the command line and would probably type the shortest possible command, something like:

> cp -Rnpv /Volumes/LocalUSB/Photos /Volumes/RemoteUSB

this will simply copy the local Photos folder itself (with all its contents) to RemoteUSB, no point worrying about all the end-slashes, and double quotes aren't necessary if the path doesn't contain spaces (although in a shell script it's important to always enclose the paths between quotes)


Note about the RAW's: Even if you want to keep the originals untouched, you might want to add or change some of the embedded EXIF, IPTC or XMP metadata in order to better manage your digital image collection. There's even an awesome command-line tool for that: exiftool (http://www.sno.phy.queensu.ca/~phil/exiftool/). Of course, then you will have to use rsync ;)

hope this helps
JM

[ Reply to This | # ]

Some of you aren't understanding my situation. My photos are RAW files. Those files never change. It's one of the major points about RAW. So my requirement is far simpler than many people are assuming below. The process for copying is best illustrated thus:

"Is this file already on the target volume?"

If the answer is yes, ignore it. Otherwise copy it. That's it!

The problem with rsync is that it is capable of doing stuff to my source directory. Given the documentation is not easy to understand (for me at least), I opted for a command that I could be sure would not touch my source directory. Get your options wrong with rsync and you could end up deleting original files.

As for speed, it was whipping through all my 2009 photos in well under a minute before copying the new stuff. The speed of my network was the bottleneck, not the speed of the command.

So in summary. cp meets my requirements. Simple is often best. If someone would care to write *clear* documentation on rsync I might find a use for it. (In fact, I do use aRsync for another task, but that was also a voyage of fear.)

[ Reply to This | # ]

@zkarj:

The problem with rsync is that it is capable of doing stuff to my source directory. Given the documentation is not easy to understand (for me at least), I opted for a command that I could be sure would not touch my source directory. Get your options wrong with rsync and you could end up deleting original files.

The first presumption that cp gives is "I'm not endangering my source files". It copies from a to b. Period.

Sorry to join the mob but, I think the fear factor there is a bit exaggerated. There is only one rsync "option" which behaves that way... and it requires the user to type the following twenty one c.h.a.r.a.c.t.e.r.s:

--remove-source-files

Nothing ambiguous or mystical about that. So long as that exact string is not used... there's nothing to fear (as far as the source files are concerned).

Other than those comments, i thought your hint was fine (in the context of RAW files).

-HI-

[ Reply to This | # ]

Not if you already know that fact. (And now I do, thank you.)

Once again - come at this with NO prior knowledge of rsync and its syntax. I honestly had no idea it existed before I tried to solve this problem. It's touted as a two-way syncing tool. Unless I understand it, I'm not trusting it to leave my source files alone.

I think you'd agree the prospect of deleting your only copy of a photo while intending to back it up is the definition of a pretty bad day.

[ Reply to This | # ]

@zkarj:

It's touted as a two-way syncing tool.

Two-way syncing tool?!? No... it's not. And this is the first time i've heard that particular misinformaton being touted. (if anything, people sometimes complain that rsync *doesn't* do bidirectional syncs). Perhaps you're thinking of of Unison.

Unless I understand it, I'm not trusting it to leave my source files alone. I think you'd agree the prospect of deleting your only copy of a photo while intending to back it up is the definition of a pretty bad day.

Folks are far more likely to accidently delete stuff by not properly escaping\ a\ space\ in some pathname, than risk their source files when running rsync (and somehow adding the --remove-source-files option by "mistake").

[ Reply to This | # ]

When I want a fried ham and cheese sandwich, I usually put a frying pan on the stove, fry the ham and let the cheese melt over it right before it's done.

Now, I could put the ham and the cheese in the microwave oven and probably get a similar result, but sometimes the "right tool for the right job" is the way to go.

Also, I find it hard to believe that the original poster had trouble with rsync's very simple local-file-copy syntax but he was able to come up with that multi-flag monstrosity in cp.

[ Reply to This | # ]

Well, believe. The first presumption that cp gives is "I'm not endangering my source files". It copies from a to b. Period. There was some testing done to perfect the flags, sure. And I admit to not realising the --dry-run option of rsync.

But all of you rsync fans have all been using it and have learnt its features. Try approaching it from the point of view of someone who didn't know it existed until they had these precious files to back up. I don't have access to a *nix command line at the moment, but I remember reading through the man pages and then searching online and getting the typical *nix expectation that I knew what I was doing and was just trying to figure out some nuance.

I've written my fair share of documentation in my 20+ year IT career and had to deal with the users that read it. I've used my fair share of technology too. I know when I understand something and I know when I don't. I know how horribly things can go wrong if I don't understand. The rsync documentation is ambiguous in places and/or uses terms without defining them. I could find no web resource that gave me any greater level of understanding. Actually, the same is true for the cp command. Because it is a 'basic' *nix command, everyone assumes you either know it inside out or you won't use it. A rather odd approach, really.

In this hint, I gave an explicit use case and an exact syntax that works so that others may find in their searching what I couldn't.

[ Reply to This | # ]

I only kid. :-) But please accept my apology, as I see you are getting quite a bit of flak for posting this hint. I didn't mean to insult you personally.

I was just trying to point out that while your usage of cp accomplishes what you are looking to do, you've got to understand that the vast majority of people here are going to say it's the wrong way to do the job. Once again, don't take it as a personal insult, but I find it exceptionally hard to believe that you couldn't find a decent rsync tutorial. A Google search for 'rsync copy local directory' should provide you with tons of useful links. I know it will, because it did for me!

I use rsync extensively both at home and on the job, but I only tap the bare minimum of its capabilities. As several people have posted here already, the rsync equivalent of what you are doing is almost like your cp command, but with less flags. Granted, this isn't the old days of time-sharing on the VAX when using an extra process when you didn't need to got you in trouble, but following those old practices are still a good idea, because they are effective and efficient. rsync in particular is one of the most efficient apps out there.

I rsync a 100 GB iTunes Library every night--maintaining metadata like playcounts and ratings--and it runs in less than a minute from one FireWire 400 drive to a USB2 drive on my Mac mini. I think it's things like this that have caused people to get so up-in-arms about pushing you toward rsync. It's an amazing app, and is really what you're looking for here. If you really couldn't find the answer by Googling, you could have posted in the forums here and got a solid answer back PDQ.

You say you have 20+ years in IT. I'm not far behind you. So as a fellow IT professional, I'm sure you've ran into some scenarios like this: At my company, we provide both Excel and FileMaker Pro. There is some cross-over in functionality, but we've got so many folks who just want to force Excel into being a database, and FMP into being a spreadsheet app. We get the Adobe Creative Suite for our marketing staff. They've got both Photoshop and Illustrator in this suite. Yet they often want to draw in Photoshop, and edit in Illustrator. In both cases, yeah you can probably do what you want to do my using the "wrong" app, but it's just much easier if you use the right tool for the right job.

Once again, no personal offense intended, and I actually learned something about cp that I didn't know before, but you've got to understand that *most* of the people here just want to help, and I think in the long run you'll be happy with learning rsync. Like I said, I just use a minimum of its capabilities, but every new thing I learn is a complete joy. Take the time, ask for help, find the solution. It will be worth it.

[ Reply to This | # ]

Thanks for the thoughtful reply. I understand and appreciate your comments. As always, knowing exactly what to search for is the key. I had no idea, until now, that "local" had special meaning in rsync terms. :-)

[ Reply to This | # ]

Could you please post as a hint your rsync solution to reconciling iTunes Libraries? It would be useful to know if you are, in fact, reconciling the songs/playcounts/ratings/playlists/etc. across multiple computers, or merely synchronizing the contents of the "iTunes Music" folders. In other words, suppose the metadata for song #1 was changed on computer A and for song #2 on computer B. Would your rsync command reflect those changes on both computers?

It also would be useful to know both the version of iTunes you are using and the version of rsync. Thanks!

[ Reply to This | # ]

Ok, to all those who prefer rsync to cp, I give you one reason why cp is better at this job: it's easier to understand. Why? Ok, on my very large terminal window cp has a man page of two pages. rsync in contrast has a man page of 38 pages! 38 pages!!! Who's gonna read all that? If I had to read all that, just to copy some files, you'd better not cross my way in one of those smug RTFM T-shirts, cause I gonna beat the hell outta you.

[ Reply to This | # ]

A further, and probably final comment. After re-reading most of the comments I feel I've learnt a lot more about rsync. But please note this.

While most of the offered rsync solutions will do the job, there are several different variations to the options and even two different versions of rsync itself, offering further variations. rsync has a whopping 106 options on it on the version I have (the built in), probably substantially more in the newer version.

I assume cp has been around with most of its 11 options since time immemorial and I doubt (though I don't know for fact) that there are different flavours of it. I stand by my claim that I chose the simple option.

That said, now that I am beginning to understand rsync, I see a future for it on my system.

[ Reply to This | # ]

Being a photographer, I see a BIG problem with the cp solution:
You assume that no raw file will ever have the same filename as an already existing one.
That is a VERY DANGEROUS assumtion.
Digital cameras have a limited number of filenames, and then starts to reuse them.
Those new files will not be copied, and you will never know...
rsync is obviously the way to go.
:)

[ Reply to This | # ]

Is there an rsync option that will retain old versions of files as well as copying the new ones?

I had a quick look at the man page and couldn't find one, but I know that, for example, Carbon Copy Cloner, which provides a GUI interface for configuring rsync clones of volumes etc. has the option to move aside and retain files that are to be replaced. This feature—or an alternative that incrementally appends something to the name of copies in the destination folder—would be essential to solving the problem you've identified where files with the same name are actually different pictures both of which should be retained.

[ Reply to This | # ]

I've read this hint and comments with interest, and I appreciate the effort our original poster, and our commenters, have put into them.

There's so much detail that I wish for an article that summarizes one or two good solutions, with explanations that nail down a good practice for Mac OSX users. It's great to learn about options that a macports user can install, to be sure. I'd like to know about the cp or rsync options I should use on an unaltered installation of Leopard or Snow Leopard. I come away from this hint & comments feeling a bit overwhelmed by the debates and options.

[ Reply to This | # ]

While apparently everyone has already said this, I think this hint is a horrible idea. Using cp is fine for copying, but backup is such a different beast. Spending a few minutes to understand rsync or better, "unison" (http://www.cis.upenn.edu/~bcpierce/unison/)

cp is simply completely inappropriate for backups, no matter what switches you use.

[ Reply to This | # ]

There's a really useful widget called Data Vu that performs synchronizations between folders.
It's powered by rsync so you get all the benefits mentioned above but the interface is incredibly simple so all you people who don't want to deal with the complexities of the command line don't have to!

http://www.apaulodesign.com/widgets/data-vu.php

(It's currently in a stable beta, but I happen to know that the full version is coming out soon).

[ Reply to This | # ]