Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

Possibly speed copies via Terminal System
I was in a hurry trying to back up my iPhoto Library to an external Firewire 400 drive, and was disappointed to see that the Finder was going to take almost an hour to back up all 11GB of pictures I had. I went into Activity Monitor to see just how slowly the Finder was going, and found it was copying at 8MB/sec, with the Finder taking 60% of my PowerBook G4's CPU. So I canceled the copy, opened up Terminal.app, and executed the following command:
cp -r ~/Pictures/iPhoto Library/ /Volumes/myexternaldrive/
Activity Monitor showed cp using only 20% CPU, and maxing out my drives at 23MB/sec! What would have taken an eternity in the Finder took only 15 minutes, thanks to Terminal.

NOTE: If you need to copy files with resource forks, install the Developer Tools and use CpMac (in /Developer » Tools) instead of cp.

[robg adds: On a faster machine, I'm not sure there will be much difference in the speed -- I did a quick test on the Mac Pro (using a 3.22GB folder with a few thousand files spread through 160 folders), and both methods took about the same amount of time. In addition to the resource fork issue, note that cp won't copy OS X alias files; it just throws an error message during the copy.

ditto (included with OS X), however, seems to copy them just fine, and handles resource forks automatically. So I tested again with ditto instead of cp. Whereas the time gap with cp wasn't really measurable, ditto was about 15 seconds quicker than the Finder on my 3.2GB test copy. So if I were copying a huge folder, I'd probably try ditto, as it could save a few minutes of copying time.]
    •    
  • Currently 3.25 / 5
  You rated: 5 / 5 (4 votes cast)
 
[21,034 views]  

Possibly speed copies via Terminal | 15 comments | Create New Account
Click here to return to the 'Possibly speed copies via Terminal' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Possibly speed copies via Terminal
Authored by: fds on Jun 05, '07 07:44:53AM

That's weird -- plain vanilla "cp" is supposed to be able to copy resource forks and all Mac-specific data just fine, ever since 10.4.0.



[ Reply to This | # ]
Possibly speed copies via Terminal
Authored by: vykor on Jun 05, '07 08:17:59AM

CpMac under 10.4 is deprecated (see man CpMac, if you have your mandirs set up correctly). OS X cp was modified to be fork-aware and preservative of extended attributes. Unless you managed to shadow/overwrite OS X cp with, say, GNU coreutils or plain BSD variants, of course.

However, cp may still have a number of gotchas. ditto is probably preferable if you're making copies for backup purposes.



[ Reply to This | # ]
Possibly speed copies via Terminal
Authored by: LC on Jun 08, '07 03:54:07PM
ditto --rsrc
right? (I always use the --rsrc option;)

[ Reply to This | # ]
Possibly speed copies via Terminal
Authored by: kikjou on Jun 05, '07 09:15:05AM

None of the files in the iPhoto Library contains a resource fork, so you should not be concerened about losing information when doing a cp. Also, as pointed out, cp does do most of the stuff that a Finder copy does, including resource forks.
Another time when I do not use the Finder copy is when copying over a network that uses DSL or, God forbid!, dial-up. The Finder does not adapt gracefully to varying speeds, and it tends to slow down or even bring down the entire computer. In this case I use scp.



[ Reply to This | # ]
Possibly speed copies via Terminal
Authored by: bitwise on Jun 05, '07 05:57:09PM

if you've unchecked iphoto's preference to "copy files to iphoto library folder when adding to library", aliases are created instead, and they are all resource fork

before cp was updated, i got familiar with ditto and have never gone back



[ Reply to This | # ]
Possibly speed copies via Terminal
Authored by: teece on Jun 05, '07 09:19:26AM

I would do cp -pr.

The 'p' tells cp to preserve as much (Unix) file metadata as it can. File mod and access times are the ones I'd really care about there.

cp won't copy hard links correctly -- but your iPhoto library shouldn't have any.

It's supposed to copy resource forks now, as others have said. And CpMac and MvMac are deprecated, as mentioned above.

Frustratingly, very few tools will actually make a perfect copy of all Mac metadata. cp is not one of those few tools. I don't think it will matter for the iPhoto library, though. (ditto and CpMar aren't perfect, either. Actually, the only one I'm aware of that is perfect is SuperDuper! In most cases, they'll all be fine, but everything does lose some metadata except SuperDuper!).



[ Reply to This | # ]
Possibly speed copies via Terminal
Authored by: vocaro on Jun 05, '07 09:33:42AM

Funny, I'm just the opposite: I always use Finder (or Path Finder) to copy files instead of the Terminal. In my experience, cp tends to consume so much of the CPU and disk that it slows everything else in my system, causing temporary lock-ups until the copy process ends. In contrast, Finder (and Path Finder) treat system resources much more fairly, and I'm able to continue to use my computer as the files are being copied.



[ Reply to This | # ]
Possibly speed copies via Terminal
Authored by: porkchop_d_clown on Jun 05, '07 09:48:32AM

For network copies, I've often seen huge speed boosts when using cp rather than Finder. NFS mounts also seem to be a lot faster than SMB ones, FWIW.

---
Everyone loves a clown, but no one will lend him money!



[ Reply to This | # ]
Possibly speed copies via Terminal
Authored by: gdsimms on Jun 05, '07 10:35:31AM
To preserve resource forks and ownership/permissions, I use

tar -cf - /Path/to/original | ( cd /Path/to/destination; tar -xvf - )

This is sometimes called a "tar rainbow," which is one reason I find this method so endearing. :-)

[ Reply to This | # ]

Possibly speed copies via Terminal
Authored by: GlowingApple on Jun 05, '07 01:04:02PM
And, if bandwidth is slower than the hard drive read/write times, such as internet copies or even slow LAN copies (e.g. old 10Mbps nets, a/b and maybe even g wireless copies), you can easily add compression with that rainbow:
tar -jcf - /Path/to/original | ( cd /Path/to/destination; tar -jxvf - )
Though this point is moot if your computer is slow.

---
~Jayson <www.kempinger.us>

[ Reply to This | # ]

Possibly speed copies via Terminal
Authored by: inik on Jun 05, '07 11:51:33AM

cp, along with all other terminal utilities that have been upgraded by Apple to preserve resource forks, will destroy other metadata associated with your files. A Finder copy remains the one and only Apple-supplied method that will copy permissions, extended attributes, BSD flags, Spotlight comments, and the Finder creation date.

Also, if you're copying a file on a single drive, you can make a hard link to the file rather than copy it (not exactly the same thing as a copy -- it's a second instance of the file in another location) and it's nearly instantaneous.



[ Reply to This | # ]
Possibly speed copies via Terminal
Authored by: grumpy on Jun 05, '07 05:02:20PM
Not sure whether Finder effectively uses 'ditto' or not to the the hard work, but using 'ditto' isn't much better than using the Finder. The specific reason for this, and why Finder is also probably slow, is that 'ditto' will by default use something called the 'Mac OS X Unified Buffer Cache'. From my experience this just slows it down when used on lots of large files. One is better turning it off. Thus, use:
  ditto --nocache ...
The documentation from manual page for this option is:
     --nocache     do not perform copies using the Mac OS X Unified Buffer
                   Cache. Files read and written will not be cached, although
                   if the file is already present in the cache, the cached
                   information will be used.
I found the same issue when wanting to copy a few hundred GBs worth of iMove files from one firewire drive to another. Didn't trust 'cp' because of resource fork issue.

[ Reply to This | # ]
Possibly speed copies via Terminal
Authored by: jmontana on Jun 06, '07 05:10:21AM

How old is the Mac? I know on my G4/400, anything which displays a progress bar will suck the life out of the CPU just to show the progress bar animation. What happens if you hide the Finder? That might speed up the copy, as it won't have to redraw the window constantly.



[ Reply to This | # ]
Possibly speed copies via Terminal
Authored by: ikioi on Jun 09, '07 10:01:28PM
I recently tested various backup tools on mac os X 10.4.9 (Intel) and I found that "cp -Rp" preserves:
  • ACLs
  • Resource Forks
  • HFS+ extended attributes (new arbitrary name value pairs)
  • BSD flags (for example "uchg")
  • all of the Carbon CatalogInfo that I tested (Creator, Type, Label, Finder Flags, and all dates)
  • Most Unix lstat() info (UID, GID, perms, etc)
The only things cp -Rp don't preserve are:
  • hard links (two hard links will be copied as independent files
  • UID and GID of a symlink when they differ from the UID and GID of the target of the link.
  • "inode" number (aka file id or folder id) , but nothing short of dd or ASR in device mode can preserve this.
All in all, unless you want to use ASR/device mode (which is a lot like ghost for windows, and sort of like dd for UNIX ) then SuperDuper and cp are the two most faithful ways of backing up files and their metadata on Mac HFS+ drives. These two did much better than CCC, ditto, ASR/file mode, Synk, hfsrsync, RsyncX, etc, in my tests.

[ Reply to This | # ]
Possibly speed copies via Terminal
Authored by: ikioi on Jun 09, '07 10:09:17PM
Correction: When I said "Carbon CatalogInfo" , I meant "HFS CatalogInfo"

[ Reply to This | # ]