Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

A caution on using command-line zip for backups UNIX
This is a warning as well as a hint. Control-clicking on a file or folder brings up a contextual menu that includes the option to Create archive of "xxx". The created archive file is a zip file that can be unzipped with /usr/bin/unzip. So, one would think that OS X is using /usr/bin/zip to create the archive. But that isn't so. It seems that OS X creates the zip file using code internal to Finder. It seems (from output of ps or top) not to be using /usr/bin/zip. Now the warning: /usr/bin/zip is broken on HFS+ file systems. It is broken in two ways:
  1. By default, it does not preserve the resource fork of files that it archives.There seems to be no way to force it to preserve resource forks. (Has anyone found a way?)
  2. Its behaviour does not correspond to its man page: /usr/share/man/man1/zip.1
Those (like me) who want to automate backups of their system using command line tools should perhaps use /usr/bin/tar. The tar utility now (as of OS X 10.4.8 and perhaps earlier) preserves resource forks. It stores the resource fork of foo as ._foo, and then recombines the data and resource forks when untarring to an HFS+ volume. To archive and compress a folder foo and all of its subfolders, one can do:
$ /usr/bin/tar cjvf foo.tbz foo
And to restore it, one can do:
/usr/bin/tar xjvf foo.tbz
This seems to preserve the resource forks correctly. For other methods, maybe see the hint Easily create HFS-aware PKZIP and unix archives.
    •    
  • Currently 2.60 / 5
  You rated: 5 / 5 (5 votes cast)
 
[28,399 views]  

A caution on using command-line zip for backups | 21 comments | Create New Account
Click here to return to the 'A caution on using command-line zip for backups' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
A caution on using zip for backups
Authored by: jsumners on Nov 30, '06 07:55:29AM

Zip is not meant for backup purposes. It is meant to compress files for transfer. TAR (Tape ARchiver) is specifically designed for backing-up data/filesystems.

http://www.gnu.org/software/tar/



[ Reply to This | # ]
A caution on using zip for backups
Authored by: MtnBiker on Nov 30, '06 08:38:10AM

Apple doesn't seem to think so as they label the file by default Archive.zip.

---
Hermosa Beach, CA USA



[ Reply to This | # ]
A caution on using zip for backups
Authored by: gslusher on Dec 04, '06 03:50:41PM

The fact that they label the file "Archive.zip" has nothing to do with "archiving" data. In compression systems, an "archive" means that it contains more than one file or folder. In fact, the "default" name used by the Finder if you archive ONE file or folder is the file or folder's name with .zip added. The only time you'll see "Archive.zip" is if you zip more than one thing at a time.



[ Reply to This | # ]
A caution on using TAR for backups
Authored by: hypert on Nov 30, '06 10:52:26AM

I made the mistake of tar'ing up some old Classic System Folders w/ tar, and everything got corrupted.

Finder's Archive, ditto, and rsyncx are now my tools.



[ Reply to This | # ]
A caution on using zip for backups
Authored by: adrianm on Nov 30, '06 07:59:16AM
There is no all-in-one solution for correctly handling files' resource forks, meta data, etc, but ditto (probably in /usr/bin) is a good one to look at.

Check out the man page.

---
~/.sig: not found

[ Reply to This | # ]

A caution on using zip for backups
Authored by: skrawcke on Nov 30, '06 08:03:14AM

and it isn't the finder doing the .zip it's
/System/Library/CoreServices/BOMArchiveHelper.app

and it uses gzip.



[ Reply to This | # ]
A caution on using zip for backups
Authored by: ojohns on Nov 30, '06 11:10:45AM

OS X does use BOMArchiveHelper.app to *unzip* archived files. But I see no evidence that it uses either gzip or that app to *create* its zip archives. When OS X is in the process of creating a large archive, both ps and top show nothing relevant running except Finder itself. And,

nm /System/Library/CoreServices/Finder.app/Contents/MacOS/Finder | grep zip

nm /System/Library/CoreServices/Finder.app/Contents/MacOS/Finder | grep BOMArchive

both give empty results. This is why I surmised that OS X *creates* the zip archive using code internal to Finder itself.

---
ODJ



[ Reply to This | # ]
A caution on using zip for backups
Authored by: boredzo on Nov 30, '06 08:30:56AM

How does zip(1) deviate from its manpage?



[ Reply to This | # ]
A caution on using zip for backups
Authored by: ojohns on Nov 30, '06 11:20:20AM

For example, the man page for zip has the entry:

-df [MacOS] Include only data-fork of files zipped into the archive.
Good for exporting files to foreign operating-systems.
Resource-forks will be ignored at all.

but if you use that switch, you get:

zip -df foo
zip error: Invalid command arguments (specify just one action)

The switch -S is another example. The man page has:

-S [MSDOS, OS/2, WIN32 and ATARI] Include system and hidden
files.
[MacOS] Includes finder invisible files, which are
ignored otherwise.

but if you use that switch, you get:

zip -S foo
zip error: Invalid command arguments (no such option: S)

I think the comment that the zip man page refers to OS 9 is probably correct. Those functions seem to be absent in OS X.



---
ODJ



[ Reply to This | # ]
A caution on using zip for backups
Authored by: DavidRavenMoon on Nov 30, '06 08:37:00AM

I use Stuffit exactly because of this problem. I posted this in the topic about Stuffit and MIME files, but it you use zip to archive files with resource forks, such as PostScript fonts, you will ruin the files.

Tiger does not suffer from this issue, at least when zipping fonts, and I seem to remember that Panther was OK, but older systems don't keep the resource fork.

---
G4/Digital Audio/1GHz, 1 GB, Mac OS X 10.4.8 • www.david-schwab.com



[ Reply to This | # ]
A caution on using zip for backups
Authored by: adrianm on Nov 30, '06 09:05:39AM
Now I'm home, I tried zipping files with resource forks using right-click, Create Archive of...

It definitely stores resource fork info in there. There's a top-level __MACOSX dir in the zip containing it. This is recombined when extracting.

Using 10.4.8.

The same thing happens when emailing a folder in Mail.app. All the bits remain intact.

Maybe I misread the original poster, or perhaps s/he's not using 10.4 ?

---
~/.sig: not found

[ Reply to This | # ]

Zip does preserve forks.
Authored by: Chas on Nov 30, '06 10:06:58AM

You are correct, and the original poster is wrong. Apple wrote a special implementation of .zip format into BOMArchiveHelper.app that preserves forks but is compatible with PC files.

This tip is totally misleading and incorrect, and should be removed.



[ Reply to This | # ]
article is about command-line zip
Authored by: hayne on Nov 30, '06 10:13:44AM

The article is rather confusingly written in that it starts off by talking about the archive facility available from Finder (which does preserve resource forks - as the article says) when the main point of the article is about the command-line version of zip not preserving resource forks.

So the article is not wrong - it correctly cautions about the use of command-line zip. But it could have been better written so as to avoid the confusion.



[ Reply to This | # ]
article is about command-line zip
Authored by: mantrid on Nov 30, '06 10:37:28AM
... and following that train of thought, these are the tools that Apple says support resource forks in 10.4 (plain old zip isn't among them):

Enhanced command-line tools:
cp, mv, scp, emacs, vim, pico. Properly handle HFS+ resource forks using the new
extended attribute APIs, as do archivers such as tar, rsync, gzip, bzip2, and cpio.

http://images.apple.com/macosx/pdf/MacOSX_UNIX_TB.pdf

[ Reply to This | # ]

man page & how to tell if a utility preserves resource forks
Authored by: hayne on Nov 30, '06 10:09:18AM
1) I think the claim about inaccuracy of the man page for zip stems from a misunderstanding of the use of the term "MacOS" in that man page. MacOS refers to the pre-OS X versions of the Mac OS (e.g. OS 9). Thus those sections are not relevant to OS X.
As far as I can see, there are no OS X-specific features in 'zip' - and hence it doesn't preserve resource forks.

2) In the Unix-level utilities that have been enhanced (in Tiger) to preserve resource forks, this has been implemented via the use of the 'copyfile' function which is in libc. So one way to quickly check if it is likely that a particular utility will preserve resource forks is to look for 'copyfile' in the output from 'nm'. Look at the following results:


% nm /bin/cp | grep copyfile
         U _copyfile
% nm /usr/bin/tar | grep copyfile
         U _copyfile
% nm /usr/bin/zip | grep copyfile
%


[ Reply to This | # ]
man page & how to tell if a utility preserves resource forks
Authored by: gslusher on Dec 04, '06 03:57:04PM
As far as I can see, there are no OS X-specific features in 'zip' - and hence it doesn't preserve resource forks.
Read the comments before yours. The Finder-based "create archive" most certainly does preserve resource forks. I've proved that in Panther (10.3.9) and others did in Tiger. The author of the tip either got things confused or didn't take the difference into account when writing the tip. The headline is a bit of a scare, as it is not true for the way that most people will zip a file/folder, using the "Create archive" option in the Finder contextual menu.

[ Reply to This | # ]
article is about command-line zip
Authored by: hayne on Dec 06, '06 10:01:37AM

I repeat what I said in an earlier comment: the article and my comment that you quote are about the command-line version of zip.
This is reflected in the (revised) headline for the article.



[ Reply to This | # ]
ditto
Authored by: ekc on Nov 30, '06 02:50:37PM
I use ditto to create Finder-style zip archives from the command line. The syntax looks something like this:

ditto -c -k --sequesterRsrc PATH PATH.zip

Use -x instead of -c when unzipping. This works under both Panther and Tiger.

You can also pipe the zip data by substituting "-" for PATH.zip, which can come in handy for backing up to a remote server. For example, you can pipe into curl to post your project directly to an ftp server or do something with ssh to back up to another Mac. I wrote a couple of shell scripts awhile back to aid me in this sort of thing.

[ Reply to This | # ]
rsync or cpio are probably better
Authored by: Sysvr4 on Dec 01, '06 04:45:11AM

I remember from my days of backing up unix servers to tape that tar was generally a bad idea. Things could have changed, but at the time if your archive became corrupt in any way you lost *everything*. With cpio you'd lose only the files affected by the corrupted area.

That said, I just use rsync (with -E for resources) now to backup. I also use it to sync my two laptops against my main machine so that I don't need a .mac account to sync things like iCal, addressbook, Mail, etc.

Hope this helps...



[ Reply to This | # ]
A caution on using command-line zip for backups
Authored by: ocdinsomniac on Dec 01, '06 08:34:32AM

I've also posted a script for creating Finder-style archives using ditto, though ekc beat me to the punch, and his looks way cooler. Anyway, for the record, my script is here:
http://systemsboy.blogspot.com/2006/11/scripts-part-6-archiver.html

Super-simple archiver/expander. Maybe someone can use it.

-systemsboy



[ Reply to This | # ]
A caution on using command-line zip for backups
Authored by: tirerim on Dec 02, '06 11:57:47AM
Note that while using the j option to tar will produce bzip2-compressed archives, which are smaller than gzip-compressed archives, it will also take longer. To use gzip compression instead, use the z option instead:

/usr/bin/tar czvf foo.tgz foo

/usr/bin/tar xzvf foo.tgz


[ Reply to This | # ]