10.6: Compress files with HFS+ compression

Sep 17, '09 07:30:00AM

Contributed by: brkirch

Snow Leopard introduces a new feature that has been used on most of the system files: HFS+ compression. This compression is rather different than most other file compression options available in the sense that it is completely transparent; there isn't even a way to tell that the files are compressed using Snow Leopard's included command line tools (in fact, contrary to what some posts on this site have suggested, command line utilities like strings will see the same file regardless if it is compressed or not).

In order to even determine if a file is compressed using HFS+ compression or not, a tool like hfsdebug is needed.

As an example, here's part of the hfsdebug output for the QuickTime X executable:

$ sudo hfsdebug /Applications/QuickTime Player.app/Contents/MacOS/QuickTime Player

...

  # Resource Fork
  logicalSize          = 6618026 bytes
  totalBlocks          = 1616
  fork temperature     = no HFC record in B-Tree
  clumpSize            = 999
  extents              =   startBlock   blockCount      % of file
                             0x21f625        0x650       100.00 %
                         1616 allocation blocks in 1 extents total.
                         1616.00 allocation blocks per extent on an average.

# Attributes
  <Attributes B-Tree node = 2452 (sector 0x1d1c8)>
  # Attribute Key
  keyLength            = 46
  pad                  = 0
  fileID               = 46007
  startBlock           = 0
  attrNameLen          = 17
  attrName             = com.apple.decmpfs
  # Inline Data
  recordType           = 0x10
  reserved[0]          = 0
  reserved[1]          = 0
  attrSize             = 16 bytes
  attrData             = 66 70 6d 63 04 00 00 00 30 1a fe 00 00 00 00 00 
                          f  p  m  c              0                      
                         
  compression magic    = cmpf
  compression type     = 4 (resource fork has compressed data)
  uncompressed size    = 16652848 bytes
This output shows that the QuickTime X executable has the xattr com.apple.decmpfs (the xattr command line utility in 10.6 hides this attribute), which means that the file is compressed using HFS+ compression.

You can also tell how much space the compression has saved from this output: The logicalSize for the resource fork is the size of the compressed data (6618026 bytes in this case), and the uncompressed size is 16652848 (the same size that the Finder reports). So the Quicktime X executable only takes up about 40% of its reported size; that's nice, but what about using this compression for our own files?

As it turns out, Apple has actually proved a way to use this new compression. If you view the manual for ditto (man ditto) in 10.6, you should see this option listed:

--hfsCompression
When copying files or extracting content from an archive, if the destination is an HFS+ volume that supports compression, all the content will be compressed if appropriate. This is only supported on Mac OS X 10.6 or later, and is only intended to be used in installation and backup scenarios that involve system files. Since files using HFS+ compression are not readable on versions of Mac OS X earlier than 10.6, this flag should not be used when dealing with non-system files or other user-generated content.


So to compress a file or directory with HFS+ compression, all you should usually need to do is type ditto --hfsCompression [src] [dst] into the Terminal, replacing [src] with the path to the source file/directory, and [dst] with the path to the destination file/directory.

Once the compression is done, use hfsdebug to check to see if the file(s) were actually compressed. Note that sometimes ditto will decide to not compress certain files (although I do not know the exact criteria for a file to be "compressible" -- perhaps someone else can provide some insight into this?)

Although Apple recommends against using HFS+ compression for anything other than system files, due to backward compatibility issues, it seems it should not matter provided you do not mount any of the drives you have compressed files on directly under any previous version of Mac OS X. (Although previous OS X versions should not damage the files, HFS+ compressed files will not be readable.)

File sharing from 10.6 is unaffected: copying HFS+ compressed files from a computer running Snow Leopard onto another computer will work fine, regardless of what OS the other computer is running. That's because the files are decompressed before they are transferred. Also, copying a compressed file decompresses it, so copying files to an external drive to use on another computer should not be a problem.

The command line utility afscexpand can be used to decompress files in place (type man afscexpand into the Terminal for details). (Thanks to Ars Technica for their great Snow Leopard review; a lot of this hint is based on information from it.)

Comments (34)


Mac OS X Hints
http://hints.macworld.com/article.php?story=20090902223042255