Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

10.6: Compress files with HFS+ compression UNIX
Snow Leopard introduces a new feature that has been used on most of the system files: HFS+ compression. This compression is rather different than most other file compression options available in the sense that it is completely transparent; there isn't even a way to tell that the files are compressed using Snow Leopard's included command line tools (in fact, contrary to what some posts on this site have suggested, command line utilities like strings will see the same file regardless if it is compressed or not).

In order to even determine if a file is compressed using HFS+ compression or not, a tool like hfsdebug is needed.

As an example, here's part of the hfsdebug output for the QuickTime X executable:
$ sudo hfsdebug /Applications/QuickTime Player.app/Contents/MacOS/QuickTime Player

...

  # Resource Fork
  logicalSize          = 6618026 bytes
  totalBlocks          = 1616
  fork temperature     = no HFC record in B-Tree
  clumpSize            = 999
  extents              =   startBlock   blockCount      % of file
                             0x21f625        0x650       100.00 %
                         1616 allocation blocks in 1 extents total.
                         1616.00 allocation blocks per extent on an average.

# Attributes
  <Attributes B-Tree node = 2452 (sector 0x1d1c8)>
  # Attribute Key
  keyLength            = 46
  pad                  = 0
  fileID               = 46007
  startBlock           = 0
  attrNameLen          = 17
  attrName             = com.apple.decmpfs
  # Inline Data
  recordType           = 0x10
  reserved[0]          = 0
  reserved[1]          = 0
  attrSize             = 16 bytes
  attrData             = 66 70 6d 63 04 00 00 00 30 1a fe 00 00 00 00 00 
                          f  p  m  c              0                      
                         
  compression magic    = cmpf
  compression type     = 4 (resource fork has compressed data)
  uncompressed size    = 16652848 bytes
This output shows that the QuickTime X executable has the xattr com.apple.decmpfs (the xattr command line utility in 10.6 hides this attribute), which means that the file is compressed using HFS+ compression.

You can also tell how much space the compression has saved from this output: The logicalSize for the resource fork is the size of the compressed data (6618026 bytes in this case), and the uncompressed size is 16652848 (the same size that the Finder reports). So the Quicktime X executable only takes up about 40% of its reported size; that's nice, but what about using this compression for our own files?

As it turns out, Apple has actually proved a way to use this new compression. If you view the manual for ditto (man ditto) in 10.6, you should see this option listed:

--hfsCompression
When copying files or extracting content from an archive, if the destination is an HFS+ volume that supports compression, all the content will be compressed if appropriate. This is only supported on Mac OS X 10.6 or later, and is only intended to be used in installation and backup scenarios that involve system files. Since files using HFS+ compression are not readable on versions of Mac OS X earlier than 10.6, this flag should not be used when dealing with non-system files or other user-generated content.


So to compress a file or directory with HFS+ compression, all you should usually need to do is type ditto --hfsCompression [src] [dst] into the Terminal, replacing [src] with the path to the source file/directory, and [dst] with the path to the destination file/directory.

Once the compression is done, use hfsdebug to check to see if the file(s) were actually compressed. Note that sometimes ditto will decide to not compress certain files (although I do not know the exact criteria for a file to be "compressible" -- perhaps someone else can provide some insight into this?)

Although Apple recommends against using HFS+ compression for anything other than system files, due to backward compatibility issues, it seems it should not matter provided you do not mount any of the drives you have compressed files on directly under any previous version of Mac OS X. (Although previous OS X versions should not damage the files, HFS+ compressed files will not be readable.)

File sharing from 10.6 is unaffected: copying HFS+ compressed files from a computer running Snow Leopard onto another computer will work fine, regardless of what OS the other computer is running. That's because the files are decompressed before they are transferred. Also, copying a compressed file decompresses it, so copying files to an external drive to use on another computer should not be a problem.

The command line utility afscexpand can be used to decompress files in place (type man afscexpand into the Terminal for details). (Thanks to Ars Technica for their great Snow Leopard review; a lot of this hint is based on information from it.)
    •    
  • Currently 3.50 / 5
  You rated: 5 / 5 (16 votes cast)
 
[42,624 views]  

10.6: Compress files with HFS+ compression | 34 comments | Create New Account
Click here to return to the '10.6: Compress files with HFS+ compression' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
10.6: Compress files with HFS+ compression
Authored by: asmeurer on Sep 17, '09 07:54:52AM

Maybe the compression criteria has to do with whether it will be faster or not. If you can only get minimal compression, you won't really save time on the disk read after decompressing the file. At least, that is the whole reason suggested for compression in the Ars Technica article.



[ Reply to This | # ]
10.6: Compress files with HFS+ compression
Authored by: kyngchaos on Sep 17, '09 08:39:59AM

Ugh, so it looks like the file size reported in the Finder is the uncompressed size. Aside from the whole KB vs. KiB change, this just creates confusion. When I see the file size in the Finder, I expect it to mean how much space it takes up on the disk (the compressed size).

When copying or reading a file, where it's uncompressed automatically to be presented as if it was the uncompressed data on the disk, sure I need to see the uncompressed size.



[ Reply to This | # ]
Command Line Utility for HFS+ Compressed Files
Authored by: brkirch on Sep 17, '09 08:56:09AM
Since I submitted this hint I've been working on a solution for the lack of information about HFS+ compression files, by developing this command line utility:
afsctool
(source is included)

Here's the usage information for it:
$ afsctool
Report if file is HFS+ compressed:                        afsctool [-v] file
Report if folder contains HFS+ compressed files:          afsctool [-v] folder
List HFS+ compressed files in folder:                     afsctool -l[vv] folder
Decompress HFS+ compressed file or folder:                afsctool -d file/folder
Create archive file with compressed data in data fork:    afsctool -a[d] src dst
Extract HFS+ compression archive to file:                 afsctool -x[d] src dst
Apply HFS+ compression to file or folder:                 afsctool -c[klvv] [compressionlevel [maxFileSize [minPercentSavings]]] file/folder
The -c flag applies in place HFS+ compression to a file or recursively to a folder, with the option to specify the compression level (default is 9), change the default maximum size (in bytes) of files to process (default is 20971520, which is 20 MiB), and specify the minimum savings for a file to be HFS+ compressed. If the -k flag is given then compressed files will be checked after compression against the uncompressed files, and if any corruption has occurred then the file will be reverted back to its uncompressed form. The -l flag lists HFS+ compressed files in a folder or if the -c flag is given, it lists the files that fail to compress. The -d flag decompresses HFS+ compressed files (similar to afscexpand, but sometimes more reliable). The -a flag flattens a HFS+ compressed file to the data fork of the destination file, decompressing the original compressed file afterwards if the -d flag is given. The -x flag does the reverse of -a, unflattening a file with HFS+ compressed data in the data fork to the destination file, then decompressed the destination file is the -d flag is given. The -v flag increases verbosity.

Here's some example output:
$ afsctool -v /System
/System:
Number of HFS+ compressed files: 118930
Total number of files: 141619
Total number of folders: 57283
Total number of items (number of files + number of folders): 198902
Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 4221541441 bytes / 4.61 GB (gigabytes) / 4.29 GiB (gibibytes)
Folder size (compressed - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 2457803070 bytes / 2.56 GB (gigabytes) / 2.39 GiB (gibibytes)
Folder size (compressed): 2537395624 bytes / 2.64 GB (gigabytes) / 2.46 GiB (gibibytes)
Compression savings: 39.9%
Appoximate total folder size (files + file overhead + folder overhead): 2722429043 bytes / 2.72 GB (gigabytes) / 2.54 GiB (gibibytes)
A warning about using this utility; do NOT try to compress the System folder. Any files that were not compressed in the System folder were left uncompressed for a reason! If you do try to compress your System folder further, then you will very likely end up having to reinstall Mac OS X.

[ Reply to This | # ]
Command Line Utility for HFS+ Compressed Files
Authored by: n8gray on Sep 17, '09 11:49:22AM
This looks really useful! I'd like to incorporate this into my Backup Bouncer test suite:

http://n8gray.org/code/backup-bouncer

Do you approve? Do you have this tool hosted somewhere so I can check for updates?

Thanks,
-n8

[ Reply to This | # ]
Command Line Utility for HFS+ Compressed Files
Authored by: brkirch on Sep 17, '09 06:58:58PM
Sure, as long as Backup Bouncer remains open source I have no problem with you reusing the code in afsctool. Currently I'm keeping this link updated with the latest version:
http://files.me.com/brkirch/ijt4f7

[ Reply to This | # ]
Command Line Utility for HFS+ Compressed Files
Authored by: yuji on Sep 17, '09 12:05:41PM

Your tool is great; the source contains a lot of magic numbers buried directly. I guess it took a lot of times for you to reverse-engineer the format!
If you could write a blog post about the format of the HFS compressed file, it would be very helpful for the rest of us...

By the way, do you think there is a private system call which does the compression? Can it be found inside the source code of xnu?



[ Reply to This | # ]
Command Line Utility for HFS+ Compressed Files
Authored by: brkirch on Sep 17, '09 07:01:13PM
The first thing that I tried to do was figure out the system call used for compression; ditto uses the private framework Bom to compress files. However, looking into it further I found that the Bom framework makes a call to another private framework: AppleFSCompression. Unfortunately the syntax for the functions in AppleFSCompression is far from obvious, especially since it is a private framework which means that there are no included headers for it (and it also means that it will probably remain closed source). It didn't in the end matter however, because I found that zlib is used for the actual compression and that library is well documented. So I decided to simply figure out how the HFS+ compressed files are constructed and added that to my program.

Here's how the HFS+ compression is applied:
1. Check the file to ensure it does not have a resource fork or com.apple.decmpfs extended attribute.
2. Construct the headers, calculate the number of 64 KiB blocks needed based on the source file size.
3. Compress the 64 KiB blocks using zlib (Apple uses compression level 5, but other compression levels also work); if there is only one block then append it to the com.apple.decmpfs extended attribute if the compressed data is 3786 bytes or less in size, otherwise the compressed data is put into the resource fork of the file. If the resource fork is used to store the compressed data, then no block is allowed to be larger after compression (if a block is larger after compression then compression for the entire file will fail). After the compressed blocks are created then their locations and sizes are written to the resource fork data header.
4. Add the com.apple.decmpfs extended attribute to the file, then the resource fork if one is needed.
5. Truncate the data fork length to zero and use chflags to set the HFS+ compression flag.

This produces compressed files that are identical to the ones produced by ditto, provided compression level 5 was used for the zlib functions.

[ Reply to This | # ]
Command Line Utility for HFS+ Compressed Files
Authored by: baltwo on Sep 17, '09 05:26:29PM
The download link is broken->goes to MacOSXHints.

Download afsctool here: http://files.me.com/brkirch/ijt4f7

[ Reply to This | # ]
Command Line Utility for HFS+ Compressed Files
Authored by: brkirch on Sep 17, '09 06:57:38PM
Thanks for posting a fixed link, when I previewed the post it looked like the redirect it added should work, but apparently not. There is also one other mistake in my post; the default compression level is 5 (the same as the compression level ditto uses).

[ Reply to This | # ]
Command Line Utility for HFS+ Compressed Files
Authored by: brh on Sep 23, '09 12:25:16PM

Just a quick q - everything I try to compress via afsctool seems to give me an 'Unable to compress file,' which isn't particularly informative… Any clue as to what could be causing everything to fail out? And it's not the obvious - I am on 10.6, HFS+…



[ Reply to This | # ]
Command Line Utility for HFS+ Compressed Files
Authored by: brkirch on Nov 20, '09 05:19:14PM

The latest version of afsctool will only fail to compress a file if one of the following is true:
-the file has a resource fork
-the file has a zero length data fork
-the file is not a regular file (if it is for example, a symbolic link)
-the file will become larger after compression is applied
-the file is greater than 2 GiB in size
-there are insufficient privileges to change the file
-user specified conditions aren't met



[ Reply to This | # ]
Website for afsctool
Authored by: brkirch on Nov 20, '09 05:11:04PM

I have made a website for my software:
http://web.me.com/brkirch

Future updates to afsctool will be posted there.



[ Reply to This | # ]
more/less doesn't uncompress first
Authored by: macgician on Sep 17, '09 04:48:47PM

> (in fact, contrary to what some posts on this site have suggested, command line utilities like strings will see the same file regardless if it is compressed or not)


Not entirely true. While "cat" will uncompress the file first, "more" and "less" will show a raw binary dump of the compressed file. That's a quick and dirty way to tell if a file is compressed, at least until Apple fixes it. Your utility is far more useful obviously.



[ Reply to This | # ]
more/less doesn't uncompress first
Authored by: brkirch on Sep 17, '09 07:04:54PM
Yes I could have worded that better. After I submitted this hint I found that there are some obscure ways to tell if a file is compressed. For example, if stat -f %f [file] returns a number that has the 32 flag set (the UF_COMPRESSED flag) then the file is HFS+ compressed. I can't say I can see how more or less can be used to identify compressed files though, as the data being printed to the screen is the uncompressed data, not the compressed (just tried it, you may have somehow gotten a different result but I tried it on some compressed files and it is definitely showing the uncompressed data).

[ Reply to This | # ]
more/less doesn't uncompress first
Authored by: scooby509 on Sep 17, '09 08:52:32PM
Less certainly works differently than cat, but Apple would have to specifically modify it to not use standard POSIX calls. Here's the source to cat; search for 'fdopen' to see where it's opening its input file.

Less is a surprisingly complex utility but it still uses a standard POSIX function to open a regular file in this source file, search for open(qopen_filename, OPEN_READ) within the file to find it.

man 3 fdopen 2 open will confirm that both of these system calls are standard POSIX calls, so they don't really know anything about HFS vs. any other file system.

(Also, 'more' is simply the compatibility mode of 'less.' They're the same file, hard linked.)

[ Reply to This | # ]

more/less doesn't uncompress first
Authored by: michelcolman on Sep 18, '09 06:57:44AM

Nope, I just compressed an rtf file, then used "more" and "less", the file appeared normally, as if nothing was special about it, no raw compressed data but the original, readable text.



[ Reply to This | # ]
10.6: Compress files with HFS+ compression
Authored by: michelcolman on Sep 18, '09 04:11:58AM

Wow, I just saved several gigabytes by compressing a few big apps (just using "sudo ditto --hfsCompression"), I can't tell the difference except for the free space I gained!

For example Photoshop still takes the same amount of time to launch after a cold reboot (even slightly quicker, if anything), does not complain about damaged files, I was afraid it might give me some DRM-like nastiness with moved/changed files, but no, everything just works like before, only taking a few hundreds of MB less!

Next up is the Application support folder :-)



[ Reply to This | # ]
10.6: Compress files with HFS+ compression
Authored by: michelcolman on Sep 18, '09 06:24:07AM

Only one strange thing: when compressing some application folders, I got messages like

ditto: /Applications/Utilities/Gecomprimeerd/ResDataBridge 1.0/.BC.n6FLVl: Permission denied
ditto: /Applications/Games/StellaOSX2.2/Roms/.BC.YRr5fp: Permission denied
ditto: /Applications/Games/MacMAME/old/Documentation/.BC.j2H1FL: Permission denied
ditto: /Applications/Games/MacMAME/old/Documentation/.BC.r0fvwu: Permission denied
ditto: /Applications/Games/MacMAME/old/Documentation/.BC.kCj925: Permission denied
ditto: /Applications/Games/MacMAME/old/Documentation/.BC.vm8yx2: Permission denied

I just listed a few, but I got similar messages for the Photoshop folder and several others.

When I go look for these mysterious files (.BC.xxxxxx), they simply do not seem to exist. "ls -a" on the original folder does not show them. The compressed applications work just fine afterwards.

What are these "files"? Is it just a bug in ditto? Is it some internal file system stuff that is not supposed to show up in the higher abstraction levels? It this were Windows I would immediately suspect a virus, but on a Mac that's obviously impossible...



[ Reply to This | # ]
Folder Action
Authored by: MVasilakis on Sep 18, '09 10:40:06AM

Now all we need is an Automator script or Applescript that will compress all files moved into a folder.



[ Reply to This | # ]
Folder Action
Authored by: tedw on Sep 18, '09 12:07:16PM
the script is easy:
on adding folder items to thisFolder after receiving theseItems
	set destinationPath to "/POSIX path/to/destination/folder"
	repeat with thisItem in theseItems
		set cmd to "ditto --hfsCompression " & quoted form of POSIX path of thisItem & " " & destinationPath
		do shell script cmd with administrator privileges
	end repeat
end adding folder items to
the 'with administrator privileges' part is unnecessary (and annoying) if you're compressing files you have ownership of. It authenticates you whether it needs to or not.

[ Reply to This | # ]
Folder Action
Authored by: wgscott on Sep 20, '09 11:13:34AM

I'm pretty certain you always need to have sudo to run the ditto --hfsCompress command.



[ Reply to This | # ]
Folder Action
Authored by: tedw on Sep 20, '09 02:19:49PM
never use sudo when calling shell scripts from applescript - opens a security hole. that's what 'with administrator privileges' is for. see this tech note

[ Reply to This | # ]
Insecurity
Authored by: TruePath on Feb 02, '10 03:32:46PM

That attitude probably creates more security holes than all technical errors put together.

Security is not an absolute matter, nor does it have the same requirements everywhere. Insisting on following rigid little rules like this creates annoyances and inconviences that people then respond to by circumventing the security mechanism entierly, e.g., telling anyone who sits down at the machine their password so they can run secure processes, chowing all the files to their own user, etc..

For instance on my laptop I add myself to the wheel group and let the wheel group use sudo without a password. Would I do this on a corporate machine or webserver? No! But does that make it a security risk on my laptop? No. The point of security on my laptop is to protect my private documents and files. If they already have access to my user account the game is over.



[ Reply to This | # ]
10.6: Compress files with HFS+ compression
Authored by: wgscott on Sep 19, '09 03:44:51PM
I nominate this for the best SL hint so far. I've been enjoying compression of many files. I've found it especially effective with PDFs and with our X-ray data files. I also squeezed by fink /sw directory down by a factor of two. By the way, "du" provides a quick way to assess hfs compression, eg:
% ls -lh less
-rwxr-xr-x  2 root  wheel   285K May 18 11:48 less

% du -h -d 0 less           120K    less


[ Reply to This | # ]
10.6: Compress files with HFS+ compression
Authored by: rmanke on Sep 22, '09 05:57:23AM

Awesome tip, thanks!

Now, all we need is some enterprising developer to make a nice user friendly app to compress and get info on compressed files.

I am a bit nervous using the terminal to modify/convert files in the event I cause a problem with a typo... ;)



[ Reply to This | # ]
10.6: Compress files with HFS+ compression
Authored by: Jacques on Feb 25, '10 08:51:01PM

I just discovered one - and it's free as a special via MacHeist.com right now.

It's called Squeeze.



[ Reply to This | # ]
10.6: Compress files with HFS+ compression
Authored by: kylekai on Mar 18, '10 11:11:15AM
There's HDCleanUp, does HFS+ Compression and a bunch of other related tasks.  Only $10.

[ Reply to This | # ]
Squeeze - GUI for this tip!
Authored by: Jacques on Feb 25, '10 08:53:06PM

MacHeist.com has just begun to give away an app for free that basically provides a nice GUI for this tip - I think the give-away is only for one week.

Go to MacHeist.com to download Squeeze, the author's website is http://www.latenitesoft.com/squeeze/

Edited on Feb 25, '10 08:53:40PM by Jacques



[ Reply to This | # ]
Squeeze - GUI for this tip!
Authored by: mdbookworm on Feb 27, '10 06:51:56AM

Has anyone found a way to make Finder accurately report the compressed file sizes?
How do I know how much space I actually have on my drive when Squeeze says I have saved 22.9 GB and Finder says I have only 2.9 GB free (the same amount I had before Squeeze squeezed)?
Path Finder also shows only 2.9 GB free.



[ Reply to This | # ]
10.6: Compress files with HFS+ compression
Authored by: mchafs on Mar 22, '10 10:23:02AM

This HFS+ compression thread is the first clue to a problem I have. After a boot failure the local apple store did an archive and install. The OS was set up for multiple users with one admin. The admin account data is fine but all the other users data is invisible with a .BC file prefix on all the user data files. The current OS is 10.5.8. Could they have used 10.6 initially and compressed the users files? All system files are OK. Is there a way to restore the 190 GB of now invisible and unreadable files? In terminal mode they all show up with a ls -l command but the filenames are coded. The only solution the apple store offers is a wipe and install.



[ Reply to This | # ]
Squeeze and Compressing /Developer
Authored by: thinkyhead on Aug 04, '10 10:57:02AM
There's a menu bar add-on called Squeeze which does HFS compression on folders in the background. But Squeeze won't compress anything in the /Developer folder. To get around this limitation and save some space in /Developer paste the following shell script into a file called dev-compress.sh ...
#!/bin/bash
#
# dev-compress.sh
# HFS compress most of /Developer
#

cd /Developer
# cd /Xcode4 ;# for xcode 4 beta

for DIR in Applications Documentation Examples Extras Library Platforms SDKs usr
do
  echo Compressing $DIR...
  mv $DIR $DIR.orig
  ditto --hfsCompression $DIR.orig $DIR
  rm -rf $DIR.orig
done
... then prep and execute...
chmod ug+x dev-compress.sh
sudo dev-compress.sh
After a few months I have seen no adverse effects on using XCode 3 or XCode 4 with these folders compressed.
---
|
| slur was here
|


[ Reply to This | # ]
10.6: Compress files with HFS+ compression
Authored by: dclunie on Aug 18, '11 04:51:43AM

Related to HFS+ Compression is the behavior of Retrospect 6.1 backup.

Apparently running Retrospect on a Snow Leopard system causes all HFS+ Compressed files to become uncompressed (and manifests itself as a multitude of execution errors described in the the log as changes of the modification date/time to the current date/time).

See "https://discussions.apple.com/thread/2674794" and "http://www.techcare.com/blogs/jk-buzz/102-jk-buzz-172010.html".



[ Reply to This | # ]
10.6: Compress files with HFS+ compression
Authored by: se30 on Nov 10, '11 05:15:20PM

I've found a nice utility that performs this as a background process:

http://latenitesoft.com/clusters/

So far it has worked well for me, including compressing all Adobe products, saving approximately 50% space, which makes quite a difference now that I've moved to a considerably smaller SSD drive.



[ Reply to This | # ]
10.6: Compress files with HFS+ compression
Authored by: Merlin748 on Jul 30, '12 03:57:09AM

There is a new GUI tool for HFS+ compression in appstore called MoreSpace Folder Compression.

http://itunes.apple.com/app/morespace-folder-compression/id521635253?mt=12

Simply but usable.



[ Reply to This | # ]