Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!


Click here to return to the 'Command Line Utility for HFS+ Compressed Files' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Command Line Utility for HFS+ Compressed Files
Authored by: brkirch on Sep 17, '09 08:56:09AM
Since I submitted this hint I've been working on a solution for the lack of information about HFS+ compression files, by developing this command line utility:
afsctool
(source is included)

Here's the usage information for it:
$ afsctool
Report if file is HFS+ compressed:                        afsctool [-v] file
Report if folder contains HFS+ compressed files:          afsctool [-v] folder
List HFS+ compressed files in folder:                     afsctool -l[vv] folder
Decompress HFS+ compressed file or folder:                afsctool -d file/folder
Create archive file with compressed data in data fork:    afsctool -a[d] src dst
Extract HFS+ compression archive to file:                 afsctool -x[d] src dst
Apply HFS+ compression to file or folder:                 afsctool -c[klvv] [compressionlevel [maxFileSize [minPercentSavings]]] file/folder
The -c flag applies in place HFS+ compression to a file or recursively to a folder, with the option to specify the compression level (default is 9), change the default maximum size (in bytes) of files to process (default is 20971520, which is 20 MiB), and specify the minimum savings for a file to be HFS+ compressed. If the -k flag is given then compressed files will be checked after compression against the uncompressed files, and if any corruption has occurred then the file will be reverted back to its uncompressed form. The -l flag lists HFS+ compressed files in a folder or if the -c flag is given, it lists the files that fail to compress. The -d flag decompresses HFS+ compressed files (similar to afscexpand, but sometimes more reliable). The -a flag flattens a HFS+ compressed file to the data fork of the destination file, decompressing the original compressed file afterwards if the -d flag is given. The -x flag does the reverse of -a, unflattening a file with HFS+ compressed data in the data fork to the destination file, then decompressed the destination file is the -d flag is given. The -v flag increases verbosity.

Here's some example output:
$ afsctool -v /System
/System:
Number of HFS+ compressed files: 118930
Total number of files: 141619
Total number of folders: 57283
Total number of items (number of files + number of folders): 198902
Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 4221541441 bytes / 4.61 GB (gigabytes) / 4.29 GiB (gibibytes)
Folder size (compressed - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 2457803070 bytes / 2.56 GB (gigabytes) / 2.39 GiB (gibibytes)
Folder size (compressed): 2537395624 bytes / 2.64 GB (gigabytes) / 2.46 GiB (gibibytes)
Compression savings: 39.9%
Appoximate total folder size (files + file overhead + folder overhead): 2722429043 bytes / 2.72 GB (gigabytes) / 2.54 GiB (gibibytes)
A warning about using this utility; do NOT try to compress the System folder. Any files that were not compressed in the System folder were left uncompressed for a reason! If you do try to compress your System folder further, then you will very likely end up having to reinstall Mac OS X.

[ Reply to This | # ]
Command Line Utility for HFS+ Compressed Files
Authored by: n8gray on Sep 17, '09 11:49:22AM
This looks really useful! I'd like to incorporate this into my Backup Bouncer test suite:

http://n8gray.org/code/backup-bouncer

Do you approve? Do you have this tool hosted somewhere so I can check for updates?

Thanks,
-n8

[ Reply to This | # ]
Command Line Utility for HFS+ Compressed Files
Authored by: brkirch on Sep 17, '09 06:58:58PM
Sure, as long as Backup Bouncer remains open source I have no problem with you reusing the code in afsctool. Currently I'm keeping this link updated with the latest version:
http://files.me.com/brkirch/ijt4f7

[ Reply to This | # ]
Command Line Utility for HFS+ Compressed Files
Authored by: yuji on Sep 17, '09 12:05:41PM

Your tool is great; the source contains a lot of magic numbers buried directly. I guess it took a lot of times for you to reverse-engineer the format!
If you could write a blog post about the format of the HFS compressed file, it would be very helpful for the rest of us...

By the way, do you think there is a private system call which does the compression? Can it be found inside the source code of xnu?



[ Reply to This | # ]
Command Line Utility for HFS+ Compressed Files
Authored by: brkirch on Sep 17, '09 07:01:13PM
The first thing that I tried to do was figure out the system call used for compression; ditto uses the private framework Bom to compress files. However, looking into it further I found that the Bom framework makes a call to another private framework: AppleFSCompression. Unfortunately the syntax for the functions in AppleFSCompression is far from obvious, especially since it is a private framework which means that there are no included headers for it (and it also means that it will probably remain closed source). It didn't in the end matter however, because I found that zlib is used for the actual compression and that library is well documented. So I decided to simply figure out how the HFS+ compressed files are constructed and added that to my program.

Here's how the HFS+ compression is applied:
1. Check the file to ensure it does not have a resource fork or com.apple.decmpfs extended attribute.
2. Construct the headers, calculate the number of 64 KiB blocks needed based on the source file size.
3. Compress the 64 KiB blocks using zlib (Apple uses compression level 5, but other compression levels also work); if there is only one block then append it to the com.apple.decmpfs extended attribute if the compressed data is 3786 bytes or less in size, otherwise the compressed data is put into the resource fork of the file. If the resource fork is used to store the compressed data, then no block is allowed to be larger after compression (if a block is larger after compression then compression for the entire file will fail). After the compressed blocks are created then their locations and sizes are written to the resource fork data header.
4. Add the com.apple.decmpfs extended attribute to the file, then the resource fork if one is needed.
5. Truncate the data fork length to zero and use chflags to set the HFS+ compression flag.

This produces compressed files that are identical to the ones produced by ditto, provided compression level 5 was used for the zlib functions.

[ Reply to This | # ]
Command Line Utility for HFS+ Compressed Files
Authored by: baltwo on Sep 17, '09 05:26:29PM
The download link is broken->goes to MacOSXHints.

Download afsctool here: http://files.me.com/brkirch/ijt4f7

[ Reply to This | # ]
Command Line Utility for HFS+ Compressed Files
Authored by: brkirch on Sep 17, '09 06:57:38PM
Thanks for posting a fixed link, when I previewed the post it looked like the redirect it added should work, but apparently not. There is also one other mistake in my post; the default compression level is 5 (the same as the compression level ditto uses).

[ Reply to This | # ]
Command Line Utility for HFS+ Compressed Files
Authored by: brh on Sep 23, '09 12:25:16PM

Just a quick q - everything I try to compress via afsctool seems to give me an 'Unable to compress file,' which isn't particularly informative… Any clue as to what could be causing everything to fail out? And it's not the obvious - I am on 10.6, HFS+…



[ Reply to This | # ]
Command Line Utility for HFS+ Compressed Files
Authored by: brkirch on Nov 20, '09 05:19:14PM

The latest version of afsctool will only fail to compress a file if one of the following is true:
-the file has a resource fork
-the file has a zero length data fork
-the file is not a regular file (if it is for example, a symbolic link)
-the file will become larger after compression is applied
-the file is greater than 2 GiB in size
-there are insufficient privileges to change the file
-user specified conditions aren't met



[ Reply to This | # ]
Website for afsctool
Authored by: brkirch on Nov 20, '09 05:11:04PM

I have made a website for my software:
http://web.me.com/brkirch

Future updates to afsctool will be posted there.



[ Reply to This | # ]