Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!


Click here to return to the 'Command Line Utility for HFS+ Compressed Files' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Command Line Utility for HFS+ Compressed Files
Authored by: yuji on Sep 17, '09 12:05:41PM

Your tool is great; the source contains a lot of magic numbers buried directly. I guess it took a lot of times for you to reverse-engineer the format!
If you could write a blog post about the format of the HFS compressed file, it would be very helpful for the rest of us...

By the way, do you think there is a private system call which does the compression? Can it be found inside the source code of xnu?



[ Reply to This | # ]
Command Line Utility for HFS+ Compressed Files
Authored by: brkirch on Sep 17, '09 07:01:13PM
The first thing that I tried to do was figure out the system call used for compression; ditto uses the private framework Bom to compress files. However, looking into it further I found that the Bom framework makes a call to another private framework: AppleFSCompression. Unfortunately the syntax for the functions in AppleFSCompression is far from obvious, especially since it is a private framework which means that there are no included headers for it (and it also means that it will probably remain closed source). It didn't in the end matter however, because I found that zlib is used for the actual compression and that library is well documented. So I decided to simply figure out how the HFS+ compressed files are constructed and added that to my program.

Here's how the HFS+ compression is applied:
1. Check the file to ensure it does not have a resource fork or com.apple.decmpfs extended attribute.
2. Construct the headers, calculate the number of 64 KiB blocks needed based on the source file size.
3. Compress the 64 KiB blocks using zlib (Apple uses compression level 5, but other compression levels also work); if there is only one block then append it to the com.apple.decmpfs extended attribute if the compressed data is 3786 bytes or less in size, otherwise the compressed data is put into the resource fork of the file. If the resource fork is used to store the compressed data, then no block is allowed to be larger after compression (if a block is larger after compression then compression for the entire file will fail). After the compressed blocks are created then their locations and sizes are written to the resource fork data header.
4. Add the com.apple.decmpfs extended attribute to the file, then the resource fork if one is needed.
5. Truncate the data fork length to zero and use chflags to set the HFS+ compression flag.

This produces compressed files that are identical to the ones produced by ditto, provided compression level 5 was used for the zlib functions.

[ Reply to This | # ]