Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

Script to delete huge sparsebundle images UNIX
I needed to restart a backup for one of my Macs, which had an 800 GB disk image on my Time Capsule. It turned out to be impossible to delete the disk image - in Finder, it would just not move to the Trash, even after waiting for several hours.

I was looking for advice and most of the suggestions involved reformatting the Time Capsule, but I couldn't do this, as I had other Macs being backed up there.

I looked for a way to do this from the command line. The Time Machine disk image - a sparsebundle, a special type of disk image that can increase in size as needed - is actually a package, which has "bands" inside, many smaller files that are named hexadecimally. In this case, there were thousands of them. So many, that even deleting them from the command line didn't work - commands like rm find limitations in the lengths of lists parsed by recursion or wildcards, so a basic command like rm -f didn't work.

I ended up writing a little script, which removes the files one by one. The biggest trick here is actually the %x switch which changes $i into a hexadecimal number.

You can then either copy and save something like this as a shell script:

for i in {0..1000000}
do
rm -rv /Volumes/Your Network\ Disk/Your\ Mac.sparsebundle/bands/$(printf "%x" $i)
Done


where you replace Your\ Network\ Disk and Your\ Mac with your details.

Or, you can copy and paste this line into Terminal:
for i in {0..1000000}; do rm -rv
then add a space, then drag and drop the disk image, and then (you might need to remove one space this time) add:

/bands/$(printf "%x" $i); done

Then just watch the bands being deleted, and when the script starts deleting non-existing bands, just kill it with ctrl-X.

With no bands, the sparsebundle gets deleted immediately.

[kirkmc adds: I haven't tested this. My first thought was that this seemed like overkill, but corresponding with the submitter made me realize the extent of the problem. Here is some more information on this limitation.]
    •    
  • Currently 2.80 / 5
  You rated: 1 / 5 (10 votes cast)
 
[15,375 views]  

Script to delete huge sparsebundle images | 21 comments | Create New Account
Click here to return to the 'Script to delete huge sparsebundle images' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Script to delete huge sparsebundle images
Authored by: richard2 on Jul 19, '12 08:12:13AM

Surely you could simply use a command that doesn't use path expansion, such as rm -R example.sparsebundle?



[ Reply to This | # ]
Script to delete huge sparsebundle images
Authored by: fracai on Jul 19, '12 08:16:35AM
I'm not sure what the problem with rm -rf /path/to/folder/ could have been, I've never had a problem with recursively removing large directories. rm -rf /path/to/folder/*? Sure, there's a limit to how many arguments can be passed on the command line. But, that's where xargs comes in.

For example:
find /path/to/folder -print0 | xargs -0 rm -f

That's it. The -print0 separates the found items with a null character so you don't have to worry about escaping characters. xargs -0 recognizes the null separation and passes the listed arguments to whatever command follows next. Plus, xargs is smart enough to not send more arguments than is allowed to the command to be executed.

But, there's an even easier option:
find /path/to/folder -print -delete

The -print is optional, but allows you to see the files as they are deleted. I'm sure it would delete faster if you omit the printing.

Add a sudo as required. Note, that the first example uses a pipe, so you'd need to use sudo before the xargs command as well.

And of course, remember that you're deleting files here, be careful.
---
i am jack's amusing sig file


[ Reply to This | # ]
Script to delete huge sparsebundle images
Authored by: richard2 on Jul 19, '12 10:04:34AM

You can simplify

find Path -print0 | xargs -0 Command

to

find Path -exec Command {} +

find's documentation describes this syntax as follows:

-exec utility [argument …] {} +
Same as -exec, except that "{}" is replaced with as many pathnames as possible for each invocation of utility. This behaviour is similar to that of xargs(1).

Edited on Jul 19, '12 10:11:53AM by richard2


[ Reply to This | # ]
Script to delete huge sparsebundle images
Authored by: fracai on Jul 19, '12 10:16:34AM

Yeah, I've always preferred piping to xargs though. I think it's the requirement to use brackets to insert the arguments and the cryptic, required terminator '+' or ';'.

And if you need to have arguments after the found files, you can always use the -I (or -J?) flags to specify an insertion character.

To each their own.

---
i am jack's amusing sig file
Edited on Jul 19, '12 10:18:50AM by fracai



[ Reply to This | # ]
Script to delete huge sparsebundle images
Authored by: wjv on Jul 26, '12 05:34:59AM
xargs is preferable as it would result in vastly fewer invocations of Command.

[ Reply to This | # ]
Script to delete huge sparsebundle images
Authored by: rotaiv on Jul 19, '12 08:18:59AM
When ever I have to delete large number of files (mac, win, linux) I simply delete the parent folder. I created a temp directory with 5,000 files and could not delete the individual files but could delete the parent folder with no issues:

$ pwd
/tmp/ManyFiles

$ ls | wc -l
50000

$ rm *
-bash: /bin/rm: Argument list too long

$ cd ..

$ rm -rf ManyFiles/



[ Reply to This | # ]

Another option
Authored by: escowles on Jul 19, '12 08:28:36AM
As several others have noted, you can just delete the parent folder. If you don't want to do that, for example if you just want to delete some of the files or move them somewhere else, you can also use head to limit the number you work with at one time.

For example, you can remove 1000 files at a time:

rm `ls | head -1000`

or remove just the JPG files:

rm `ls | grep -i jpg | head -1000`

[ Reply to This | # ]
Another option
Authored by: karstenw on Jul 19, '12 08:42:18AM

Sorry, but you can't just delete the parent folder if it contains millions of files. I watched the Finder counting for three days, preparing to delete 12 million files in one folder and when the count exceeded the number of files on the volume I killed it.



[ Reply to This | # ]
Another option
Authored by: fracai on Jul 19, '12 06:55:46PM
The Finder is a bit different from the command line though. For one, you don't hear many people chanting "Fix the *censored*ing Command Line" The comments, for example, on the link that you posted seem to indicate that rm can handle recursively removing large numbers of files.

I replicated the example from the comments:

$ mkdir large

$ time for i in {0..999999} ; do echo > large/$i ; done
real	6m41.485s
user	0m36.963s
sys	2m28.405s

$ time rm -r large
real	3m21.800s
user	0m3.100s
sys	2m9.808s

Worked fine for me.
---
i am jack's amusing sig file


[ Reply to This | # ]
Script to delete huge sparsebundle images
Authored by: karstenw on Jul 19, '12 08:35:45AM
Script to delete huge sparsebundle images
Authored by: merlyn on Jul 21, '12 02:20:48PM

Heh... I was just going to link to this! Thanks!



[ Reply to This | # ]
Script to delete huge sparsebundle images
Authored by: Mechcozmo on Jul 19, '12 09:16:00AM
This is a pretty common issue, and a very frustrating one because the error isn't clear. Wildcards in the command line expand, which if you have 250,000 files in a directory means you are sending:
rm file1 file2 file3 ... file250000
That's where you get too many arguments from. (It's also technically a kernel-level error, not a bug with rm; see link below) However, you don't need an AppleScript, and in some cases you may not have AppleScript available (*NIX machines, for example). As explained here, you can use the find command in order to generate a list and then run the rm command on each result that is found, instead of running rm with every result. This command will delete all files in the current directory:
find . -name '*' | xargs rm


[ Reply to This | # ]
Script to delete huge sparsebundle images
Authored by: bdm on Jul 20, '12 06:22:14AM

There is a possible reason for slow behaviour that hasn't been mentioned. A directory file has an internal structure consisting of a sequence of records, one per file, with the inode number, the name of the file, etc. Do "man 5 dir" to see the details. The records are of different sizes because they depend on the file name length, so there is a non-trivial memory management problem if there are a large number of files. When you delete a file the structure has to be cleaned up and this can take a while. I haven't tested MacOSX in this regard, but I remember SunOS used to have a quadratic algorithm in it so deleting the files from a large directory in random order took approximately forever. Some of the tests being reported here where the files have mostly the same length names and are deleted in approximately they same order as they were created are not likely to find the worst case.



[ Reply to This | # ]
Script to delete huge sparsebundle images
Authored by: wallybear on Jul 20, '12 02:27:12PM
I had to delete a big TimeCapsule sparsebundle two days ago only to find that Finder wasn't able to delete it; as it was taking forever, I stopped it, launched Terminal and with a simple

rm -rf /Volumes/MyTimeCapsule/myMac.sparsebundle
I got rid of it in a couple of minutes. No need to use wildcards (this would give 'Argument list too long' errors), scripts or find/xargs.

[ Reply to This | # ]
Script to delete huge sparsebundle images
Authored by: Sven G on Jul 21, '12 03:31:15AM

Exactly! A simple "rm -rf" has always worked for me, too (while the Finder sometimes might work, sometimes not; anyway, certainly a piece of software that should be improved, the Finder: it still isn't completely reliable, after so many years)...



[ Reply to This | # ]
Script to delete huge sparsebundle images
Authored by: AttilaNab on Sep 16, '12 10:44:58AM

Long Path Tool helped me in this situation. http://PathTooDeep.com



[ Reply to This | # ]
Script to delete huge sparsebundle images
Authored by: mpstx on Oct 15, '12 11:34:49AM

Big thanks for sharing this approach -- one delete command per file, running over and over from cmdline.

I was experiencing issues deleting the bands out of sparsebundle image on a time capsule. Approx 75,000 files here. Tried Finder and it crawled (1 file every ten minutes), the basic rm -rfdv commandline approach (with a directory name) hung for hours, the rm rf with a wildcard failed out trying to expand the wildcard.

This is the only approach that seems to be making steady progress deleting bands out of a sparsebundle on my time capsule. Signed up for a user account here just to say thanks.



[ Reply to This | # ]
Script to delete huge sparsebundle images
Authored by: RTench on Nov 30, '12 10:50:17PM

I prefer to use Windows. It's more steps, but you get a progress bar. It's quite fast:

1. Get the Time Capsule’s IP address from Airport Utility, e.g. 192.168.1.74
2. Fire up VirtualBox (or Parallels or VMWare)
3. Open Windows Explorer
4. In the address field key in \\<IP>, e.g. \\192.168.1.74
5. Key in your user ID for ID and the Time Capsule administrative password (not the wireless password and not your logon password).
6. Click on the sparsebundle in Windows Explorer
7. Delete it.
8. You will get a nice dialog, a time estimate, number of minutes left, etc. and it runs quite quickly.



[ Reply to This | # ]
Script to delete huge sparsebundle images
Authored by: JohnsonCW on Dec 13, '12 06:53:28AM

Thanks! A VirtualBox hosted image of Windows 7 did the trick for me and I probably would not have considered using Windows without your comment.

My Time Capsule resident sparsebundle was 628GB -- I couldn't even "ls" the "bands" directory from Terminal much less open the directory in Finder. "rm -rfv" ran for 8 hours with no visible effect.

Switching to Windows, I was able to use Windows Explorer to mount the Time Capsule share and *delete* (as opposed to "recycle") the sparsebundle file. After about 30 minutes of "Calculating ...", it finally started deleting 80,467 items (which I presume is the about the number of bands + control files). It's not done yet (18+ hours and 260GB remaining) but it's making steady progress.

Seems like it's time for me to learn to proactively prune my backup tree before it gets so large.



[ Reply to This | # ]
Script to delete huge sparsebundle images
Authored by: PeterAhlstrom on Mar 05, '13 11:30:57AM

I just used the main hint as described above. I tried what everyone else said too (except for deleting it from a Windows machine), but the main hint description is the only one that seemed to work on my 400 GB sparsebundle. It's possible the perl script was working, but I couldn't tell. The main script lists the names of what's being deleted, so you can see the progress. I also checked in a different Terminal window, in the sparsebundle directory (not the bands directory) using ls -l every once in a while and I could see the bands directory size going down. It did take a couple of hours but it seems this is the fastest way short of formatting the disk.

Also, for those who are interested, the "%x" part in the script takes the integer $i and turns it into a hexadecimal, because the files in the bands folder are named hexidecimally.



[ Reply to This | # ]
Script to delete huge sparsebundle images
Authored by: cmeffie on Jul 23, '13 05:01:05PM
Rsync worked quickly for me. This command will recursively copy all of the bands and delete them from the source directory.

rsync --remove-source-files -av --progress /path/to/src/ /path/to/dest

[ Reply to This | # ]