(There is a previous hint that supplies a script for deleting duplicate sound loops, but that script relied upon lists of the duplicates having previously been prepared. There's also a hint on finding and removing iTunes duplicates via a Perl script.)
Today's hint supplies a script that will search for duplicate files in specified folders, and output a list of the files that are duplicates of each other. (i.e. it is something that will produce the lists that were used in the first previous hint.)
The script is actually very general, so it can be used to search for duplicates of any type of file. What it does is compare the files based on the "MD5 hash," which is a sequence of characters that is computed from the content of the file. It is extremely unlikely (although theoretically possible) that two files with different content would have the same MD5 value. The file comparison does not look at the file names at all, just the content of the files.
To use it, first copy and paste the source into your favorite pure text editor. Save it with some name, and then you would need to do the usual things for running a script -- see this Unix FAQ in the forums if you have questions on that. Note that there's also a macosxhints' forums thread on this topic, and it will contain any updates made to the script. To put it another way, if you're reading this hint at some point in the future, you may wish to check that thread for a newer version than that shown in the above source.
If the script file is called findDupeFiles, and it is in your current folder, then you could run it on the two folders /Documents/Apple Loops for Soundtrack and /Library/Application Support/GarageBand as follows:
./findDupeFiles '.aif|.aiff' "/Documents/Apple Loops for Soundtrack" \
"/Library/Application Support/GarageBand"
The first argument (.aif|.aiff) specifies that you want to look at files with either a .aif or .aiff suffix. You need to have this argument inside quotes, since the vertical bar (|) that separates the two file suffixes is a special character for the shell. You need to have the folder paths (the other two arguments) in quotes because the paths contain spaces. Note that this command will typically take several minutes to finish, and you won't see any output until just before the end.
The output (in the Terminal window) from the above command would list all the duplicates it found in those folders (and sub-folders). Each set of duplicates is separated from the next in the output by a line like this:
-----------------------
The above example showed how to use the script when looking for duplicates of AIFF files. You could use it similarly to find duplicates of any other files that have definite suffixes. But sometimes your data files don't have a uniform set of suffixes, or perhaps any suffixes at all. You can tell the script to search across all files (independent of suffix) by using an empty string (two quotes right next to each other with no characters in between) as the first argument to the script. For example...
./findDupeFiles '' ~/Documents
...would search your Documents folder for duplicate files of any suffix. (You do need to supply that empty string as a first argument, since otherwise the script would interpret ~/Documents as the suffix to search for.)
Note also that if you don't supply any folder names when you invoke the script, it will search under the current folder. And if you don't supply any arguments at all, it will search all files under your current folder.

