Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

Use Spotlight to speed up grep searches UNIX
The grep -ri command (recursively search for matches, ignoring case) in Terminal is incredibly useful, but incredibly slow. In OS X, we can leverage mdfind to quickly find the files containing our search string, then use grep to find the strings within only those files. Here's a script to do just that.
#!/bin/bash
#
# Spotlight metadata find and grep by Daniel.Hoherd at gmail dot com

## Check for at least two arguments, print usage if else
if [ $# -lt 1 ] ; then
  echo "usage: $0 searchstring [dir or file] [dir2 or file2]"
  exit ;
fi

ss=$1;
shift;
files=$@;

until [ -z "$1" ] ; do
  thisitem=$1
  onlyin="-onlyin '$thisitem' $onlyin"
  shift
done;
eval mdfind -0 $onlyin "$ss" | xargs -0 grep -Hi "$ss"
Leveraging mdfind to eliminate files that do not contain the matching string makes this operation an order of magnitude quicker, cutting it to mere seconds instead of minutes on over 100GB of data in my tests. The downside to this is that Spotlight does not index every directory, so you may be missing some files such as system files.

If you don't want to use this script, the same thing can be accomplished by hand by writing mdfind -0 -onlyin searchdir searchstring | xargs -0 grep -Hi searchstring. I've included the -H and -i options to more closely match the output of mdfind, but you can safely remove them.
    •    
  • Currently 3.32 / 5
  You rated: 4 / 5 (22 votes cast)
 
[10,526 views]  

Use Spotlight to speed up grep searches | 7 comments | Create New Account
Click here to return to the 'Use Spotlight to speed up grep searches' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Use Spotlight to speed up grep searches
Authored by: Anonymous on Mar 23, '10 10:20:36AM

Combine this hint with this alias for even better mind-blowing insanely fast searches :

alias grep='LC_ALL="C" grep'



[ Reply to This | # ]
Use Spotlight to speed up grep searches
Authored by: neuralstatic on Mar 23, '10 11:03:33AM

in some recent testing, we actually found grep about 4x faster over a directory of about 10,000 files.
if it were disk-wide that might give different results, but by no means is spotlight a clear winner always.



[ Reply to This | # ]
Use Ack
Authored by: squarepegsys on Mar 23, '10 12:33:15PM
Or just use Ack. A Perl program that is much faster than grep in most cases, especially if you just want to search different file types. It is available via CPAN and MacPorts.

[ Reply to This | # ]
Use Spotlight to speed up grep searches
Authored by: baltwo on Mar 23, '10 01:15:48PM
The downside to this is that Spotlight does not index every directory, so you may be missing some files such as system files.
Methinks you're wrong here. Spotlight indexes everything, but many areas are excluded from the results, by default. The way around this is to include system files and invisible files in the search criteria.

[ Reply to This | # ]
Splitting hairs
Authored by: lullabud on Mar 23, '10 03:33:51PM

This hint ( http://www.macosxhints.com/article.php?story=2005050222125145 ) says otherwise, but either way it's splitting hairs. The behavior is still the same.



[ Reply to This | # ]
Use Spotlight to speed up grep searches
Authored by: donmoemu on Mar 24, '10 05:53:04AM

baltwo, Can you provide the workaround that would search the system and invisible files? It would also be nice if you could document where is states that spotlight indexes the above data.



[ Reply to This | # ]
Use Spotlight to speed up grep searches
Authored by: mario_grgic on Mar 23, '10 06:24:20PM

I'm not sure this is all that useful. Usually when I search globally in spotlight I'm interested in wider array of file types (unless I know my file is say pdf then I restrict the search to kind:pdf etc).

But in terminal I'm usually not interested in pdf, doc files, presentations etc. I care about plain text files or code files. In that case find with xargs and grep is sufficient.

find command allows you to restrict your search to only files of certain type, files that were created/modified/accessed at certain date (range), it allows to recursively go into directories up to specified depth, skip certain files or directories etc. find search criteria alone can be quite complex (if it needs to be, usually it isn't), and then xargs and grep allow you to search inside files that passed through find filter. I find this always works for me and it's reasonably fast.



[ Reply to This | # ]