Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

10.4: Configure Spotlight to index excluded directories System 10.4
Tiger only hintBy default, Spotlight excludes many hidden, System, and developer directories from the metadata index for simplicity and ease of use by the majority of users. These exclusions include things like hidden UNIX directories and ADC Reference Documentation. It would be very helpful for some users to have some of these excluded areas available for Spotlight searching. While mdimport -f will allow you to manually force Spotlight to add a directory to its index, many of Spotlight's command-line tools make references to filter rules. I wondered where these rules might be kept, and how they might be altered.

I was hoping there would be a text-based configuration file somewhere, but this appeared at first glance to not be the case. It turns out the excluded directories are hardcoded into Spotlight's mds binary, located in /System: Library: Frameworks: CoreServices.framework: Versions: A: Frameworks: Metadata.framework: Versions: A: Support: mds. You can do a strings on the above binary to see them -- or here's the full list of excluded directories.

However, while looking in the .Spotlight-V100 directory at the root of the drive, I found a file called _rules.plist, which can be used to tweak the exclusions.

The contents of this file, by default, are (note that some rows have been split for a narrower display -- you shouldn't be copying and pasting this bit anyway!):
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
 "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>EXCLUDE</key>
        <array/>
        <key>INCLUDE</key>
        <array/>
        <key>NOTE</key>
        <string>Specify paths to include or exclude, preceeding
         rules which target user-homes with ~/</string>
</dict>
</plist>
To add excluded directories to the metadata index, _rules.plist can be modified by adding an array of directories that you'd like searched:
<array>
    <string>/private</string>
    <string>/usr</string>
</array>
Add this section directly below the <key>INCLUDE<key> line. In this example, the /private and /usr directories have been added to the metadata index. After this change, execute sudo mdutil -E / to remove the index and force the volume to reindex. After this, Spotlight has indexed and is able to search /private (including things like /etc) and /usr.

A side effect of this modification is that these areas now become searchable by the Finder's conventional "Find" functionality (Find...; Command-F; or search field in Finder window). Before, a search for httpd.conf yields no results (unless you happen to have other files named "httpd.conf" files around in non-excluded areas). After adding /private, which contains /etc (which is really a symbolic link to /private/etc), "httpd.conf" now appears in search results at its expected location of /private/etc/httpd/httpd.conf.

Further, be advised that explicitly excluded items may need to be explicitly included. For example, Spotlight specifically excludes /Developer/ADC Reference Library by default. Adding /Developer alone to the INCLUDE stanza may not index it; you may need to explicitly include /Developer/ADC Reference Library to cause it to be indexed. Note that I did not yet test this theory; this point is mentioned for information only.
    •    
  • Currently 3.43 / 5
  You rated: 2 / 5 (7 votes cast)
 
[60,529 views]  

10.4: Configure Spotlight to index excluded directories | 28 comments | Create New Account
Click here to return to the '10.4: Configure Spotlight to index excluded directories' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
10.4: Configure Spotlight to index excluded directories
Authored by: ppmax on May 11, '05 12:05:03PM

The only "bummer" about Spotlight so far is that there dont appear to be content importers available for system files like .conf, .plist, and other "generic" files like hostconfig.

I may be missing something but doing mdimport on a directory like /private only imports metadata like create/modify dates, and a couple other somewhat "low value" bits of info. You cant search inside the files.

I saw a suggestion to use SetFileInfo (a cli tool installed with the Developer Tools) that will change the file Kind to a generic text file...which of course will get indexed.

Any other suggestions?



[ Reply to This | # ]
10.4: Configure Spotlight to index excluded directories
Authored by: alblue on May 11, '05 01:24:17PM

Can we move away from using ':' to separate paths? It's really old-skool and the Unix kernel uses posix paths (with / ) for everything these days. The : is really only seen in AppleScripts and some old classic apps; but for a 10.4 hint to use : is really ancient ...



[ Reply to This | # ]
10.4: Configure Spotlight to index excluded directories
Authored by: robg on May 11, '05 02:42:56PM

I use a slash on shorter paths that can be copied and pasted. However, on long paths, which I pad with spaces so they'll wrap, I switched to colons.

Why? Because I was tired of answering tech support emails about why a copy and pasted path didn't work -- people were just copying the whole line and pasting it into the Terminal. I'm open to other suggestions, but since switching, I haven't had one person email asking why a pasted path didn't work...

-rob.



[ Reply to This | # ]
10.4: Configure Spotlight to index excluded directories
Authored by: nevyn on May 12, '05 12:42:22PM

[code]
/this/is/a/long/path/\
split/over/several/rows/\
that/can/be/copy/pasted/
[/code]

---
?



[ Reply to This | # ]
10.4: Configure Spotlight to index excluded directories
Authored by: samsprograms on May 17, '05 03:17:10PM

I think it would be even better to just use CSS to automatically wrap the hints, and stop al of these annoyances...



[ Reply to This | # ]
10.4: Configure Spotlight to index excluded directories
Authored by: sdunlapa on May 11, '05 08:00:15PM

I have changed my _rules.plist to this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs$
<plist version="1.0">
<dict>
<key>EXCLUDE</key>
<array/>
<key>INCLUDE</key>
<array>
<string>/private</string>
<string>/var</string>
<string>/usr</string>
<string>/System</string>
<string>/Library</string>
</array>
<array/>
<key>NOTE</key>
<string>Specify paths to include or exclude, preceeding rules which target user-$
</dict>
</plist>

I have re indexed and still Spotlight will not find the folders or anything in them. What have I done wrong?

BTW I used pico under root to change this file.



[ Reply to This | # ]
10.4: Configure Spotlight to index excluded directories
Authored by: blgrace on Jun 13, '05 11:06:24PM

I used pico too and it doesn't work - unless you remove the extra <array/> after the </array> - - if that makes sense.
If you don't - the _rules.plist just gets over-written back to default.
The following worked for me:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/Propert$
<plist version="1.0">
<dict>
<key>EXCLUDE</key>
<array/>
<key>INCLUDE</key>
<array>
<string>/usr</string>
<string>/System</string>
<string>/Library</string>
<string>/private</string>
<string>/var</string>
</array>
<key>NOTE</key>
<string>Specify paths to include or exclude, preceeding rules which target user-homes wi$
</dict>
</plist>



[ Reply to This | # ]
10.4: Configure Spotlight to index excluded directories
Authored by: valexa on Jul 31, '09 08:50:44AM

the hint does not work anymore , i tested on Leopard
1 - cleaned cache and created a fresh one with sh-3.2# sudo mdutil -E /
2 - sh-3.2# du -h -d 1 /.Spotlight-V100
259M /.Spotlight-V100/Store-V1
309M /.Spotlight-V100
3 - added to _rules.plist in INCLUDE
<array>
<string>/private</string>
<string>/var</string>
<string>/usr</string>
<string>/System</string>
<string>/Library</string>
</array>
4 - rebuilt cache
5 - sh-3.2# du -h -d 1 /.Spotlight-V100
259M /.Spotlight-V100/Store-V1
308M /.Spotlight-V100

as you can see the database has the same size as before



[ Reply to This | # ]
10.4: Configure Spotlight to index excluded directories
Authored by: sdunlapa on May 11, '05 08:07:33PM
Here is the actual _rules.plist without anything cut off

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>EXCLUDE</key>
        <array/>
        <key>INCLUDE</key>
        <array>
        <string>/private</string>
        <string>/var</string>
        <string>/usr</string>
        <string>/System</string>
        <string>/Library</string>
        </array>
        <array/>
        <key>NOTE</key>
        <string>Specify paths to include or exclude, preceeding rules which target user-homes with ~/</string>
</dict>
</plist>


[ Reply to This | # ]
10.4: Configure Spotlight to index excluded directories
Authored by: sdunlapa on May 11, '05 08:21:12PM
so once i change the _rules.plist and save it than I run
mdutil -E /
After I remove the index and it is re created mine defaults back to the original. Is this what is supposed to happen? What I do know is that Spotlight is not showing any files or folders that I added to _rules.plist

[ Reply to This | # ]
10.4: Configure Spotlight to index excluded directories
Authored by: jesboat on May 11, '05 10:52:08PM
After every key line in the file, there's supposed to be only one
<array>
tag. Can you try deleting the second
<array/>
line?

-Jesboat

---
--
With no walls or fences on the 'net, who needs Windows or Gates?

[ Reply to This | # ]

10.4: Configure Spotlight to index excluded directories
Authored by: sdunlapa on May 12, '05 01:26:43AM

Thanks jesboat. worked like a champ
Amazes me how simple some things are.



[ Reply to This | # ]
10.4: Configure Spotlight to index excluded directories
Authored by: stux on May 12, '05 02:18:13AM
I found a much easier way of editing the _rules.plist was to use the following cmd line
sudo /Developer/Applications/Utilities/Property\ List\ Editor.app/Contents/MacOS/Property\ List\ Editor /.Spotlight-V100/_rules.plist
This of course assumes you have the dev tools installed. It will open the _rules.plist in a su'd Property List editor, then you just add siblings for the INCLUDE element and simply fill in as many paths as you want.

[ Reply to This | # ]
10.4: Configure Spotlight to index excluded directories
Authored by: stux on May 12, '05 03:46:04AM

I of course meant "add children" instead of "add siblings"



[ Reply to This | # ]
10.4: Configure Spotlight to index excluded directories
Authored by: BraindeadMac on May 12, '05 07:14:12AM

And using the Property List Editor you can drag and drop folders from the Finder....



[ Reply to This | # ]
10.4: Configure Spotlight to index excluded directories
Authored by: micaelus on May 12, '05 10:29:24AM

There's a good discussion at Apple about this topic as well:

http://discussions.info.apple.com/webx?13@@.68ae41d9/6



[ Reply to This | # ]
Apple brain-dead
Authored by: eno on May 13, '05 10:50:19AM

To me Apple's decision to exclude folders like "/Developer/ADC Reference Library" and "/Developer/Documentation" can only be described as brain-dead.

Spotlight as implemented has a lot of potential, but potential only. ie. it's about 10% as good as it should be and still has 90% to go.



[ Reply to This | # ]
Apple brain-dead
Authored by: samsprograms on May 17, '05 03:21:43PM

This is probably because Xcode searched the developer documentation in its own app...



[ Reply to This | # ]
Just copy the docs and stop worrying
Authored by: mrob27 on Aug 02, '05 07:58:17PM

I solved this is a fairly dumb way. Unfortunately others have tried better methods but the better methods don't work.

Copy selected (or entire) contents of the ADC Reference Library folder into some directory in your home. It's 600 megs, 46000+ files.

This might sound idiotic, but it's great for me because later on I can delete all the APIs I don't care about!

Just to test it I copied the whole thing. My ContentIndex.db grew to 96 megs, and the "mds" process has RSIZE of 7.4 megs, VSIZE of 46 megs, when I invoke Spotlight and do a search.

On my system it took 2 minutes to copy and 10 minutes for Spotlight to index (as evidenced by CPU monitor or load averages in UNIX "w" command). My system is a dual G5 so I imagine other users might take as much as an hour (but you only have to do it once :-)

- Robert



[ Reply to This | # ]
10.4: Configure Spotlight to index excluded directories
Authored by: kirkmc on May 13, '05 10:51:48AM

The list of exclusions linked from the article is incorrect. Spotlight clearly indexes all of these folders:

Desktop
/Documents
/Library/Caches/com.apple.AddressBook/MetaData

/Library/Mail
/Music
/Movies

/Pictures
/Library/Caches/Metadata
/Library/Fonts

/Library/Internet Plug-Ins
/Library/Metadata
/Library/Preferences

/Library/PreferencePanes
/Library/Screen Savers
/Library/Spotlight

/Library/Widgets
/Library/Caches
/Library/Printers

/Applications
/Users/Shared
/System/Library/CoreServices/Help Viewer.app

/System/Library/Fonts
/System/Library/PreferencePanes
/Users

As for the rest, some of them are not indexed, such as all the system folders, but it's not clear exactly which ones.

---
Read my blog: Kirkville -- http://www.mcelhearn.com
Musings, Opinion and Miscellanea, on Macs, iPods and more



[ Reply to This | # ]
10.4: Configure Spotlight to index excluded directories
Authored by: ptejad on May 13, '05 07:36:35PM
Here is a one-liner that will search out all Text files on your drive and set the filetype to TEXT. Modifying what the grep(s) search for and the "TEXT" part of SetfFile this one liner can be used for other types of files that the "file" command can distinguish. Nake sure you use the "file" command to see what it outputs for the type of file you want to change, and use GetFileInfo on example file of that type to see the four-char code needed.

find -x / -type f -print0 | xargs -0 -L 1 file | grep "ASCII" \
| grep "text" | awk -F":" '{print $1}' | grep "/" \
| xargs -n 1 -t -J % /Developer/Tools/SetFile -t "TEXT" "%"


[ Reply to This | # ]
10.4: Configure Spotlight to index excluded directories
Authored by: kirkmc on May 14, '05 05:37:34AM

Spotlight doesn't index TEXT files just because their type is TEXT; they must have the .txt extension.

---
Read my blog: Kirkville -- http://www.mcelhearn.com
Musings, Opinion and Miscellanea, on Macs, iPods and more



[ Reply to This | # ]
10.4: Configure Spotlight to index excluded directories
Authored by: zakwilcox on May 06, '09 11:40:47AM

Thanks for this sub-hint. I'm a pikey who's still on 10.4 and I use a personal wiki to keep random notes. I set up MediaWiki on the system Apache install yonks ago, but it got annoying lately that Spotlight didn't index it. I installed MoinMoin Desktop Edition thinking its plain text files would be the answer but Spotlight didn't seem to index the files except the metadata. Running SetType -t TEXT on the "pages/*/revisions/" trees fixed the problem (which would seem to contradict kirkmc's comment). It's still not perfect - every time you add a new page or edit an existing one you have to set the type on the new file. I'll address this with a cron job to pick those up every hour or so, and just hope that anything I've been editing/creating more recently than that is fresh in my mind anyway :)

I expect that in 10.5 there's a way to write a plugin to feed Spotlight the right data without this hack (perhaps even linkifying the results so they're URLs), or a way to simply configure Spotlight differently, but by now I think I'm better off waiting for 10.6.



[ Reply to This | # ]
use BBEdit
Authored by: Gregory on Nov 10, '05 08:02:56AM

a reminder. you can use other text editors to open and edit the files. I used BBEdit by launching a new instance of BBEdit via the Terminal with a sudo command. it worked perfectly.

sudo path/to/BBEdit.app/Contents/MacOS/BBEdit

this was great for me because I have no knowledge of the Terminal text editors; eg, pico.



[ Reply to This | # ]
use BBEdit
Authored by: Puzo on Nov 15, '05 10:46:39AM

1. BBEdit: Preferences > Tools > Install Command Line Tools
2. Terminal: sudo bbedit <file>
3. For more: man bbedit
4. Enjoy.



[ Reply to This | # ]
It is not just simplicity, a bit safety too
Authored by: Ilgaz on Nov 10, '05 09:39:16PM

Remember Apple said to delete windowserver.plist files to fix a 10.4.1 glitch (and reset firmware) causing windowserver high cpu usage.

As I have Path Finder running, I was lazy and did a clever(!) thing. Went to root directory, clicked find and searched windowserver.plist files. E.g. you may do it too as /etc will be indexed.

Guess what happened? :) 3 results, I blindly deleted, reset firmware, happily booting. I dropped to terminal people!

Look what is named windowserver.plist on Tiger:
cable25-100:/private/etc/mach_init.d ilgaz$ ls W*
WindowServer.plist

Yes, it is "THE" real plist which instructs kernel init to load windowserver.

Thanks to Pacifist, I figured it (booting from repair partition) and put file back.

Just keep such things in mind when you are into "uninstall" party with your uber spotlight. ;)

If I didn't hurry, path finder shows paths of search results. I don't think the initial,fast results of spotlight will show them.




[ Reply to This | # ]
Update for 10.5
Authored by: daveschroeder on Jan 20, '09 07:02:31AM

Hello,

I received this note from user "snipper" with information for 10.5:

I ran into a problem with Spotlight. I couldn't find php.ini because
it's in /private/etc/apache2 and /private is not indexed by
Spotlight.

Meanwhile, the current OS X version is 10.5 and the path of the
file to be altered is:
/.Spotlight-V100/Store-V1/Exclusions.plist

Also, for most users, it's still hard to edit a hidden file that also
needs super user privileges. Maybe you could mention it can be
done by dropping you text editor on Pseudo first. Pseudo let's you
use applications with root rights.



[ Reply to This | # ]
10.4: Configure Spotlight to index excluded directories
Authored by: 321liftoff on Dec 15, '10 01:51:12PM

Was this hint to allow Tiger 10.4 to index the names of files within bsd directories such as "/usr" or to index their content as well?

In Snow Leopard 10.6, the names of system files are indexed by default, but not the content. If I copy a file from a directory such as "/usr" to the desktop, for instance, Spotlight will index the content as well.

If this hint also allows content to be indexed in Tiger 10.4, I don't know what to do for Snow Leopard 10.6. The _rules.plist file no longer exists and the Exclusions.plist file that is in /.Spotlight-V100/Store-V1/ does not have a tree for including directories, just a tree for excluding directories.

Any thoughts?



[ Reply to This | # ]