Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

10.4: Create Spotlight indexes for networked volumes System 10.4
Tiger only hintI was quite disappointed when I read Friday evening that Spotlight would only index volumes physically attached to my machine (e.g., FireWire or internal). I have a Linux server in the basement with lots of data that I was hoping to index and mine with Spotlight. This evening, while digging around, I found a command line utility called mdutil. It has a switch, -i, to enable indexing on a volume. So I just went ahead as root (sudo bash) and ran:
 mdutil /Volumes/ldm -i on
Then I checked the status with
 mdutil /Volumes/ldm -s
And, behold ... I got back the following:
 /Volumes/ldm/:
   Status: Indexing Enabled
A second later, the Spotlight Icon started pulsating. Half an hour later, it stopped. I went to a Finder window, typed a query, and selected the network volume ldm. To my delight, matches were coming back instantaneously on files deep down in the file hierarchy on that volume. I have not checked whether newly added files are indexed on the fly, too, and I do not know where the .Spotlight-V100 file is stored (not in the root of the volume in question it seems).

I thought that others may give it a try to confirm whether this is working as expected. The volume I tested this with was an AFP volume served via NETATALK (v. 2.0.2) by my Linux server (a x86 debian box). Unfortunately, it appears to fail with SAMBA. I have not checked NFS. One caveat: you must rerun the mdutil command each time the volume is mounted.

Hope it helps someone!
    •    
  • Currently 3.50 / 5
  You rated: 4 / 5 (8 votes cast)
 
[89,124 views]  

10.4: Create Spotlight indexes for networked volumes | 31 comments | Create New Account
Click here to return to the '10.4: Create Spotlight indexes for networked volumes' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
10.4: Create Spotlight indexes for networked volumes
Authored by: schneb on May 04, '05 11:35:28AM

I hope they change this and keep a hidden index on each network volume. No use having an enterprize system if Spotlight only works locally.



[ Reply to This | # ]
10.4: Create Spotlight indexes for networked volumes
Authored by: t0fukrunch on May 04, '05 12:06:06PM

The databases for network volumes with indexing enabled seem to be stored in /var/db/Spotlight-V100/.



[ Reply to This | # ]
10.4: Create Spotlight indexes for networked volumes
Authored by: extra88 on May 04, '05 12:45:49PM

Storing the index on the network volume itself would require each user to have read/write access to the index which may not be desirable. Also, there would be no way to control what contents of the index are revealed to a given user. What if there are folders on the network volume you're not supposed to have access to? You could then learn about the contents of those folders.

To really do it right, the indexing services on the client and on the server need to talk to each other so the server can keep a complete index of what's on it but only tell the specific client user about the files they're allowed to see.



[ Reply to This | # ]
10.4: Create Spotlight indexes for networked volumes
Authored by: ericdano on May 04, '05 01:57:26PM

This might be something Tiger server does. I imagine on non-Apple servers your stuck.



[ Reply to This | # ]
10.4: Create Spotlight indexes for networked volumes
Authored by: koalacurioso on May 12, '05 11:36:10AM
-p Publishes the local copies of the metadata stores for the volumes
indicated to their real drives.


This is a attribute of mdutil command.
May this is a way to keep a copy of the metadata on the remote volume too. Isn't it?

[ Reply to This | # ]
10.4: Create Spotlight indexes for networked volumes
Authored by: whee on May 04, '05 12:14:38PM

Can't running mdutil be automated with a folder action and a very small script?



[ Reply to This | # ]
10.4: Create Spotlight indexes for networked volumes
Authored by: rufo on May 04, '05 12:42:39PM

Does Spotlight preserve the index between mountings of the volume? (e.g., does the fact that it comes up as a non-indexed volume every time you mount it cause Spotlight to have to re-index every time you mount?)



[ Reply to This | # ]
10.4: Create Spotlight indexes for networked volumes
Authored by: ldm on May 04, '05 01:15:48PM
The index is preserved. It is saved under a private tree (you can do lsof | grep Volumes To see which files are open after enabling the indexing and you will see the exact location of the DB. The spotlight index is not stored on the networked volume, it is stored on the local host.

---
--
Laurent

[ Reply to This | # ]

Local Index Database
Authored by: lullabud on May 05, '05 12:38:36PM

Stored locally? That's would mean each system would need to index the host causing severe strain on the network. *sigh* I suppose I'll give it a whirl anyways and see if I can come up with anything...



[ Reply to This | # ]
publishing metadata to the network share
Authored by: lullabud on May 06, '05 03:05:03PM
From the man page:
-p Publishes the local copies of the metadata stores for the volumes indicated to their real drives.
I just tried sudo mdutil -p /Volumes/networkshare/ on a linux hosted smb share and it did indeed put the spotlight index in the root of the shared folder, returned an error about the indexing status, then started indexing the disk. I'm waiting for spotlight to finish indexing everything right now to see if it works.

My guess is that if spotlight finds an index on a networked drive it'll use it and update it accordingly whenever accessing data on that disk. If so, my job just got a LOT less stressful.

[ Reply to This | # ]
Index status flag not persistent
Authored by: lullabud on May 06, '05 03:24:13PM
Well, it appears to me as though you'd have to enable indexing each time you mount a network share. Connecting to an SMB share which has an index on it already then checking the indexing status with sudo mdutil -s /Volumes/sharename/ says that indexing is disabled. If I enable it, unmount the share and remount it the flag will be reset, and I need to turn indexing on again. Using the spotlight search gui doesn't search the network drive's index, but using mdfind -onlyin /Volumes/networkshare/ does. I'd sure like to see a howto on this.

[ Reply to This | # ]
Possible problems?
Authored by: dzurn on May 04, '05 01:18:49PM
I don't have Tiger yet, but from the Ars Technica discussion of Spotlight, the index for a file is automatically updated when the Tiger Kernel is used to save the file. That means that potentially other user's modifications to a network share would not be updated automatically (and it's not clear if your own modifications would, if the saving is done other than by the kernel).

And it appears each user would have to reindex the shared volume for themselves. Keep in mind that you can't query Spotlight while it's updating an index, so I don't think you could use Spotlight on your local drive while the network drive indexes were being updated.

Intriguing, but I'd love to see more exploration of this topic for sure!

---
Madness takes its toll.
Please have exact change.

[ Reply to This | # ]

Re: Possible problems?
Authored by: ldm on May 04, '05 03:36:41PM

You are correct on all counts. If a share is modified by another machine/user the change would not be visible. This is not meant to serve that purpose though. In this case, I have a 'personal' file server that holds quite a bit of data and I am the sole person accessing it (it is my home directory on the server). I still have problems if I access the share in read/write from two distinct machines. I guess that to resolve this one the only option is to use Tiger Server (and throw away the linux box in the process to get a Mac...) . I am told that with Tiger Server, the indexing happens on the server... which of course solves the issue since the client forwards its queries to the server and all is well.



---
--
Laurent



[ Reply to This | # ]
What about NFS mounts?
Authored by: bankshot on May 04, '05 05:41:59PM
I have a few shares automounted from my FreeBSD server, and I sure would like to index those in Spotlight. I was poking around with the md* commands and tried this the other day, but it didn't work:

% sudo mdutil -i on /private/var/automount/home/mp3
/private/var/automount/home/mp3:
        Could not set indexing status for volume.

I've tried enabling root access in the exports on my server (that gives me the willies) but it didn't help. Mounting the share directly using the mount_nfs command (bypassing the automounter) didn't help either. I suppose I could try remounting them as Samba shares to see if there's any inherent difference. Don't know why there would be, though.

[ Reply to This | # ]

What about NFS mounts?
Authored by: MattHaffner on May 04, '05 05:46:49PM
Well, I don't know the details of the setup, but our uber-admin just e-mailed me this in response to some Spotlight tips I forwarded:

I just "finished" installing 10.4 on one of the "clustered" Macs and when I tried using splotlight I was amazed to find that it had already indexed my (NFS mounted) home directory.


[ Reply to This | # ]
Indexing works, but no results
Authored by: markju on May 12, '05 09:56:33AM
I have several NFS mounts (from Solaris and OSX) and tried the following (assuming the NFS share is mounted as /nfs):
# mdimport -f /nfs (takes a while...)
# mdutil -i on /nfs
/nfs:
    Status: Indexing Enabled
# mdutil -E /nfs
/nfs:
    Volume index removed. Index will be rebuilt automatically.
Spotlight starts indexing, but no results show up.

[ Reply to This | # ]
10.4: Create Spotlight indexes for networked volumes
Authored by: mahakali on May 06, '05 12:51:32AM

Can this command be used for disabling indexing on certain volume?



[ Reply to This | # ]
10.4: Create Spotlight indexes for networked volumes
Authored by: jkcarter on May 06, '05 11:25:59AM
You certainly can use this command to turn off indexing:

mdutil -i off /path/to/volume

Alternatively, you can set up folders/volumes for Spotlight to ignore in System Preferences. Just go to Spotlight->Privacy, and add folders/files/volumes to ignore.

[ Reply to This | # ]

10.4: Create Spotlight indexes for networked volumes
Authored by: phdonnelly on May 06, '05 11:23:22PM
Hmm, I've got pretty much the same setup as you (Debian x86 box running netatalk 2.0.2 serving up a couple shares), but whenever I try to enable indexing on them I get "Could not set indexing status for volume". I can get the indexing status just fine(disabled), which is more than I can do for my NFS/Samba shares, but still can't set it. Would you mind posting the pertinent parts of your afpd.conf and AppleVolumes.default files?

[ Reply to This | # ]
10.4: Create Spotlight indexes for networked volumes
Authored by: phdonnelly on May 06, '05 11:31:28PM

Nevermind, found the solution to my own problem; the AFP shares have to be read/write and not read-only or you can't enable indexing. Seems a little strange to me as all the spotlight info is stored locally.



[ Reply to This | # ]
10.4: Create Spotlight indexes for networked volumes
Authored by: jdb8167 on May 08, '05 03:55:58PM

I went to a Tiger Tech talk from Apple in Boston last year where Spotlight was a big part of the presentation. Afterward there was Q&A and a lot of questions were asked about Spotlight indexing servers and read-only media. At the time, the Apple evangelist, Xavier Legros seemed unsure on how Spotlight would end up working. The developers at the talk brought up the bandwidth issue and also the fact that indexing remote volumes and leaving around large index files was sure to get system administrators up in arms as well.

It looks like Apple decided to be conservative, leaving the functions available from the command line but disabling it altogether from the UI. It is also interesting that Apple seems to have taken the feedback seriously because at the time, I got the distinct impression that they really didn't see a problem with indexing remote volumes automatically. Several of us made a strong argument that indexing from every client would be a bad way to get Tiger deployed into the enterprise.



[ Reply to This | # ]
10.4: Create Spotlight indexes for networked volumes
Authored by: areyouwishing on May 12, '05 11:12:17PM

It would be nice if Tiger Server kept track of all incoming and outgoing files and indexed as needed and kept that index in a place that the client would know where to look. This prevents unnecessary traffic and it keeps networked volumes for the client up-to-date. I know a lot of IT dept. that would switch for that feature alone.



[ Reply to This | # ]
10.4: Create Spotlight indexes for networked volumes
Authored by: somynona on May 18, '05 02:06:37PM
The same comment was just made on Apple's discussion forum:

[ Reply to This | # ]
10.4: Create Spotlight indexes for networked volumes
Authored by: chargrill on May 18, '05 04:44:52PM

i just found this hint and read through the comments and was saddened by the reports that this wouldn't work for samba mounted shares. i decided to try it anyhow, and was joyous when it worked. i then decided it was too much work to do this every time my computer is re-connected to the "work" network, so i wrote a script and put it in root's crontab. (note: you may be not want to do this, as it involves a lot of root activity).

here's the shell script:


#!/bin/sh

echo "`date +'%Y%m%d %H:%M:%S'` - checking for VOLUME_NAME" >> /var/root/reindex.VOLUME_NAME.log

if [[ `df -k | grep VOLUME_NAME` != "" ]]
  then
    echo "`date +'%Y%m%d %H:%M:%S'` - VOLUME_NAME mounted" >> /var/root/reindex.VOLUME_NAME.log
    if [[ `mdutil -s /Volumes/VOLUME_NAME | grep Enabled` == "" ]]
      then
        echo "`date +'%Y%m%d %H:%M:%S'` - reindexing VOLUME_NAME" >> /var/root/reindex.VOLUME_NAME.log
        mdutil -i on /Volumes/VOLUME_NAME
    fi
fi

i chmod +x'd the script, put it at /var/root/reindex.VOLUME_NAME, and added the following to root's crontab:


*/5 8-18 * * 1-5 /var/root/reindex.VOLUME_NAME > /dev/null 2>&1

this basically checks every five minutes of the workday if it needs to re-enable indexing on my SMB mounted share and turns it on if it isn't already. and keeps a log on its activities for debugging purposes. i've already removed those lines as i don't need to keep an eye on it after i know it's setup and working properly.

note: figuring out how to do all this as root is left as an exercise to the reader. i don't want to be responsible for anyone hosing their system :)



[ Reply to This | # ]
10.4: Create Spotlight indexes for networked volumes
Authored by: JesperHansen on May 31, '05 08:37:06PM

Instead of polling each minutes, its now possible with Tigers new launchd to only run commands when a specific folder changes (like /Volumes).

Here is a small example of how I enable indexing when I mount my "stuff" smb volume.

Two files are involved, first a Launch Daemon config file:
/Library/LaunchDaemons/enableindexing.plist


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>Label</key>
        <string>enableindexing</string>
        <key>ProgramArguments</key>
        <array>
                <string>/usr/local/bin/enableindexing.sh</string>
        </array>
        <key>WatchPaths</key>
        <array>
                <string>/Volumes</string>
        </array>
</dict>
</plist>
This basically tells the daemon to run the script "/usr/local/bin/enableindexing.sh" when something in /Volumes is changed.

My "/usr/local/bin/enableindexing.sh" scripts look like this:


#!/bin/sh
sleep 1 # let things settle down a bit
if mdutil -s "/Volumes/stuff" | grep -q "Indexing Disabled" ; then
	mdutil -i on "/Volumes/stuff" >/dev/null
fi

If your smb volume is not called stuff, you might want to change the script a bit :)
The "sleep 1" line is enough on my machine (G5), but maybe a longer sleep is needed on a slower machine.
Also remember to run "chmod +x /usr/local/bin/enableindexing.sh"

To tell the Launch Daemon about this new script, either reboot or do "sudo launchctl load /Library/LaunchDaemons/enableindexing.plist"

[ Reply to This | # ]

10.4: Create Spotlight indexes for networked volumes
Authored by: rlaan on Jun 13, '05 08:11:52AM

Very interesting, but what happens if the volume is an automount that mounts on login? e.g. in a Workgroup Environment

Ruben

---
The box said, Windows 95 or better, so I bought OS X.



[ Reply to This | # ]
10.4: Create Spotlight indexes for networked volumes
Authored by: regulus on Jul 06, '05 05:08:45PM

You are KING! That works great!

This tip is so good it should be its own hint.



[ Reply to This | # ]
10.4: Create Spotlight indexes for networked volumes
Authored by: matteus on Jul 12, '05 04:10:05PM
This works great, and I can verify that my SMB shared files are indexed with:
mdls /Volumes/volumename/filename
However neither the spotlight menubar interface or mdfind searches the indices that contain these SMB volumes. This is a great tip to keep my SMB volumes indexed, but if I can't search these indices then what's really the point? Any help appreciated.

Thanks,
matteus

[ Reply to This | # ]
10.4: Create Spotlight indexes for networked volumes
Authored by: regulus on Jul 12, '05 04:27:18PM

To search on them you first cmd-f to bring up a find window and then click "servers"... that will search any mounted drives.



[ Reply to This | # ]
10.4: Create Spotlight indexes for networked volumes
Authored by: DMB33Z on May 23, '05 06:14:57AM

I get this when i run that command:
The-Fifth-Element:/Volumes root# mdutil /Volumes/ldm -i on
/Volumes/ldm:
Could not set indexing status for volume.

.... my mounted voulme is actually an Server 10.4 Shared folder could that be the problem?



[ Reply to This | # ]
10.4: Create Spotlight indexes for networked volumes
Authored by: mitra on May 29, '05 09:48:35AM

It seems you can get this error on Spotlight even on your "/" Volume e.g.

sudo mdutil -s /
/:
Error: Could not get indexing status for volume.

After I set the indexing on for one of my firewire drives, everything to do with Spotlight seemed to break, and it no longer finds any search results for anything and gives errors like the one above. So beware, it seems that its not as easy as suggested to make Spotlight be aware of external drives

P.S. Any help to fix this bug, or diagnostic tools - files to look into etc appreciated.



[ Reply to This | # ]