Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

Mounting a shared file system for Xgrid Network
Xgrid is Apple's cluster computing system built into the OS. Typical jobs that use Xgrid are data intensive tasks that need remote access to input to files and/or large databases. Unfortunately, for security reasons, on a distributed Xgrid network the jobs running on agents can't access /Volumes and thus can't access remote files systems that are mounted in the usual way. So where can a remote file system be mounted that Xgrid agents can access on demand?

One of the few places on an agent computer that the Xgrid job is allowed to read from is /tmp so you mount it in /tmp. At first this seems unsuitable since any directory created for the mount point won't last past a reboot (/tmp is erased every reboot). But an unexpected feature of the mount process on OS X as opposed to Linux is that it will auto_create a directory for you if it is missing when the mount command is issued. This makes this possible to work using just the standard automounter.

For example, suppose an NFS disk server named myDiskServer on the local network is exporting a directory found on its disk at /Volumes/MacHD/export_directory. Then to import this shared NFS file system just append the following line to the end of /etc/auto_master on each agent computer:
/private/tmp/xsfs myDiskServer.local:/Volumes/MacHD/export_directory
Then after you execute 'automount' (or reboot) you will find you can access the disk server from an Xgrid job at /tmp/xsfs/.

Note this assumes that the NFS disk server is allowing exports to that agent computer. How to set up a disk server itself is beyond the scope of this hint and varies from Linux, to Windows, to Mac OS X. But if you are using an OS X Snow Leopard Server as the fileserver then you definitely need to check out this recipe to avoid pitfalls .

Alternatively, instead of NFS, there are previous hints for SMB and AFP automounting. (But heed this precaution for SMB) Also here is an informative discussion of why NFS may be sometimes better for Xgrid. But, regardless of the protocol, the key point is that mounting the filesystem in /tmp will work for Xgrid.

I'd like to give a hat tip to the new TenGrid.com wiki for the solution to this.

[crarko adds: I haven't tested this one.]
    •    
  • Currently 2.42 / 5
  You rated: 1 / 5 (12 votes cast)
 
[7,180 views]  

Mounting a shared file system for Xgrid | 3 comments | Create New Account
Click here to return to the ' Mounting a shared file system for Xgrid' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Mounting a shared file system for Xgrid
Authored by: Shawn Parr on Jun 10, '10 08:50:26AM

We recently ran across this on a Podcast Producer system setup. The xgrid workers didn't have permissions to the file system as they were pointed at something under /Volumes.

We changed it to be pointed under /Network/Servers/<servername>/<path to file storage area> which should be a valid path on any machine the NFS server has exported paths to. Well at least if the server hosting the files is running Mac OS X server which is what we are running.



[ Reply to This | # ]
Mounting a shared file system for Xgrid
Authored by: SOX on Jun 10, '10 04:32:22PM

Whether or not /network is allowed depends on the sandbox seatbelt in use. For common xgrid configurations /network is not allowed by the seatbelt in 10.5 and 10.6



[ Reply to This | # ]
Mounting a shared file system for Xgrid
Authored by: Shawn Parr on Jun 11, '10 07:14:30AM

I looked into it and now see that we were both right.... :)

Podcast Producer can run xgrid processes by specifying a user account. That account will have access to anywhere it should based on what directory system it is pulled from. In our case we have an Active Directory account that has full permissions to directories being exported. The agents then run as that user and can read and write from that space. Although as I mentioned before, even if the share shows up under /Volumes, you need to point it at /Network otherwise it doesn't work right.

Other processes use 'nobody' as the user, which is why you needed to use tmp.



[ Reply to This | # ]