Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

Force fsck on headless servers and remote macs System
Following a system crash I had a server which was failing 'Verify Disk' in Disk Utility on the boot drive. Read on for how to use Safe Boot mode to repair a boot drive over ssh.

I administer a network of Mac clients and OSX server for a small business. I'm usually remote from the office so I look for ways of doing system admin tasks via ssh wherever possible. The main production OSX server, an ageing PowerMac G5 running headless, had crashed recently requiring a forced reboot (getting someone in the office to hold down the power button).

After any crash I usually run diskutil verify /.

In this case diskutil found corruption on the boot disk. To fix this on any Machine I have physical access to I would usually just start up in single user mode (hold down 'S' when booting), then run fsck -fy but in this case I couldn't get to the Machine, and there's no keyboard or monitor attached to it in any case.

For emergencies, I set the Macs up with an install DVD image in a partition. It would be possible to force the server to boot into the install partition, and then use Remote Desktop or ssh to access the box. I was looking for a simpler way.

Reading this Apple support article, I figured out that a cleaner way is to:
  • Login to the remote Mac using ssh with an admin account.
  • Enter sudo nvram boot-args="-x".
  • Enter sudo shutdown -r now, you will be logged off and kicked out.
  • Wait 5-10 minutes as the Mac will reboot in Safe Mode and carry out fsck until the file system is clean.
  • Then login with ssh back into the Mac, run diskutil verify / to double check that the file system is clean.
  • If all is okay set the boot mode back to normal by entering sudo nvram boot-args="" followed by sudo shutdown -r now.
That should produce the same results as sitting at the console directly.

[crarko adds: I haven't tested this one.]
    •    
  • Currently 2.39 / 5
  You rated: 4 / 5 (23 votes cast)
 
[19,074 views]  

Force fsck on headless servers and remote macs | 12 comments | Create New Account
Click here to return to the 'Force fsck on headless servers and remote macs' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Force fsck on headless servers and remote macs
Authored by: zeniam1 on Aug 18, '10 08:01:40AM

I'm missing a step before #1. If your boot drive is not passing fsck, how do you get on it via ssh?



[ Reply to This | # ]
Force fsck on headless servers and remote macs
Authored by: joshewah on Aug 18, '10 09:06:51AM

HFS+ Is a journaled filesystem, so if its anything like ext3 or reiserfs etc, it will replay the journal and act like nothing happend. There may still be errors that can be fixed doing an extensive check, one that if you don't add -f to fsck would skip.



[ Reply to This | # ]
Force fsck on headless servers and remote macs
Authored by: Nem on Aug 18, '10 09:12:35AM

I don't think this is a good idea. I've been a SysAdmin for 10+ years and I know its a bother (and often boring), but you should be sitting in front of a machine watching it fsck. If its not physically convenient, hire somebody else to watch it (that's what interns are for).

---
Nem W. Schlecht
http://geekmuse.net/



[ Reply to This | # ]
Force fsck on headless servers and remote macs
Authored by: bitwise on Aug 18, '10 01:43:33PM

I use this process too. It's just a way of doing a safe boot remotely. Is your concern with Apple's safe boot mechanism? Have you ever seen a safe boot cause harm? I've wondered this myself, but have never seen it happen. I will agree that doing this in the middle of the night with no one local to help if necessary is asking for trouble. But this seems to go for any system restart, and especially one with a less than fully healthy startup drive.



[ Reply to This | # ]
Boot Args
Authored by: lullabud on Aug 18, '10 10:01:45AM
You should probably check what your boot args are before overwriting them. Run this command before beginning then replace the boot args in the last step with what you find:

nvram boot-args

[ Reply to This | # ]
Force fsck on headless servers and remote macs
Authored by: joelbruner on Aug 18, '10 10:36:25AM
Always a multitude of ways, so if I might chip in my 2 cents on the comments

This does something similar, except it boots to single user mode, using the .profile for root check it is single user mode, runs fsck, mounts the drive, writes the result to a file, then deletes the .profile, sure the fsck could be more elegantly redirected or tee'd but take it for what it is

#modify nvram for single user
nvram boot-args="-s"

#modify Single User mode for auto fsck -fy
echo 'if [ $HOSTNAME = localhost ] || [ -z $SECURITYSESSIONID ]; then clear; echo "Starting File System Check in 5 seconds, Control-C to quit"; /bin/sleep 6; fsckResult=$(/sbin/fsck -fy); echo "$fsckResult"; /sbin/mount -uw /; echo "$fsckResult" >> /var/log/fsck.log; /bin/rm /var/root/.profile; nvram boot-args=""; echo "Rebooting in 10 seconds."; /bin/sleep 10; /sbin/shutdown -r now; fi' > /var/root/.profile

reboot

the reboot at the end is to immediately reboot the client computer, useful for sending via ARD, but be careful all commands must be run as root of course :)

[ Reply to This | # ]
Force fsck on headless servers and remote macs
Authored by: ernst_mulder on Aug 19, '10 12:07:56AM

Your (nice btw) method is missing a "touch /var/log/fsck.log". Appending to /var/log/fsck.log using >> will result in an error if /var/log/fsck.log does not exist.



[ Reply to This | # ]
Force fsck on headless servers and remote macs
Authored by: zeniam1 on Aug 19, '10 09:33:51AM
Since when?

% ls -l foobar
ls: foobar: No such file or directory
% echo 'hello' >> foobar
% ls -l foobar
-rw-r--r--  1 howes  staff  6 Aug 19 12:32 foobar
%


[ Reply to This | # ]
Force fsck on headless servers and remote macs
Authored by: chuckadile on Oct 03, '12 01:35:50PM

I have found this extremely useful. I want to write this into an applescript that can run on a local machine like this:

tell application "Terminal" to do shell script "sudo echo 'if [ $HOSTNAME = localhost ] || [ -z $SECURITYSESSIONID ]; then clear; echo "Checking File System. The Computer will reboot in 1 minute..."; /bin/sleep 6; fsckResult=$(/sbin/fsck -fy); echo "$fsckResult"; /sbin/mount -uw /; echo "$fsckResult" >> /var/log/fsck.log; /bin/rm /var/root/.profile;cd /Library/Preferences; plutil -s *.plist; rm `plutil -s *.plist | grep "plist:" | sed "s/://g"`; nvram boot-args=""; echo "Rebooting in 10 seconds."; /bin/sleep 10; /sbin/shutdown -r now; fi' > /var/root/.profile " with administrator privileges

However since the whole command needs to be inside quotations the the quotations inside the command prevent it from compiling
I have gotten this to work:

tell application "Terminal" to do shell script "sudo echo 'if [ $HOSTNAME = localhost ] || [ -z $SECURITYSESSIONID ]; then clear; /bin/sleep 6; fsckResult=$(/sbin/fsck -fy); /sbin/mount -uw /;/var/log/fsck.log; /bin/rm /var/root/.profile; nvram boot-args=; /bin/sleep 10; /sbin/shutdown -r now; fi' > /var/root/.profile " with administrator privileges

But not as nice without messages to let the user know what's going on.

Any suggestions?



[ Reply to This | # ]
Force fsck on headless servers and remote macs
Authored by: bitwise on Aug 18, '10 01:57:31PM

Couple additional points. In my experience a safe boot disables airport/wireless. So if this is how the remote mac connects, it will no longer be accessible. If you need/wish to reset this nvram variable while at the machine without using terminal, I believe a zap of the pram will do the trick.



[ Reply to This | # ]
Force fsck on headless servers and remote macs
Authored by: pglock on Aug 26, '10 06:18:12AM

Interesting point about airport. From my experience all of the macbook and macbook pro laptops we have will boot up with airport working. I seem to remember some iBooks being a bit problematical, but we don't have any in the network any more.

Reading some of the other comments, I like the idea of saving the results of fsck (the command that's one typo away from being rude) for later analysis.

I'm a little surprised that there isn't a more elegant way of doing this in ARD, including a way of capturing a log of the results.

---
Too lazy to think of an original signoff...



[ Reply to This | # ]
Force fsck on headless servers and remote macs
Authored by: pglock on Mar 11, '11 01:49:15PM
just a quick update for Snow Leopard:
diskutil verify / 
is now
diskutil verifyVolume / 
---
Too lazy to think of an original signoff...


[ Reply to This | # ]