My solution was to make two copies of my private key (well three, actually, as I use Putty on the PC and it requires a key in its own format): one on the drive itself, and one in a sparseimage on the drive. That way, linux had access to the key on the VFAT partition, and I can mount the sparse image while I am at my Mac.
The next issue was how to get Mac OS X, or any unix, to use a key on a USB drive. It turns out that this is actually very easy. In a Terminal window, do the following:
$ cp /etc/ssh_config ~/.ssh/config
$ pico ~/.ssh/config
Now find this line:
# IdentityFile ~/.ssh/identity
Remove the # to uncomment it, and change the path to that of your key on the USB drive. Finish by logging out then back in. If everything is set up right, your public key authentication will not work without your USB drive and/or sparse image mounted. As soon as your drive is mounted, authentication works like expected.
I decided to go "whole hog" with this, and the sparse image is encrypted, my private key is DSA with -b 2048 used, and a long randomly generated passphrase. Obviously, one needs to be very careful what Windows PC one puts this drive in. I have the luxury of being Admin of the PCs I have to deal with, and I went to great lengths to ensure I could very quickly cleanse them when necessary.

