So I searched for the preference file that was modified whenever I changed display settings. Low and behold: com.apple.finder.plist. It (like most plist files) is stored in your user's Library/Preferences directory. These plist files are just plain text, so you can edit them using TextEdit, or BBEdit, or SubEthaEdit, or any text editor. IN the section that begings with this:
<key>DesktopViewOptions</key>
<dict>
Find this property:
<key>FontSize</key>
<integer>10</integer>
Now just edit the integer to whatever size you want (9 was small enough for me; 8 was getting hard to read). Save the file, re-launch the Finder (Option-click and hold on its icon in the Dock), and it's done!

