To connect to the vnc server, type vncviewer localhost:2 at an X11 terminal prompt, and enter the password when prompted. You will then have your X11 desktop, and it should be running twm as its window manager. You can change this (I set mine to icewm, installed through Fink) by editing ~/.vnc/xstartup and changing the line:
twm &to (for example)
icewm &to use the Ice Window Manager. I have also used KDE and Gnome with this method, all installed from Fink, but icewm seems to be the most responsive. So why bother with vnc on localhost?
Well, the state is retained when you close the viewer window, which means you can leave all your X11 apps running without cluttering your screen, and when you need them again, simply reconnect to localhost:2 and there they are. You can also, using SSH, access your desktop from anywhere in the world, by using port forwarding:
ssh -C -L 5902:localhost:5902 user@mac.ip.addressThe -C enables compression over the connection, which will help speed things up (replace user and mac.ip.address with the proper values for your machine). You then connect to the vnc server using a viewer on the local machine, again with the command vncviewer localhost:2, and there's your home desktop, with the X11 apps still running as you left them.
Hope this helps!
[robg adds: I modified this hint just a bit to refer to "X11" instead of the originally written "X" where appropriate, in order to reduce the confusion between the two environments. To learn more about X11, check out the XFree86 homepage and Apple's X11 page.]

