When viewing a movie in full screen mode under Mac OS X, VLC freezes if you pause the movie for a minute or two. Because you are in the full screen mode, you cannot force the program to quit and you cannot reach out for the Dock. One solution to this that I have tried out successfully is to construct a one-line script:
kill -9 `ps -auwx | grep VLC/VLC | grep -v grep | awk '{print $2}'`
And then bind that script to a keystroke, for example, in QuicKeys. That way, you will be able to quit VLC even when you are in the full screen mode.
Mac OS X Hints
http://hints.macworld.com/article.php?story=20030920052727284