In Safari, hold down Command+Shift and then drag a Java applet to its own window.
Here's a simple Rubik's Cube applet to try it with. You'll notice when you press the keys an enclosing window appears, which can then be moved off on its own.
[crarko adds: It's actually pretty slick, how that works. I'm in Safari 5.1.1; I couldn't say how long this has been available.]
Mac OS X Hints
http://hints.macworld.com/article.php?story=20111028201809489