This hint requires you to have Quartz Composer (it's installed with the Xcode Developer Tools), and Photo Booth (included with all Macs with built-in iSight cameras).
I was playing around with Quartz Composer when I found you could add an iSight as a video source. So I played with that for a while, and found that some of the effects you could create were eerily similar to those seen in Photo Booth. Navigate to Photo Booth in the Applications folder, control-click on it and choose Show Package Contents, and then navigate to Contents -> Resources. There you will find a file named Video.qtz. Make a copy of this, and store it somewhere safe! To be extra-safe, duplicate the Photo Booth application itself as well.
Now, double-click on the original Video.qtz file, and it should open in Quartz Composer. I get errors when I do this, but they don't seem to cause any problems; just 'Continue' past them. Now you should have a large patch grid that creates all the effects that Photo Booth uses. You can proceed to change the effects as you see fit. Once you have made your changes, save the file. On my first adjustment, I had to Save As, but after that, Save seemed to work fine -- possibly a permissions thing? I haven't tried adding new effects, but with a bit more work, I don't see why that wouldn't work.
I haven't outlined how to add / change patches. I really recommend that if you aren't comfortable using Quartz Composer you don't follow this hint. Play with the composer first, see how it works. After making my edits, the previews don't seem to show my changes, but I haven't experimented further with it to see what the problem may be.
I imagine that it may actually be quite easy to break this set-up. Hence you have the back-up Video.qtz file and the backup Photo Booth.app file, in case things go badly wrong.
Good luck and have fun.
Mac OS X Hints
http://hints.macworld.com/article.php?story=20060629032801200