If you'd prefer Safari never remembered favicons, here's a simple way to do just that, while also removing any currently-stored favicons. Quit Safari if it's running, then open Terminal (in Applications » Utilities) and enter these commands (don't enter the $; that's just the prompt):
$ cd ~/Library/Safari
$ rm WebpageIcons.db
$ ln -s /dev/null WebpageIcons.db
The first command sets the proper working directory, the second removes the current favicon database, and the third recreates the database file, but links it to /dev/null, so that any output written to the file is summarily discarded.
Mac OS X Hints
http://hints.macworld.com/article.php?story=2009042105513528