Mac OS X's quarantine feature not only sets a special quarantine bit on files you download - so it can alert you the first time you open them, and so its built-it anti-malware feature can scan for certain threats - but it also keeps a list of the files you have downloaded. OS X Daily pointed out the command you can run in Terminal to see the full list of these downloads. Run this command:
sqlite3 ~/Library/Preferences/com.apple.LaunchServices.QuarantineEventsV* 'select LSQuarantineDataURLString from LSQuarantineEvent'
sqlite3 ~/Library/Preferences/com.apple.LaunchServices.QuarantineEventsV* 'select LSQuarantineDataURLString from LSQuarantineEvent' | sort
sqlite3 ~/Library/Preferences/com.apple.LaunchServices.QuarantineEventsV* 'delete from LSQuarantineEvent'
[kirkmc adds: Note that, as a commenter mentions below, this won't be every file you've downloaded, but those files you downloaded with applications that set the quarantine bit.]
Mac OS X Hints
http://hints.macworld.com/article.php?story=20120717025305161