~/Library/Caches/com.apple.SoftwareUpdate/ : 425.5 MB
~/Library/Caches/Desktop/ : 496.5 MB
~/Library/Caches/QuickTime/ : 512.6 MB
Since they're caches, and they'll be rebuilt if needed, I dragged those three folders to the Trash and restarted. I did not encounter any problems, so I emptied the Trash and now my ~/Library/Caches folder is only 154.3 MB.
[kirkmc adds: This Caches folder holds lots of stuff that you really don't need. It's good, if you need to save space, or if you are planning a backup and want it to go faster, to check from time to time, sorting by size, to see which folders take up the most. You may find folders for applications that you no longer user, or, as in this hint, folders that contain much data that you can delete.
I don't know why the SoftwareUpdate cache has so much in it; I've never seen that. The Desktop cache contains Desktop pictures, and if you change often, may grow. As for the QuickTime cache, you can set the maximum size, or turn of caching entirely, in the QuickTime preference pane on the Browser tab.]

