I had thought that, too. But all the folders now just contained orphaned .jpg, .txt and, most importantly, .pdf files which I then had to manually re-add (well, at least the .pdfs) to the respective music folders. These of course live now in a folder called "Music".

You're right, it was all just jpg files (checked with grep for mp3 and m4a). For some reasons, all my PDF files did end up in the right places, but only Apple-supplied ones showed up in iTunes.

For the record, there were 350 freaking folders left and I only had 500 albums. I was expecting some stuff to be left behind from others' comments, but not this much.

For people who have the same problem as me, I think it's okay to junk those folders. Be careful you don't erase the vital folders now actually holding your stuff though. They are:

Audiobooks, Automatically Add to iTunes, iPod Games (only if you own a scroll wheel iPod), Music, Movies, Podcasts, and TV Shows. There are three more folders if you own an iPhone/iPod Touch: Mobile Applications, Ringtones, and Voice Memos.