I have always deleted preference panes, by going into the various preference pane folders (/Library/Preference Panes and ~/Library/Preference Panes) and deleting the preference pane that I wanted to get rid of.
I found out yesterday that right-clicking or Control-clicking on the preference pane you want to delete brings up a contextual menu saying "Remove [preference pane name] preference pane".
I don't know how many people know about this but it was new to me.
[kirkmc adds: I've known this for a while, but have also found a handful of cases where this doesn't work. I don't currently have any preference panes installed that won't display this menu, but I suspect it has something to do with whether they are in /Library or not...]

