If you want to prevent this but use the plug-in in Internet Explorer, you can delete the Speed Download plug-in out of the "Internet Plugins" folder. If you want to download something in IE with Speed Download, you just have to control-click the link and select "Download with Speed Download."
[robg adds: I have no experience with this as I don't use IE or Speed Download, but I thought this may be of interest to others.]

