I found this hint on Mozilla's bug tracker (bugzilla), listed as bug #11459. Within this bug, you'll find comment #72:
I'm not sure if this applies to all builds of Mozilla or just the Mac OS X build, but adding a file called user.js to your profile folder (the same folder which contains prefs.js) and placing the following text in that file will fix this bug:user_pref("network.protocol-handler.external.mailto", true);From what I understand, this also works for ftp and nntp. This was pointed out to me by someone in the netscape.public.mozilla.macosx newsgroup.Save the file as a plain text file, and place it in your Mozilla profile folder (buried in your Library -> Mozilla -> Profiles -> username -> some_mess.slt directory). Now quit and relaunch Mozilla, and voila, mailto links open with you default mail client.
Mac OS X Hints
http://hints.macworld.com/article.php?story=20020502091605391