Launch Adobe Reader, and then open its Preferences. Under the Internet section, update your network speed to LAN (or whatever's appropriate) from 56kbps (default).
This really speeds up the Adobe Reader PDF viewer plug-in from within Safari.
[robg adds: In my testing, I didn't notice a huge speed increase, but setting the preferences to your proper connection speed seems like a generally good thing to do.]
Mac OS X Hints
http://hints.macworld.com/article.php?story=20060403171152376