My Mac and other computers all have RFC 1918 (private internets, i.e. 10.0.0.x, etc.) static IP addresses and sit behind an OpenBSD 3.7 box running the pf firewall. pf performs stateful packet-filtering and network address translation (NAT). The OpenBSD firewall gets a statically assigned public IP address from a DHCP pool on its external interface.
I had never had any problems using Software Update, until I installed Tiger on my G5. A day after installation, Software Update stopped working -- it would just hang and then time-out. I searched Apple's forums and noticed many other people experiencing a similar problem. Messages on the forums suggested that some people had luck when they switched their Macs to DHCP. The quickest way I could get a DHCP address for my Mac was to connect it directly to the Internet, outside of my OpenBSD firewall. When I did this, Software Update worked. But whenever the Mac was behind the OpenBSD firewall with a static IP address, Software Update didn't work. I assumed this was an OS bug, especially since Software Update used to work, nothing had changed in my firewall configuration, and tcpdump did show traffic between my Mac and swscan.apple.com when it tried to do a Software Update. 10.4.1 and 10.4.2 came out, and I still could not run Software Update (unless my Mac was connected directly to the Internet via DHCP).
To add to my troubles, I also had issues with connecting to the iTunes Music Store. I had no problems connecting in the past ... until iTunes 5 came out. The day I installed iTunes 5, any connections to the Music Store would hang and then time out. Again, I assumed this was some new, frustrating bug -- this time in iTunes.
Then I remembered some posts I had come across last year discussing an apparent RFC 1323 (TCP Extensions for High Performance) implementation problem specific to apple.com. So I modified my OpenBSD pf.conf firewall ruleset, and changed my scrub rules to:
scrub on $ExtIF from any to swscan.apple.com random-id
scrub on $ExtIF from swscan.apple.com to any random-id
scrub on $ExtIF random-id reassemble tcp
After doing this, both Software Update and iTunes Music Store worked!
Mac OS X Hints
http://hints.macworld.com/article.php?story=20050920032541132