Disclaimer: The following is a highly technical hint.
Summary: This hint is for Network Engineers who want their firewalls to accept VPN connections from standard OS X L2TP / IPSec clients (should also work for Windows and Linux clients). If you are not a network engineer, but are having trouble connecting to one of these devices, you can also forward this tip to your company's "firewall person," so that they can fix it.
Problem: A Cisco ASA or PIX firewall can be a VPN server, but a basic VPN configuration will not allow the default OS X L2TP/IPSec client to connect, even though the Cisco client will. It may not be convenient to distribute the Cisco VPN clients, or your users may not wish to use them.
Solution:
Step 1: First use the Cisco documentation to setup the VPN on the ASA or PIX. Make sure it works as desired with the Cisco client first.
Step 2: DefaultRAGroup
In Panther, and on Windows 2k/XP, the VPN client does not allow the user to specify which tunnel group they belong to, so use the DefaultRAGroup to setup all your VPN rules instead of a custom named group. By default, any client not specifying a group goes into DefaultRAGroup. Tiger allows you to specify a group.
Step 3: 3des, sha, group2
I tried to use aes encryption, but it didn't seem like OS X likes that type, so make sure you use 3des encryption, sha, and Diffie Hellman group2. On an ASA 5520 with 8.0 code, it looks like the following:
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
Step 4: transform set mode should be transportcrypto ipsec transform-set VPNTRANS mode transport
Step 5: mschap passwordsusername thisuser password thatpassword mschap
Step 6: Client configuration
Mac OS X Hints
http://hints.macworld.com/article.php?story=20070827135109248