Summary: This hint is for Network Engineers who want their firewalls to accept VPN connections from standard OS X L2TP / IPSec clients (should also work for Windows and Linux clients). If you are not a network engineer, but are having trouble connecting to one of these devices, you can also forward this tip to your company's "firewall person," so that they can fix it.
Problem: A Cisco ASA or PIX firewall can be a VPN server, but a basic VPN configuration will not allow the default OS X L2TP/IPSec client to connect, even though the Cisco client will. It may not be convenient to distribute the Cisco VPN clients, or your users may not wish to use them.
Solution:
Step 1: First use the Cisco documentation to setup the VPN on the ASA or PIX. Make sure it works as desired with the Cisco client first.
Step 2: DefaultRAGroup
In Panther, and on Windows 2k/XP, the VPN client does not allow the user to specify which tunnel group they belong to, so use the DefaultRAGroup to setup all your VPN rules instead of a custom named group. By default, any client not specifying a group goes into DefaultRAGroup. Tiger allows you to specify a group.
Step 3: 3des, sha, group2
I tried to use aes encryption, but it didn't seem like OS X likes that type, so make sure you use 3des encryption, sha, and Diffie Hellman group2. On an ASA 5520 with 8.0 code, it looks like the following:
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
Step 4: transform set mode should be transportYou have to set the transform set mode to 'transport' to work with OS X (or Windows). It looks like the following on my box:
crypto ipsec transform-set VPNTRANS mode transport
Step 5: mschap passwordsI was not using an Authentication server in my setup, instead creating user accounts on the ASA to use for authentication. If you do this, the password has to be of type mschap, or the authentication will fail. Note: After you set the password, it will show up as nt-encrypted in the config, but it is really mschap. The line you enter for the user should be something like:
username thisuser password thatpassword mschap
Step 6: Client configuration
On OS X: Open Internet Connect, File -> New Connection -> L2TP over IPSec. Click drop-down box by Configuration -> Edit Configurations. Enter Description, Server IP, Account Name, Password, and Shared Secret, Group Name (if you didn't use DefaultRAGroup), then click OK. You can optionally click Connect -> Options -> uncheck "Send all traffic over VPN connection", click OK.
If you leave that last item checked, it will try to send all your internet traffic over the VPN. Unchecked, it only sends traffic destined for the VPN addresses over the VPN and the rest goes out your normal Internet connection.

