PeteNetLive - KB0000216 - Troubleshooting Phase 1 Cisco Site to Site (L2L) VPN Tunnels

0000216

Dated

01/04/10

Revision

0.01

Troubleshooting Phase 1 Cisco Site to Site (L2L) VPN Tunnels

Problem

Site to Site VPN's either work faultlessly straight away, or involve head scratching and a call to Cisco TAC, or someone like me to come and take a look. If I'm honest, the simplest and best answer to the problem is "Remove the Tunnel from both ends and put it back again". Just about every VPN tunnel I've put in that did not work, was a result of my fat fingers putting in the wrong subnet, IP address or shared secret.

However you can,t always remove the tunnel and start again, especially if you only have control of your end of the tunnel. In that case you need to do some troubleshooting and debugging.

Solution

To Troubleshoot and debug a VPN tunnel you need to have an appreciation of how VPN Tunnels work READ THIS.

Now you have read that you are an expert on IKE VPN Tunnels :)

Step 1

To bring up a VPN tunnel you need to generate some "Interesting Traffic" Start by attempting to send some traffic over the VPN tunnel.

Step 2 See if Phase 1 has completed.

Connect to the firewall and issue the following commands.

User Access Verification

Password:
Type help or '?' for a list of available commands.
PetesASA> en
Password: ********
PetesASA#show crypto isakmp

You may see a lot more information if you have Existing VPN tunnels, but what you are looking for is this,

Active SA: .....1
Rekey SA:......0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1.......IKE Peer: 123.123.123.123
.........Type : L2L ...............Role ..: initiator <<< YOUR SIDE BROUGHT THE VPN UP
.........Rekey : no ...............State : MM_ACTIVE <<< THIS IS WHAT YOU WANT

If you see MM_ACTIVE (This means phase 1 has completed in Main Mode, and is active) So phase 1 has completed successfully, you need to jump forward and troubleshoot Phase 2.

Note: If you see AG_{something} this means you are trying to bring the tunnel up in aggressive mode!

If there is nothing listed at all - then your side is not even trying to bring up the tunnel. Try and generate a lot of VPN traffic - Like a persistent ping {ping 192.168.1.1 -t} and issue the show crypto isakmp command a few times to be sure. if you never see anything then its not getting as far as phase 1!

If your still reading this, then your problem is with Phase 1, and you have an ISAKMP SA state error.

ISAKMP SA MESSAGE STATES (On the Initiator)
MM_WAIT_MSG2	Message 1 has been sent to the responder but there has been no reply. Causes: 1. There is no network connectivity to the firewall\security device at the other end, can you ping it? 2. The IP address of the far firewall is incorrect in the tunnel-group, issue a "show run tunnel-group" command, check you have a tunnel group with the correct IP address. 3. The IP address in the "Crypto Map" is incorrect, issue a "show run crypto map" command and check the line that ends "crypto map {name} {number} set peer xxx.xxx.xxx.xxx" to make sure. 4. You do not have a matching phase 1 policy with the other end, issue a "show run crypto isakmp" command make sure the other end has a matching policy, if you cant check the other end then generate some VPN traffic, issue the following command and check for the following,
MM_WAIT_MSG2	EXAMPLE PHASE 1 POLICIES DONT MATCH Password: Type help or '?' for a list of available commands. PetesASA> en Password: ******** PetesASA#debug crypto isakmp 200 <<<<<<<LOTS Of DEBUG TEXT REMOVED>>>>>>> Apr 01 14:48:48 [IKEv1]: IP = 123.123.123.123, IKE_DECODE RECEIVED Message (msgid=ce4a3ffe) with payloads : HDR + NOTIFY (11) + NONE (0) total length : 56 Apr 01 14:48:48 [IKEv1]: IP = 123.123.123.123, Received an un-encrypted NO_PROPOSAL_CHOSEN notify message, dropping Apr 01 14:48:48 [IKEv1]: IP = 123.123.123.123, Information Exchange processing failed <<<<<<<LOTS Of DEBUG TEXT REMOVED>>>>>>>
MM_WAIT_MSG4	The Phase 1 Policies have been agreed with both peers, the initiator is waiting for the responder to send it its keying information. I've seen two things cause this. 1. Different Vendors equipment talking the the ASA, or simply the version of OS on the ASA have been different. 2. There is a comms error, check there's no router with firewall capabilities in the link.
MM_WAIT_MSG6	Check your Pre-Shared Keys match on the ASA issue a "more system:running-config" then keep pressing the space bar till you see the tunnel- group and shared key e.g. tunnel-group 123.123.123.123 ipsec-attributes pre-shared-key this-is-the-pre-shared-key Again if you can't check the other end then issue the following debug and the following will tell you if there is a key mismatch. This error can also be seen if one end has PFS set and the other end does not. In this case the error will appear and dissapear and the connection is repeatedly "torn down" e.g crypto map outside_map 20 set pfs
MM_WAIT_MSG6	EXAMPLE PHASE 1 PRE SHARED KEYS DONT MATCH Password: Type help or '?' for a list of available commands. PetesASA> en Password: ******** PetesASA#debug crypto isakmp 200 <<<<<<<LOTS Of DEBUG TEXT REMOVED>>>>>>> Apr 01 15:11:47 [IKEv1]: IP = 123.123.123.123, IKE_DECODE RECEIVED Message (msgid=5456d64e) with payloads : HDR + NOTIFY (11) + NONE (0) total length : 56 Apr 01 15:11:47 [IKEv1]: Group = 123.123.123.123, IP = 123.123.123.123, Received an un-encrypted PAYLOAD_MALFORMED notify message, dropping Apr 01 15:11:47 [IKEv1]: Group = 123.123.123.123, IP = 123.123.123.123, Error, peer has indicated that something is wrong with our message. This could indicate a pre-shared key mismatch. Apr 01 15:11:47 [IKEv1]: Group = 123.123.123.123, IP = 123.123.123.123, Information Exchange processing failed <<<<<<<LOTS Of DEBUG TEXT REMOVED>>>>>>>
ISAKMP SA MESSAGE STATES (On the Responder)
MM_WAIT_MSG3	The Phase 1 Policies have been agreed with both peers, the responder is waiting for the initiator to send it its keying information. I've seen two things cause this. 1. Different Vendors equipment talking the the ASA, or simply the version of OS on the ASA have been different. 2. There is a comms error, check there's no router with firewall capabilities in the link.
MM_WAIT_MSG5	Make sure the Pre-Shared Keys Match

Here's an Example of Phase one completing message by message successfully.

MESSAGE 1 (Leaving the Initiator)

Apr 01 11:38:51 [IKEv1]: IP = 123.123.123.123, IKE Initiator: New Phase 1, Intf inside, IKE Peer 123.123.123.123 local Proxy Address 192.168.1.0, remote Proxy Address 172.16.1.0, Crypto map (outside_map)
Apr 01 11:38:51 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing ISAKMP SA payload
Apr 01 11:38:51 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing NAT-Traversal VID ver 02 payload
Apr 01 11:38:51 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing NAT-Traversal VID ver 03 payload
Apr 01 11:38:51 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing NAT-Traversal VID ver RFC payload
Apr 01 11:38:51 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing Fragmentation VID + extended capabilities payload
Apr 01 11:38:51 [IKEv1]: IP = 123.123.123.123, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 168

MESSAGE 2 (Arriving at the Initiator)

Apr 01 11:38:52 [IKEv1]: IP = 123.123.123.123, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 117
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, processing SA payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, Oakley proposal is acceptable
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, processing VID payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing ke payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing nonce payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing Cisco Unity VID payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing xauth V6 VID payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, Send IOS VID
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, Constructing ASA spoofing IOS Vendor ID payload (version: 1.0.0, capabilities: 20000001)
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing VID payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, Send Altiga/Cisco VPN3000/Cisco ASA GW VID

MESSAGE 3 (Leaving the Initiator)

Apr 01 11:38:52 [IKEv1]: IP = 123.123.123.123, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + KE (4) + NONCE (10) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 256

MESSAGE 4 (Arriving at the Initiator)

Apr 01 11:38:52 [IKEv1]: IP = 123.123.123.123, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + KE (4) + NONCE (10) + NONE (0) total length : 228
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, processing ke payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, processing ISA_KE payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, processing nonce payload

MESSAGE 5 (Leaving the Initiator)

Apr 01 11:38:52 [IKEv1]: IP = 123.123.123.123, Connection landed on tunnel_group 123.123.123.123
Apr 01 11:38:52 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, Generating keys for Initiator...
Apr 01 11:38:52 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, constructing ID payload
Apr 01 11:38:52 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, constructing hash payload
Apr 01 11:38:52 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, Computing hash for ISAKMP
Apr 01 11:38:52 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, constructing dpd vid payload
Apr 01 11:38:52 [IKEv1]: IP = 123.123.123.123, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + ID (5) + HASH (8) + VENDOR (13) + NONE (0) total length : 84

MESSAGE 6 (Arriving at the Initiator)

Apr 01 11:38:53 [IKEv1]: IP = 123.123.123.123, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + ID (5) + HASH (8) + NONE (0) total length : 64
Apr 01 11:38:53 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, processing ID payload
Apr 01 11:38:53 [IKEv1 DECODE]: Group = 123.123.123.123, IP = 123.123.123.123, ID_IPV4_ADDR ID received 123.123.123.123
Apr 01 11:38:53 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, processing hash payload
Apr 01 11:38:53 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, Computing hash for ISAKMP
Apr 01 11:38:53 [IKEv1]: IP = 123.123.123.123, Connection landed on tunnel_group 123.123.123.123
Apr 01 11:38:53 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, Oakley begin quick mode
Apr 01 11:38:53 [IKEv1 DECODE]: Group = 123.123.123.123, IP = 123.123.123.123, IKE Initiator starting QM: msg id = 26f952ae
Apr 01 11:38:53 [IKEv1]: Group = 123.123.123.123, IP = 123.123.123.123, PHASE 1 COMPLETED

Note: You can debug Phase 1 traffic on a particular tunnel, with the following command.

debug crypto condition peer 123.123.123.123

If you have got this far the next step is to troubleshoot Phase 2

If this post helped you, PLEASE take the time to +1 it.

Please be aware, all information is provided free, but it does cost me to have this site hosted, if I've helped you in any way, or saved you some time/cost please take time to make a donation.

If you have anything to add to an article, or have an article you would like us to publish please feel free to contact PeteNetLive. (Please be aware I get a LOT of email, I cannot assist and fix everyone's problems, please do not be offended if you do not get a response).

References - Credits - Or External Links