KB ID 0000216

Problem

Site to Site VPN’s either work faultlessly straight away, or involve head scratching and a call to Cisco TAC, or someone like me to come and take a look. If I’m honest, the simplest and best answer to the problem is “Remove the Tunnel from both ends and put it back again”. Just about every VPN tunnel I’ve put in that did not work, was a result of my fat fingers putting in the wrong subnet, IP address or shared secret.

However you can’t always remove the tunnel and start again, especially if you only have control of your end of the tunnel. In that case you need to do some troubleshooting and debugging.

Solution

To Troubleshoot and debug a VPN tunnel you need to have an appreciation of how VPN Tunnels work READ THIS.

Now you have read that you are an expert on IKE VPN Tunnels 🙂

Step 1

To bring up a VPN tunnel you need to generate some “Interesting Traffic” Start by attempting to send some traffic over the VPN tunnel.

Step 2 See if Phase 1 has completed.

Connect to the firewall and issue the following commands.

[box]

User Access Verification
Password: 
Type help or '?' for a list of available commands. 
PetesASA> enable 
Password: ******** 
PetesASA# show crypto isakmp

[/box]

You may see a lot more information if you have Existing VPN tunnels, but what you are looking for is this,

[box]

IKEv1 SAs:

   Active SA: 2
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 2

1   IKE Peer: 123.123.123.123
    Type    : L2L             Role    : initiator
    Rekey   : no              State   : MM_ACTIVE <<YOUR SIDE BROUGHT THE VPN UP

There are no IKEv2 SAs

[/box]

If you see MM_ACTIVE (This means phase 1 has completed in Main Mode, and is active) So phase 1 has completed successfully, you need to jump forward and troubleshoot Phase 2.

Note: If you see AG_{something} this means you are trying to bring the tunnel up in aggressive mode!

If there is nothing listed at all – then your side is not even trying to bring up the tunnel. Try and generate a lot of VPN traffic – Like a persistent ping {ping 192.168.1.1 -t} and issue the show crypto isakmp command a few times to be sure. if you never see anything then its not getting as far as phase 1!

If your still reading this, then your problem is with Phase 1, and you have an ISAKMP SA state error.

ISAKMP SA MESSAGE STATES (On the Initiator)

MM_WAIT_MSG2

Message 1 has been sent to the responder but there has been no reply.

Causes:

1. There is no network connectivity to the firewallsecurity device at the other end, can you ping it?

2. The IP address of the far firewall is incorrect in the tunnel-group, issue a “show run tunnel-group” command, check you have a tunnel group with the correct IP address.

3. The IP address in the “Crypto Map” is incorrect, issue a “show run crypto map” command and check the line that ends “crypto map {name} {number} set peer xxx.xxx.xxx.xxx” to make sure.

4. You do not have a matching phase 1 policy with the other end, issue a “show run crypto isakmp” command make sure the other end has a matching policy, if you cant check the other end then generate some VPN traffic, issue the following command and check for the following,

[box]

EXAMPLE PHASE 1 POLICIES DONT MATCH

Password: Type help or ‘?’ for a list of available commands. PetesASA> en Password: ******** PetesASA#debug crypto isakmp 200

<<<<<<<LOTS Of DEBUG TEXT REMOVED>>>>>>>

Apr 01 14:48:48 [IKEv1]: IP = 123.123.123.123, IKE_DECODE RECEIVED Message (msgid=ce4a3ffe) with payloads : HDR + NOTIFY (11) + NONE (0) total length : 56 Apr 01 14:48:48 [IKEv1]: IP = 123.123.123.123, Received an un-encrypted NO_PROPOSAL_CHOSEN notify message, dropping Apr 01 14:48:48 [IKEv1]: IP = 123.123.123.123, Information Exchange processing failed

<<<<<<<LOTS Of DEBUG TEXT REMOVED>>>>>>>

[/box]

MM_WAIT_MSG4

The Phase 1 Policies have been agreed with both peers, the initiator is waiting for the responder to send it its keying information. I’ve seen two things cause this.

1. Different Vendors equipment talking the the ASA, or simply the version of OS on the ASA have been different.

2. There is a comms error, check there’s no router with firewall capabilities in the link.

MM_WAIT_MSG6

If there’s a firewall ‘in-between’ make sure UDP port 4500 is open for both peers.

Check your Pre-Shared Keys match on the ASA issue a “more system:running-config” then keep pressing the space bar till you see the tunnel- group and shared key

e.g.

tunnel-group 123.123.123.123 ipsec-attributes pre-shared-key this-is-the-pre-shared-key

Again if you can’t check the other end then issue the following debug and the following will tell you if there is a key mismatch.

This error can also be seen if one end has PFS set and the other end does not. In this case the error will appear and dissapear and the connection is repeatedly “torn down”

e.g

crypto map outside_map 20 set pfs [box]

EXAMPLE PHASE 1 PRE SHARED KEYS DONT MATCH

Password: Type help or ‘?’ for a list of available commands. PetesASA> en Password: ******** PetesASA#debug crypto isakmp 200

<<<<<<<LOTS Of DEBUG TEXT REMOVED>>>>>>>

Apr 01 15:11:47 [IKEv1]: IP = 123.123.123.123, IKE_DECODE RECEIVED Message (msgid=5456d64e) with payloads : HDR + NOTIFY (11) + NONE (0) total length : 56 Apr 01 15:11:47 [IKEv1]: Group = 123.123.123.123, IP = 123.123.123.123, Received an un-encrypted PAYLOAD_MALFORMED notify message, dropping Apr 01 15:11:47 [IKEv1]: Group = 123.123.123.123, IP = 123.123.123.123, Error, peer has indicated that something is wrong with our message. This could indicate a pre-shared key mismatch. Apr 01 15:11:47 [IKEv1]: Group = 123.123.123.123, IP = 123.123.123.123, Information Exchange processing failed

<<<<<<<LOTS Of DEBUG TEXT REMOVED>>>>>>>

[/box]

ISAKMP SA MESSAGE STATES (On the Responder)

MM_WAIT_MSG3

The Phase 1 Policies have been agreed with both peers, the responder is waiting for the initiator to send it its keying information. I’ve seen two things cause this.

1. Different Vendors equipment talking the the ASA, or simply the version of OS on the ASA have been different.

2. There is a comms error, check there’s no router with firewall capabilities in the link.

3. I’ve seen this on a VPN from a VMware Edge Gateway, that had PFS (perfect forward secrecy) enabled, and the ASA did not.

Also see: Cisco ASA VPN to Cisco Router “MM_WAIT_MSG3”

MM_WAIT_MSG5

Make sure the Pre-Shared Keys Match

If there’s a firewall ‘in-between’ make sure UDP port 4500 is open for both peers.

Here’s an Example of Phase one completing message by message successfully.

MESSAGE 1 (Leaving the Initiator)

[box]

Apr 01 11:38:51 [IKEv1]: IP = 123.123.123.123, IKE Initiator: New Phase 1, Intf inside, IKE Peer 123.123.123.123 local Proxy Address 192.168.1.0, remote Proxy Address 172.16.1.0, Crypto map (outside_map) Apr 01 11:38:51 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing ISAKMP SA payload Apr 01 11:38:51 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing NAT-Traversal VID ver 02 payload Apr 01 11:38:51 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing NAT-Traversal VID ver 03 payload Apr 01 11:38:51 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing NAT-Traversal VID ver RFC payload Apr 01 11:38:51 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing Fragmentation VID + extended capabilities payload Apr 01 11:38:51 [IKEv1]: IP = 123.123.123.123, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 168

[/box]

MESSAGE 2 (Arriving at the Responder)

[box]

Apr 01 11:38:52 [IKEv1]: IP = 123.123.123.123, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 117
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, processing SA payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, Oakley proposal is acceptable
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, processing VID payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing ke payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing nonce payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing Cisco Unity VID payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing xauth V6 VID payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, Send IOS VID
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, Constructing ASA spoofing IOS Vendor ID payload (version: 1.0.0, capabilities: 20000001)
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing VID payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, Send Altiga/Cisco VPN3000/Cisco ASA GW VID

[/box]

MESSAGE 3 (Leaving the Initiator)

[box]

Apr 01 11:38:52 [IKEv1]: IP = 123.123.123.123, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + KE (4) + NONCE (10) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 256

[/box]

MESSAGE 4 (Arriving at the Initiator)

[box]

Apr 01 11:38:52 [IKEv1]: IP = 123.123.123.123, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + KE (4) + NONCE (10) + NONE (0) total length : 228
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, processing ke payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, processing ISA_KE payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, processing nonce payload

[/box]

MESSAGE 5 (Leaving the Initiator)

[box]

Apr 01 11:38:52 [IKEv1]: IP = 123.123.123.123, Connection landed on tunnel_group 123.123.123.123
Apr 01 11:38:52 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, Generating keys for Initiator...
Apr 01 11:38:52 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, constructing ID payload
Apr 01 11:38:52 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, constructing hash payload
Apr 01 11:38:52 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, Computing hash for ISAKMP
Apr 01 11:38:52 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, constructing dpd vid payload
Apr 01 11:38:52 [IKEv1]: IP = 123.123.123.123, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + ID (5) + HASH (8) + VENDOR (13) + NONE (0) total length : 84

[/box]

MESSAGE 6 (Arriving at the Initiator)

[box]

Apr 01 11:38:53 [IKEv1]: IP = 123.123.123.123, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + ID (5) + HASH (8) + NONE (0) total length : 64
Apr 01 11:38:53 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, processing ID payload
Apr 01 11:38:53 [IKEv1 DECODE]: Group = 123.123.123.123, IP = 123.123.123.123, ID_IPV4_ADDR ID received 123.123.123.123
Apr 01 11:38:53 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, processing hash payload
Apr 01 11:38:53 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, Computing hash for ISAKMP
Apr 01 11:38:53 [IKEv1]: IP = 123.123.123.123, Connection landed on tunnel_group 123.123.123.123
Apr 01 11:38:53 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, Oakley begin quick mode
Apr 01 11:38:53 [IKEv1 DECODE]: Group = 123.123.123.123, IP = 123.123.123.123, IKE Initiator starting QM: msg id = 26f952ae
Apr 01 11:38:53 [IKEv1]: Group = 123.123.123.123, IP = 123.123.123.123, PHASE 1 COMPLETED

[/box]

Note: You can debug Phase 1 traffic on a particular tunnel, with the following command.

debug crypto condition peer 123.123.123.123

or, simply;

debug crypto isakmp

Phase 1 Debug Errors

[box]

Petes-ASA((config)# debug crypto isakmp
Feb 29 11:49:08 [IKEv1]Group = 123.123.123.123, IP = 123.123.123.123, QM FSM error (P2 struct &0x00007fda92308b10, mess id 0xc02b7a5d)!
Feb 29 11:49:08 [IKEv1]Group = 123.123.123.123, IP = 123.123.123.123, Removing peer from correlator table failed, no match!
Feb 29 11:49:08 [IKEv1]Group = 123.123.123.123, IP = 123.123.123.123, Session is being torn down. Reason: crypto map policy not found

[/box]

This was due to more than one misconfiguration, firstly the source and destination network objects in the interesting traffic ACL were the wrong way round! (Don’t forget to check your static NAT statement as well). And the TRANSFORM SET didn’t match, (sometimes you can see phase one established but then it disappears).

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

[box]

Petes-ASA((config)# debug crypto ikev1
Apr 19 16:36:10 [IKEv1]IP = 123.123.123.123, No Group found by matching OU(s) from ID payload:   Unknown
Apr 19 16:36:10 [IKEv1]IP = 123.123.123.123, Trying to find group via IKE ID...
Apr 19 16:36:10 [IKEv1]IP = 123.123.123.123, Trying to find group via IP ADDR...
Apr 19 16:36:10 [IKEv1]IP = 123.123.123.123, Connection landed on tunnel_group 123.123.123.123
Apr 19 16:36:10 [IKEv1 DEBUG]Group = 123.123.123.123, IP = 123.123.123.123, peer ID type 2 received (FQDN)
Apr 19 16:36:10 [IKEv1]Group = 123.123.123.123, IP = 123.123.123.123, Unable to compare IKE ID against peer cert Subject Alt Name
Apr 19 16:36:10 [IKEv1 DEBUG]Group = 123.123.123.123, IP = 123.123.123.123, IKE MM Responder FSM error history (struct &0x00007ffd9e230670)  <state>, <event>:  MM_DONE, EV_ERROR-->MM_BLD_MSG6, EV_COMPARE_IDS-->MM_BLD_MSG6, EV_CERT_OK-->MM_BLD_MSG6, NullEvent-->MM_BLD_MSG6, EV_ACTIVATE_NEW_SA-->MM_BLD_MSG6, NullEvent-->MM_BLD_MSG6, EV_VALIDATE_CERT-->MM_BLD_MSG6, EV_UPDATE_CERT
Apr 19 16:36:10 [IKEv1 DEBUG]Group = 123.123.123.123, IP = 123.123.123.123, IKE SA MM:3a1ed893 terminating:  flags 0x01018002, refcnt 0, tuncnt 0
Apr 19 16:36:10 [IKEv1 DEBUG]Group = 123.123.123.123, IP = 123.123.123.123, sending delete/delete with reason message

[/box]

The ASA did not like the certificate presented by the remote peer, (Even though is was a good cert issued by NDES). To get past this you need to make a change to the tunnel group.

[box]

Petes-ASA(config)# tunnel-group TG-TUNNEL-HQ ipsec-attributes
Petes-ASA(config-ipsec)# peer-id-validate nocheck

[/box]

Or if you prefer the ASDM;

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

[box]

Petes-ASA((config)# debug crypto ikev1
%ASA-3-717009: Certificate validation failed. Peer certificate key usage is invalid, serial number: 6B00002B3F8571E2605FA02883000100002C3E, subject name: hostname=Petes-Router-Petes-HQ.petenetlive.com.
%ASA-3-717027: Certificate chain failed validation. Certificate chain is either invalid or not authorized.

[/box]

The ASA did not like the certificate presented by the remote peer, (Even though is was a good cert issued by NDES). To get past this you need to make a change to the trustpoint on the ASA.

[box]

Petes-ASA(config)# crypto ca trustpoint PNL-Trustpoint
Petes-ASA(config-ca-trustpoint)# ignore-ipsec-keyusage 

[/box]

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

[box]

Petes-ASA# debug crypto ikev1
Petes-ASA# Feb 17 12:25:17 [IKEv1]Group = 123.123.123.123, IP = 123.123.123.123, Received encrypted Oakley Main Mode packet with invalid payloads, MessID = 0
Feb 17 12:25:17 [IKEv1]Group = 212.20.251.44, IP = 123.123.123.123, ERROR, had problems decrypting packet, probably due to mismatched pre-shared key. Aborting
Feb 17 12:25:23 [IKEv1]IP = 123.123.123.123, Header invalid, missing SA payload! (next payload = 4)
\

[/box]

Amazingly this had nothing to do with a mismatched pre shared key, the other end was set to use PFS (Perfect Forward Secrecy,) and my end (the ASA) was not.

If you have got this far the next step is to troubleshoot Phase 2

Related Articles, References, Credits, or External Links

Troubleshooting Phase 2 Cisco Site to Site (L2L) VPN Tunnels

Thanks To Steve Housego for the Certificate Phase 1 Error details.