Troubleshooting Phase 2 Cisco Site to Site (L2L) VPN Tunnels

KB ID 0000625 

Problem

It’s been over two years since I wrote Troubleshooting Phase 1 Cisco Site to Site (L2L) VPN Tunnels. I’ve always meant to come back and write the ‘Phase 2’ article but never got around to it.

This article is NOT intended to be a ‘fix all” for phase 2 problems, it’s designed to point you in the right direction to locate the source of the problem.

Solution

Here’s my fictional VPN setup,

1. Before you start: We are looking at phase 2 problems, MAKE SURE phase 1 has established!

[box]

Petes-ASA>
Petes-ASA> en
Password: ********
Petes-ASA# show crypto isakmp

IKEv1 SAs:

Active SA: 1 Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1 IKE Peer: 234.234.234.234
Type : L2L Role : responder
Rekey : no State : MM_ACTIVE <<<< Phase 1 has established!

[/box]

2. At the first site, issue a ‘show crypto ipsec sa’ command. Note: if you have a lot of tunnels and the output is confusing use a ‘show crypto ipsec sa peer 234.234.234.234’ command instead.

Click for Larger Image

Note: Yes I can zero in on the problem here, but your output may be different (And if you already know why are you reading this!).

3. Check the IP addresses and networks listed highlighted in yellow are correct, then repeat for the ones highlighted in pink (I don’t care if you don’t think it’s pink! I’m a man, I see in 16 colours like a Sinclair ZX Spectrum!). Check the access-list (above shown in blue text) that access-list should be referenced in the crypto map like so;

[box]

Petes-ASA# show run crypto
{Lots of output removed for the sake of space}
crypto map outside_map 1 match address outside_38_cryptomap <<<<Good!!
crypto map outside_map 1 set pfs
crypto map outside_map 1 set peer 234.234.234.234
crypto map outside_map 1 set ikev1 transform-set ESP-3DES-SHA <<<< Only seen after v8.4!
{Lots of output removed for the sake of space}

[/box]

4. Now we need to make sure that the traffic is NOT subject to NAT, issue a “show run nat” command;

[box]

Output with an Operating System Newer than 8.3 

Petes-ASA# show run nat
{Lots of output removed for the sake of space}
nat (inside,outside) source static NETWORK_OBJ_192.168.1.0_24 NETWORK_OBJ_192.168.1.0_24 destination static NETWORK_192.168.2.0_24 NETWORK_OBJ_192.168.2.0_2
{Lots of output removed for the sake of space}
Petes-ASA# show run object network <<<<Lets check those objects exist
object network obj_any
subnet 0.0.0.0 0.0.0.0
object network NETWORK_OBJ_192.168.1.0_24
subnet 192.168.1.0 255.255.255.0
object network NETWORK_OBJ_192.168.2.0_24
subnet 192.168.2.0 255.255.255.0


Output with an Operating System Older than 8.3

Petes-ASA# show run nat
nat (inside) 0 access-list inside_nat0_outbound
nat (inside) 1 192.168.1.0 255.255.255.0
Petes-ASA# show run access-list inside_nat0_outbound <<<<Lets make sure that ACL is correct
access-list inside_nat0_outbound extended permit ip 192.168.1.0 255.255.255.0 192.168.2.0 255.255.255.0

[/box]

5. Now repeat the process at the other end;

Click for Larger Image

6. You will notice I’ve shown the SPI numbers in green, these should match (but be the opposite way round) at both ends. (Geek Note: These denote the TWO tunnels IPSEC brings up inside the original ISAKMP tunnel that it then passes information up one and down the other – like a two lane road).

7. If you compare both outputs look at the pkts encaps (in red) and the pkts decaps (in purple). In this case we can see that the tunnel is working as it should from the 234.234.234.234 site but no traffic is getting encrypted from the 123.123.123.123 site. THAT’S WHERE THE PROBLEM IS.

8. Now you know where the problem is you can issue a “debug crypto ipsec” command there. Then try to bring up the tunnel and analyse the output. Note: If debug shows nothing make sure there is NOT another crypto map pointing to the same subnet, with a different peer.

9. So on our ‘problem’ end we see something like;

So on that firewall, locate the ACL that is being used for the crypto map, and make sure its ‘hit count’ is going up as you try and send traffic over the VPN tunnel. If not then the ACL is wrong, there’s a routing problem or a subnet mask is wrong on the firewalls internal interface.

[box]

Petes-ASA# show run crypto
crypto ipsec ikev1 transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto map outside_map 1 match address outside_1_cryptomap << Here!
crypto map outside_map 1 set pfs
crypto map outside_map 1 set peer 234.234.234.234
crypto map outside_map 1 set ikev1 transform-set ESP-3DES-SHA
crypto map outside_map interface outside
crypto ikev1 enable outside
crypto ikev1 policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
Petes-ASA# show access-list outside_1_cryptomap
access-list outside_1_cryptomap; 1 element; name hash: 0xcf826bcb
access-list outside_1_cryptomap line 1 extended permit ip 192.168.1.0 255.255.255.0 192.168.2.0 255.255.255.0 (hitcnt=93) 0xf965c6f9

[/box]

10. As a last resort recycle/restart the VPN Tunnel.

Cisco ASA 5500 – Reset / Recycle VPN Tunnels

11. You’ve done your best and still it wont establish! Then I would upgrade the ASA(s) to the latest OS (70% of the calls I log to Cisco TAC for VPN issues are fixed by simply upgrading them, 29% are caused by a configuration error, and 1% need a version of the operating system that has not been released yet). If you’re under warranty or Cisco SmartNet, you can then log a call to Cisco TAC. If not then I would suggest heading over to EE and asking for help. (You might even get hold of me).

Related Articles, References, Credits, or External Links

Troubleshooting Phase 1 Cisco Site to Site (L2L) VPN Tunnels

Cisco ASA Site to Site VPN’s

Cisco ASA5500 Site to Site VPN from ASDM

Cisco ASA 5500 Site to Site VPN (From CLI)