I seem to have done a lot of PKI the last 18 months. This week I needed an OCSP server deploying for the CA server on my test bench so I took the time to document it for future use. One of the most overlooked parts of a PKI deployment, is how to cope with ‘revoking’ certificates.
Traditionally this has been done with a CRL, but there is a downside to CRL’s. Network devices tend to cache them, and they don’t get updated as often as they should. Which is why OCSP is a lot better, it runs as a service and answers revocation requests immediately.
Solution
This is a walk though on configuring OCSP, I’m assuming you already have your PKI and your CRL is already setup, if not take a look at the following article;
6. Open the properties of the Certificate Services Server.
7. Extensions Tab > Change the drop-down to Authority Information Access (AIA) > Add >Enter the URL of the server that will serve OCSP, this will need to be resolvable internally and of the service is presented to the outside works the URL will need registering in public DNS > OK.
8. Select the URL you have just entered > Tick ‘Include in the online certificate status protocol (OCSP) extension’ > Apply > OK.
9. On the server that you want to install the OCSP service > Launch Server Manager > Manage > Add Roles and Services > Add in the Active Directory Certificate Services Role.
10. On the Role Services page, remove ‘Certificate Authority’ and then add ‘Online Responder’ > Complete the wizard.
11. Run through the post install wizard, and accept all the defaults.
12. Launch the Online Responder Management console.
13. Revocation Configuration > Add Revocation Configuration > Enter a name > Next.
14. Select your method of locating a CA certificate > Next.
As with most things, before you have a hope of fixing something, you will stand a better chance if you know how it works in the first place. Below is a quick run though of what’s happening with your site to site VPN‘s and how they work.
For the entire process we will have two Cisco ASA 5500 firewalls and a site to site VPN.
Solution
What’s an Initiator and a Responder?
1. Our Laptop 192.168.1.50 wants to talk to a server on the other site at 172.16.1.50
2. To get out of the local network the Laptop goes through the ASA at its local site, The ASA knows that traffic destined for 172.16.1.50 needs to be sent down the VPN tunnel, so it needs to bring up the tunnel. IT BECOMES THE INITIATOR, contacts the ASA on the other site THAT BECOMES THE RESPONDER.
3 Once that’s complete the tunnel is up and traffic can pass.
So how does it bring up the Tunnel?
To establish an ISAKMPVPN tunnel 3 things have to happen.
1. Phase 1 has to complete.
2. Phase 2 has to complete.
3. The Traffic has to be allowed to pass.
VPN Phase 1 (ISAKMP)
This stage brings up the first secure tunnel (eventually there will be three tunnels) and for it to establish the firewalls need to agree what they are going to do to bring up the tunnel, then Secure the tunnel. This process uses SIX MESSAGES (Note: We are dealing to Main Mode here not Aggressive mode). Both firewalls need a matching Phase 1 Policy to continue. And the Policy is proposed in MESSAGE1 and accepted in MESSAGE2.
A Phase 1 policy consists of,
1. The Authentication method (either a pre shared key or an RSA signature is usual).
5. Lifetime (In seconds before phase 1 should be re-established – usually 86400 seconds [1 day]).
MESSAGE 1
The Initiator sends policies that it proposes to use, for phase 1 to the other ASA.
MESSAGE 2
Providing the responder has a matching policy it will accept one of those proposed by the initiator and send it back in message 2.
Now the two ends have agreed HOW they will establish phase 1, they then need to agree on a “Shared Key” both ends must use the same shared key, but the shared key cant be sent between them because the network link is not secure. To do this they use a Diffie Hellman key exchange, this uses a mathematical process called modular exponentiation, a simple example of how that works (The math’s involved in a real key exchange are much more complicated!).
How Diffie Hellman works (simply)
Problem Site A and Site B need to use the same secret key (which will be a big long number). they cant send that number to each other because if they do it will be seen.
Solution:
Both sites pick a random number, and they have a common number, this common number can be passed between sites, In our example Site A chooses 4 and Site B chooses 5
Both sites use the common number and raise it by the power of the random number they are using so Site A arrives at 16, and Site B at 32.
The sites then send the number they have arrived at, to the other site.
Each site uses the other sites total and raises it to the power of their original random number, this results in them both having the same key, with only the numbers 2, 16 and 32 being passed between them.
Back to our VPN Tunnel
The next two messages are the initiator and responder swapping their Diffie Hellman information, Each side produces a DH Public Key, and mathematically computes a long number called a “Nonce”
MESSAGE 3
The initiator generates a “Public Key” also called the DH Public Value or Xa It also generates a Nonce or Ni and sends both of them to the responder.
MESSAGE 4
The responder generates a “Public Key” also called the DH Public Value or Xb It also generates a Nonce or Nr and sends both of them to the initiator.
At this point both the initiator and the responder can calculate the DH Shared secret key, they then use the DH Secret Key, the “Shared Secret” that is manually entered onto both peers, and the Nonce from the other peer to create 3 DIGITAL KEYS, because of the nature of Diffie Hellman each end will produce the same keys.
Key 1 = SKEYID_d Used to work out any future IPsec keying Key 2 = SKEYID_a Used for data integrity and authentication (IKE) Key 3 = SKEYID_e Used to encrypt all further IKE traffic.
MESSAGE 5
The initiator now sends its ID to the responder (this is either its IP address or a hostname). It also sends a “Hash” this authenticates the initiator to the responder as its made from the SKEYID, the pre-shared key and other information only known to the two peers.
MESSAGE 6
Message 6 is basically the mirror of Message 5, the responder sends its ID (IP or Hostname) Back the the initiator with its “Hash” and authenticates itself back to the initiator.
At this point both peers recalculate the hash they have received from the other peer, and they should both come out the same, if this happens then the IKESA’s are established and phase 1 is complete.
So what’s PFS?
Perfect Forward Secrecy is a method by which new keys are generated, each new key is mathematically linked to the key that came before it, the prior key being a “Grandfather” key. With PFS enabled this link is broken so a key can not be forward/reverse engineered to guess a previous/new key value). Every new negotiation produces a new fresh key.
Once Phase 1 has completed the second stage of the VPN can start. Like phase 1 this state also requires messages to be sent between the peers, IPsec usually executes in “Quick mode” this means that there are only 3 MESSAGES.
Note: If PFS is configured only on one end then it will fail at this point with an “Attribute not supported” error.
MESSAGE 1
The Initiator sends another Hash to the responder, this is similar to the one used in phase 1 but also includes info within this message to guarantee integrity.
4. The SPI – This number is the LABEL for the end of the tunnel the initiator will use for outbound traffic.
Tunnel mode (Tunnel or Transport). A timeout in seconds is specified, as is the ID (usually the subnet of both ends of the tunnel).
MESSAGE 2
The Responder replies with its own “Hash” with the accepted proposal and its own SPI for outgoing encrypted traffic from the responder, and finally its own Key Exchange Payload.
Once this is complete both peers generate new DH secret keys and combine them with the SKEYID_d key from phase 1 to create keys for IPsec encryption.
MESSAGE 3
The final Message is sent from imitator to responder, and serves to inform the responder that its previous message was received.
Once phase 2 is complete IPsec SA’s have been established and the tunnel is up.
Related Articles, References, Credits, or External Links
Site to Site VPN’s either work faultlessly straight away, or involve head scratching and a call to Cisco TAC, or someone like me to come and take a look. If I’m honest, the simplest and best answer to the problem is “Remove the Tunnel from both ends and put it back again”. Just about every VPN tunnel I’ve put in that did not work, was a result of my fat fingers putting in the wrong subnet, IP address or shared secret.
However you can’t always remove the tunnel and start again, especially if you only have control of your end of the tunnel. In that case you need to do some troubleshooting and debugging.
Solution
To Troubleshoot and debug a VPN tunnel you need to have an appreciation of how VPN Tunnels work READ THIS.
Now you have read that you are an expert on IKE VPN Tunnels 🙂
Step 1
To bring up a VPN tunnel you need to generate some “Interesting Traffic” Start by attempting to send some traffic over the VPN tunnel.
User Access Verification
Password:
Type help or '?' for a list of available commands.
PetesASA> enable
Password: ********
PetesASA# show crypto isakmp
[/box]
You may see a lot more information if you have Existing VPN tunnels, but what you are looking for is this,
[box]
IKEv1 SAs:
Active SA: 2
Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 2
1 IKE Peer: 123.123.123.123
Type : L2L Role : initiator
Rekey : no State : MM_ACTIVE <<YOUR SIDE BROUGHT THE VPN UP
There are no IKEv2 SAs
[/box]
If you see MM_ACTIVE (This means phase 1 has completed in Main Mode, and is active) So phase 1 has completed successfully, you need to jump forward and troubleshoot Phase 2.
Note: If you see AG_{something} this means you are trying to bring the tunnel up in aggressive mode!
If there is nothing listed at all – then your side is not even trying to bring up the tunnel. Try and generate a lot of VPN traffic – Like a persistent ping {ping 192.168.1.1 -t} and issue the show crypto isakmp command a few times to be sure. if you never see anything then its not getting as far as phase 1!
If your still reading this, then your problem is with Phase 1, and you have an ISAKMPSA state error.
ISAKMP SA MESSAGE STATES (On the Initiator)
MM_WAIT_MSG2
Message 1 has been sent to the responder but there has been no reply.
Causes:
1. There is no network connectivity to the firewallsecurity device at the other end, can you ping it?
2. The IP address of the far firewall is incorrect in the tunnel-group, issue a “show run tunnel-group” command, check you have a tunnel group with the correct IP address.
3. The IP address in the “Crypto Map” is incorrect, issue a “show run crypto map” command and check the line that ends “crypto map {name} {number} set peer xxx.xxx.xxx.xxx” to make sure.
4. You do not have a matching phase 1 policy with the other end, issue a “show run crypto isakmp” command make sure the other end has a matching policy, if you cant check the other end then generate some VPN traffic, issue the following command and check for the following,
[box]
EXAMPLE PHASE 1 POLICIES DONT MATCH
Password: Type help or ‘?’ for a list of available commands. PetesASA> en Password: ******** PetesASA#debug crypto isakmp 200
<<<<<<<LOTS Of DEBUG TEXT REMOVED>>>>>>>
Apr 01 14:48:48 [IKEv1]: IP = 123.123.123.123, IKE_DECODE RECEIVED Message (msgid=ce4a3ffe) with payloads : HDR + NOTIFY (11) + NONE (0) total length : 56 Apr 01 14:48:48 [IKEv1]: IP = 123.123.123.123, Received an un-encrypted NO_PROPOSAL_CHOSEN notify message, dropping Apr 01 14:48:48 [IKEv1]: IP = 123.123.123.123, Information Exchange processing failed
<<<<<<<LOTS Of DEBUG TEXT REMOVED>>>>>>>
[/box]
MM_WAIT_MSG4
The Phase 1 Policies have been agreed with both peers, the initiator is waiting for the responder to send it its keying information. I’ve seen two things cause this.
1. Different Vendors equipment talking the the ASA, or simply the version of OS on the ASA have been different.
2. There is a comms error, check there’s no router with firewall capabilities in the link.
MM_WAIT_MSG6
If there’s a firewall ‘in-between’ make sure UDP port 4500 is open for both peers.
Check your Pre-Shared Keys match on the ASA issue a “more system:running-config” then keep pressing the space bar till you see the tunnel- group and shared key
Again if you can’t check the other end then issue the following debug and the following will tell you if there is a key mismatch.
This error can also be seen if one end has PFS set and the other end does not. In this case the error will appear and dissapear and the connection is repeatedly “torn down”
e.g
crypto map outside_map 20 set pfs [box]
EXAMPLE PHASE 1 PRE SHARED KEYS DONT MATCH
Password: Type help or ‘?’ for a list of available commands. PetesASA> en Password: ******** PetesASA#debug crypto isakmp 200
<<<<<<<LOTS Of DEBUG TEXT REMOVED>>>>>>>
Apr 01 15:11:47 [IKEv1]: IP = 123.123.123.123, IKE_DECODE RECEIVED Message (msgid=5456d64e) with payloads : HDR + NOTIFY (11) + NONE (0) total length : 56 Apr 01 15:11:47 [IKEv1]: Group = 123.123.123.123, IP = 123.123.123.123, Received an un-encrypted PAYLOAD_MALFORMED notify message, dropping Apr 01 15:11:47 [IKEv1]: Group = 123.123.123.123, IP = 123.123.123.123, Error, peer has indicated that something is wrong with our message. This could indicate a pre-shared key mismatch. Apr 01 15:11:47 [IKEv1]: Group = 123.123.123.123, IP = 123.123.123.123, Information Exchange processing failed
<<<<<<<LOTS Of DEBUG TEXT REMOVED>>>>>>>
[/box]
ISAKMP SA MESSAGE STATES (On the Responder)
MM_WAIT_MSG3
The Phase 1 Policies have been agreed with both peers, the responder is waiting for the initiator to send it its keying information. I’ve seen two things cause this.
1. Different Vendors equipment talking the the ASA, or simply the version of OS on the ASA have been different.
2. There is a comms error, check there’s no router with firewall capabilities in the link.
3. I’ve seen this on a VPN from a VMware Edge Gateway, that had PFS (perfect forward secrecy) enabled, and the ASA did not.
If there’s a firewall ‘in-between’ make sure UDP port 4500 is open for both peers.
Here’s an Example of Phase one completing message by message successfully.
MESSAGE 1 (Leaving the Initiator)
[box]
Apr 01 11:38:51 [IKEv1]: IP = 123.123.123.123, IKE Initiator: New Phase 1, Intf inside, IKE Peer 123.123.123.123 local Proxy Address 192.168.1.0, remote Proxy Address 172.16.1.0, Crypto map (outside_map) Apr 01 11:38:51 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing ISAKMP SA payload Apr 01 11:38:51 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing NAT-Traversal VID ver 02 payload Apr 01 11:38:51 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing NAT-Traversal VID ver 03 payload Apr 01 11:38:51 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing NAT-Traversal VID ver RFC payload Apr 01 11:38:51 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing Fragmentation VID + extended capabilities payload Apr 01 11:38:51 [IKEv1]: IP = 123.123.123.123, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 168
[/box]
MESSAGE 2 (Arriving at the Responder)
[box]
Apr 01 11:38:52 [IKEv1]: IP = 123.123.123.123, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 117
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, processing SA payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, Oakley proposal is acceptable
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, processing VID payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing ke payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing nonce payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing Cisco Unity VID payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing xauth V6 VID payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, Send IOS VID
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, Constructing ASA spoofing IOS Vendor ID payload (version: 1.0.0, capabilities: 20000001)
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, constructing VID payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, Send Altiga/Cisco VPN3000/Cisco ASA GW VID
[/box]
MESSAGE 3 (Leaving the Initiator)
[box]
Apr 01 11:38:52 [IKEv1]: IP = 123.123.123.123, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + KE (4) + NONCE (10) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 256
[/box]
MESSAGE 4 (Arriving at the Initiator)
[box]
Apr 01 11:38:52 [IKEv1]: IP = 123.123.123.123, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + KE (4) + NONCE (10) + NONE (0) total length : 228
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, processing ke payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, processing ISA_KE payload
Apr 01 11:38:52 [IKEv1 DEBUG]: IP = 123.123.123.123, processing nonce payload
[/box]
MESSAGE 5 (Leaving the Initiator)
[box]
Apr 01 11:38:52 [IKEv1]: IP = 123.123.123.123, Connection landed on tunnel_group 123.123.123.123
Apr 01 11:38:52 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, Generating keys for Initiator...
Apr 01 11:38:52 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, constructing ID payload
Apr 01 11:38:52 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, constructing hash payload
Apr 01 11:38:52 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, Computing hash for ISAKMP
Apr 01 11:38:52 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, constructing dpd vid payload
Apr 01 11:38:52 [IKEv1]: IP = 123.123.123.123, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + ID (5) + HASH (8) + VENDOR (13) + NONE (0) total length : 84
[/box]
MESSAGE 6 (Arriving at the Initiator)
[box]
Apr 01 11:38:53 [IKEv1]: IP = 123.123.123.123, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + ID (5) + HASH (8) + NONE (0) total length : 64
Apr 01 11:38:53 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, processing ID payload
Apr 01 11:38:53 [IKEv1 DECODE]: Group = 123.123.123.123, IP = 123.123.123.123, ID_IPV4_ADDR ID received 123.123.123.123
Apr 01 11:38:53 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, processing hash payload
Apr 01 11:38:53 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, Computing hash for ISAKMP
Apr 01 11:38:53 [IKEv1]: IP = 123.123.123.123, Connection landed on tunnel_group 123.123.123.123
Apr 01 11:38:53 [IKEv1 DEBUG]: Group = 123.123.123.123, IP = 123.123.123.123, Oakley begin quick mode
Apr 01 11:38:53 [IKEv1 DECODE]: Group = 123.123.123.123, IP = 123.123.123.123, IKE Initiator starting QM: msg id = 26f952ae
Apr 01 11:38:53 [IKEv1]: Group = 123.123.123.123, IP = 123.123.123.123, PHASE 1 COMPLETED
[/box]
Note: You can debug Phase 1 traffic on a particular tunnel, with the following command.
debug crypto condition peer 123.123.123.123
or, simply;
debug crypto isakmp
Phase 1 Debug Errors
[box]
Petes-ASA((config)# debug crypto isakmp
Feb 29 11:49:08 [IKEv1]Group = 123.123.123.123, IP = 123.123.123.123, QM FSM error (P2 struct &0x00007fda92308b10, mess id 0xc02b7a5d)!
Feb 29 11:49:08 [IKEv1]Group = 123.123.123.123, IP = 123.123.123.123, Removing peer from correlator table failed, no match!
Feb 29 11:49:08 [IKEv1]Group = 123.123.123.123, IP = 123.123.123.123, Session is being torn down. Reason: crypto map policy not found
[/box]
This was due to more than one misconfiguration, firstly the source and destination network objects in the interesting traffic ACL were the wrong way round! (Don’t forget to check your static NAT statement as well). And the TRANSFORM SET didn’t match, (sometimes you can see phase one established but then it disappears).
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
[box]
Petes-ASA((config)# debug crypto ikev1
Apr 19 16:36:10 [IKEv1]IP = 123.123.123.123, No Group found by matching OU(s) from ID payload: Unknown
Apr 19 16:36:10 [IKEv1]IP = 123.123.123.123, Trying to find group via IKE ID...
Apr 19 16:36:10 [IKEv1]IP = 123.123.123.123, Trying to find group via IP ADDR...
Apr 19 16:36:10 [IKEv1]IP = 123.123.123.123, Connection landed on tunnel_group 123.123.123.123
Apr 19 16:36:10 [IKEv1 DEBUG]Group = 123.123.123.123, IP = 123.123.123.123, peer ID type 2 received (FQDN)Apr 19 16:36:10 [IKEv1]Group = 123.123.123.123, IP = 123.123.123.123, Unable to compare IKE ID against peer cert Subject Alt Name
Apr 19 16:36:10 [IKEv1 DEBUG]Group = 123.123.123.123, IP = 123.123.123.123, IKE MM Responder FSM error history (struct &0x00007ffd9e230670) <state>, <event>: MM_DONE, EV_ERROR-->MM_BLD_MSG6, EV_COMPARE_IDS-->MM_BLD_MSG6, EV_CERT_OK-->MM_BLD_MSG6, NullEvent-->MM_BLD_MSG6, EV_ACTIVATE_NEW_SA-->MM_BLD_MSG6, NullEvent-->MM_BLD_MSG6, EV_VALIDATE_CERT-->MM_BLD_MSG6, EV_UPDATE_CERT
Apr 19 16:36:10 [IKEv1 DEBUG]Group = 123.123.123.123, IP = 123.123.123.123, IKE SA MM:3a1ed893 terminating: flags 0x01018002, refcnt 0, tuncnt 0
Apr 19 16:36:10 [IKEv1 DEBUG]Group = 123.123.123.123, IP = 123.123.123.123, sending delete/delete with reason message
[/box]
The ASA did not like the certificate presented by the remote peer, (Even though is was a good cert issued by NDES). To get past this you need to make a change to the tunnel group.
Petes-ASA((config)# debug crypto ikev1
%ASA-3-717009: Certificate validation failed. Peer certificate key usage is invalid, serial number: 6B00002B3F8571E2605FA02883000100002C3E, subject name: hostname=Petes-Router-Petes-HQ.petenetlive.com.
%ASA-3-717027: Certificate chain failed validation. Certificate chain is either invalid or not authorized.
[/box]
The ASA did not like the certificate presented by the remote peer, (Even though is was a good cert issued by NDES). To get past this you need to make a change to the trustpoint on the ASA.
[box]
Petes-ASA(config)# crypto ca trustpoint PNL-Trustpoint
Petes-ASA(config-ca-trustpoint)# ignore-ipsec-keyusage
[/box]
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
[box]
Petes-ASA# debug crypto ikev1
Petes-ASA# Feb 17 12:25:17 [IKEv1]Group = 123.123.123.123, IP = 123.123.123.123, Received encrypted Oakley Main Mode packet with invalid payloads, MessID = 0
Feb 17 12:25:17 [IKEv1]Group = 212.20.251.44, IP = 123.123.123.123, ERROR, had problems decrypting packet, probably due to mismatched pre-shared key. Aborting
Feb 17 12:25:23 [IKEv1]IP = 123.123.123.123, Header invalid, missing SA payload! (next payload = 4)
\
[/box]
Amazingly this had nothing to do with a mismatched pre shared key, the other end was set to use PFS (Perfect Forward Secrecy,) and my end (the ASA) was not.
If you have got this far the next step is to troubleshoot Phase 2
Related Articles, References, Credits, or External Links