This week I had a client who had a head office and three satellite sites. They had old firewalls (a 5510 and 5505’s), and my firm had installed FTTC circuits, into the sites for them. My job was to reconfigure the firewalls and the site to site VPN tunnels (each site had a tunnel to the other sites), then disconnect their old ADSL connections, change the firewalls public IP, then connect to the shiny new FTTC circuits.
To save on downtime, my plan was to create new tunnel-groups for all the new IP addresses with the same shared-secrets, then add the new IPs as an alternative crpytomap peers. That way I could migrate all the sites and , the only downtime would be when I changed the firewall to the new IP and plugged into the new router, cool eh?
All was going well until I hit the third satellite site and tried to add a second VPN peer like so…
[box]crypto map outside_map 1 set peer {new-ip-address} {old-ip-address} [/box]
It returned this error;
ERROR: Multiple Peers can be specified only with originate-only connections
Solution
None of the other sites had done this, and I’ve done redundant VPN configs many times, (see the failover ISP article at the bottom of the page.) Never had I seen this error?
I made the ‘mistake‘ of adding;
[box]crypto map outside_map 1 set connection-type originate-only [/box]
A few minutes later I got an email “That sites VPNs have all gone down?”. On investigation the remote site thought the tunnel was up, (it was even encrypting and decrypting layer two traffic?) The main site didn’t even say phase one was attempting. I changed all the crypto maps back to a single peer IP and removed the ‘connection-type originate-only’ from all the crypto entries as well, everything started working again?
I found a bug report for something similar (CSCsd21514) but that affected version 7, I did a show version on the firewall it was running 7.2 (eeurgh.) I updated it to 8.3, (yes I could go to 9 but lets not tempt fate). Problem disappeared, it accepted the redundant VPN config and everything worked, (I flipped the circuit on this problem firewall this morning and downtime was less than 10 seconds).
Related Articles, References, Credits, or External Links
I’ve been posting domain time articles for a long time, and on more than one occasion I’ve really needed to take my Windows time from a Cisco Device and failed miserably. I’ve even used third party NTP software to solve this problem on my own test network.
On a client network, my colleague deployed ACS5 this week, I secured the ASA5585-X for AAA and it failed authentication. Logging revealed a clock skew error, so we manually set the time on the domain PDC. Within half an hour it was failing. The network topology prevented me syncing to a public NTP server from the domain PDC.
We did however have all the network devices syncing from a public time source, if only we could use one of those?
Solution
Step 1 Configure NTP on your Cisco Device.
Here I’m using a 7200 Router in GNS3, the NTPIP addresses I use are UK based NTP servers, I suggest you replace them with some public NTP servers on your own continent. I’m using two for redundancy.
[box]
Petes-Router#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
Petes-Router(config)#ntp server 130.88.202.49 prefer
Petes-Router(config)#ntp server 194.35.252.7
[/box]
NOTE: You need to force the Cisco device to advertise itself with a low stratum, typically the lower the stratum, the closer to atomic time you are supposed to be, (so we are actually forcing the device to lie, but if we don’t, Windows wont trust it!)
[box]
Petes-Router(config)#ntp master 5
[/box]
It can take a while for NTP, (go and have a coffee), then check it’s synchronised, DO NOT proceed until the Cisco device has synchronised.
[box]
R1#show ntp statusClock is synchronized, stratum 5, reference is 127.127.1.1
nominal freq is 250.0000 Hz, actual freq is 250.0000 Hz, precision is 2**19
ntp uptime is 364600 (1/100 of seconds), resolution is 4000
reference time is D898D3A0.319A96D4 (23:05:04.193 GMT Wed Feb 25 2015)
clock offset is 0.0000 msec, root delay is 0.00 msec
root dispersion is 0.26 msec, peer dispersion is 0.23 msec
loopfilter state is 'CTRL' (Normal Controlled Loop), drift is -0.000000000 s/s
system poll interval is 16, last update was 3 sec ago.
[/box]
Step 2 Configure Windows to use Cisco NTP Time
In the past I’ve said “Windows Does not use NTP, it uses Win32 Time” This is not strictly true, it does use NTP, but by default it uses ‘Symmetric Active Mode NTP’ and your Cisco Device expects its NTP requests to be submitted via ‘Client Mode NTP‘. (See MS KB 875424 for more info).
Note: By default Windows Domains take their time from the PDC emulator, carry this procedure out on that server!
Open an elevated command prompt and execute the following commands (the Cisco device IP is shown in red, change accordingly);
[box]
w32tm /config /manualpeerlist:"123.123.123.148",0x8 /syncfromflags:MANUAL
net stop "windows time"
net start "windows time"
w32tm /resync
Note: If you want to specify TWO Cisco devices, use the following syntax
w32tm /config /manualpeerlist:"123.123.123.148,123.123.123.149",0x8 /syncfromflags:MANUAL
[/box]
Now in the Servers System log, you should see the following two events logged.
Event ID 37
Log Name: System
Source: Microsoft-Windows-Time-Service
Date: 25/02/2015 22:33:19
Event ID: 37
Task Category: None
Level: Information
Keywords:
User: LOCAL SERVICE
Computer: 2012-DC-CA.petenetlive.com
Description:
The time provider NtpClient is currently receiving valid time data from 123.123.123.148,
0x8 (ntp.m|0x8|0.0.0.0:123->123.123.123.148:123).
Event ID 35
Log Name: System
Source: Microsoft-Windows-Time-Service
Date: 25/02/2015 22:21:17
Event ID: 35
Task Category: None
Level: Information
Keywords:
User: LOCAL SERVICE
Computer: 2012-DC-CA.petenetlive.com
Description:
The time service is now synchronizing the system time with the time source 123.123.123.148,
0x8 (ntp.m|0x8|0.0.0.0:123->123.123.123.148:123).
Windows and Cisco NTP Problems and Errors
Event ID 47
Log Name: System
Source: Microsoft-Windows-Time-Service
Date: 25/02/2015 22:11:07
Event ID: 47
Task Category: None
Level: Warning
Keywords:
User: LOCAL SERVICE
Computer: 2012-DC-CA.petenetlive.com
Description:
Time Provider NtpClient: No valid response has been received from manually configured
peer 123.123.123.148 after 8 attempts to contact it. This peer will be discarded as a
time source and NtpClient will attempt to discover a new peer with this DNS name. The
error was: The peer is unreachable.
On your Cisco Device you will see debug output like so, (it will repeat 8 times);
[box]
Petes-Router#debug ntp all
NTP events debugging is on
NTP core messages debugging is on
NTP clock adjustments debugging is on
NTP reference clocks debugging is on
NTP packets debugging is on
Petes-Router#
000031: Feb 25 22:07:45.831: NTP message received from 123.123.123.151 on interface 'GigabitEthernet0/0' (123.123.123.148).
000032: Feb 25 22:07:45.835: NTP Core(DEBUG): ntp_receive: message received
000033: Feb 25 22:07:45.835: NTP Core(DEBUG): ntp_receive: peer is 0x67A57898, next action is 1.
Petes-Router#
000034: Feb 25 22:07:54.967: NTP message received from 123.123.123.151 on interface 'GigabitEthernet0/0' (123.123.123.148).
000035: Feb 25 22:07:54.967: NTP Core(DEBUG): ntp_receive: message received
000036: Feb 25 22:07:54.971: NTP Core(DEBUG): ntp_receive: peer is 0x67A57898, next action is 1.
Petes-Router#
[/box]
Causes:
This is a pretty generic error, but in this case, one of the following situations can cause this;
1. UDP Port 123 is blocked between Windows and the Cisco NTP device.
2. The Cisco NTP device has not synchronised form a reliable NTP source.
3. The stratum of the Cisco NTP device is to high.
4. Windows is attempting to sync time using ‘Symmetric Active Mode NTP‘ See my comments above.
Related Articles, References, Credits, or External Links
