Cisco Catalist Upgrading 2900, 5500 and 3700 Stacks

KB ID 0001630

Problem

People are often nervous about doing this, I’m not sure why because Cisco have made it painfully simple now. That’s because instead of the old /bin files we used to use, you can now upgrade a switch (or a switch stack) using a .tar file with one command, (and it will also upgrade all the stack members and the firmware on any other network modules you have in the switches at the same time).

Yes it does take a while*, and for long periods of time theres no updated output on the screen, which is worrying if you’ve never done it before.

*Note: The procedure below was updating two 2960-X switches and took about 45-50 minutes. If anyone wants to post any further timings below as a help to others, state the switch types and quantities, and versions you used, etc.

Solution

First things first, BACK UP YOUR SWITCH CONFIG. I also have a habit of copying out the original .bin file from the flash to my TFTP server as an extra ‘belt and braces’ precaution, in case everything ‘Goes to hell in a hand cart!’

I find it easier to do this with the update file on a USB Drive, (format the drive as Fat32). If you dont have a USB Drive, or the switch does not have a working USB port then don’t panic, you can use ftp or tftp to upgrade also.

Place your new upgrade .tar file on your USB Drive and insert it into the master switch, you should see the following;

[box]

Dec 19 13:13:18.466: %USBFLASH-5-CHANGE: usbflash0 has been inserted!

[/box]

Note: If yours says usbflash1, or usbflash2 etc. Then that’s just the switch numbering in the stack, use the number it tells you!

Make sure the switch can see your upgrade file;

[box]

Petes-Switch# dir usbflash1:
Dec 19 16:56:45.712: %USBFLASH-5-CHANGE: usbflash0 has been inserted!

Directory of usbflash0:/
 -rw- 37488640 Nov 25 2019 10:08:34 +00:00 c2960x-universalk9-tar.152-7.E0a.tar

8036286464 bytes total (7997743104 bytes free)

[/box]

You can execute the entire upgrade with this one command;

[box]

Petes-Switch# archive download-sw /overwrite usbflash0:/c2960x-universalk9-tar.152-7.E0a.tar

[/box]

Note: If using tftp then use archive download-sw /overwrite tftp:/{ip-of-tftp-server}/{image-name}.tar instead.

It will take quite a long time, as soon as it says extracting xyz….go and have a coffee, wait until it says ‘All software images installed.’

[box]

---LOTS OF OUTPUT OMITTED FOR THE SAKE OF BREVITY---
New software image installed in flash2:/c2960x-universalk9-mz.152-7.E0a
Deleting old files from dc profile dir "flash:/dc_profile_dir"
extracting dc profile file from "flash:/c2960x-universalk9-mz.152-7.E0a/dc_default_profiles.txt" to "flash:/dc_profile_dir/dc_default_profiles.txt"
Deleting old files from dc profile dir "flash2:/dc_profile_dir"
extracting dc profile file from "flash2:/c2960x-universalk9-mz.152-7.E0a/dc_default_profiles.txt" to "flash2:/dc_profile_dir/dc_default_profiles.txt"
All software images installed.

[/box]

Now let’s do a couple of checks just for our ‘peace of mind‘, first make sure the images are in all the relevant switches flash storage;

[box]

Petes-Switch#dir flash1:
Directory of flash:/

    2  -rwx        5486  Dec 19 2019 16:55:40 +00:00  private-config.text
    3  -rwx          33   Aug 7 2019 08:28:12 +00:00  pnp-tech-time
    4  -rwx       11114   Aug 7 2019 08:28:14 +00:00  pnp-tech-discovery-summary
    5  -rwx        3096  Dec 19 2019 16:55:40 +00:00  multiple-fs
  699  drwx         512  Dec 19 2019 17:35:25 +00:00  c2960x-universalk9-mz.152-7.E0a
  480  drwx         512  Dec 19 2019 17:35:28 +00:00  dc_profile_dir
  696  -rwx         796   Aug 9 2019 09:48:30 +00:00  vlan.dat
  698  -rwx        7539  Dec 19 2019 16:55:40 +00:00  config.text

122185728 bytes total (84392960 bytes free)
Petes-Switch#dir flash2:
Directory of flash2:/

    2  -rwx        5486  Dec 19 2019 16:55:40 +00:00  private-config.text
    3  -rwx          33   Aug 7 2019 08:32:38 +00:00  pnp-tech-time
    4  -rwx       11126   Aug 7 2019 08:32:40 +00:00  pnp-tech-discovery-summary
    5  -rwx        7539  Dec 19 2019 16:55:40 +00:00  config.text
    6  drwx         512  Dec 19 2019 17:35:26 +00:00  c2960x-universalk9-mz.152-7.E0a
  481  drwx         512  Dec 19 2019 17:35:28 +00:00  dc_profile_dir
  696  -rwx        3096   Aug 8 2019 10:21:29 +00:00  multiple-fs
  697  -rwx         796  Dec 11 2019 10:55:22 +00:00  vlan.dat
  698  -rwx        7514  Dec 19 2019 16:55:40 +00:00  config.text.backup
  699  -rwx        5486  Dec 19 2019 16:55:40 +00:00  private-config.text.backup

122185728 bytes total (84378624 bytes free)

[/box]

Note: Repeat for each switch in the stack, if you have further switches.

Why does it not have .tar or .bin on the end? Because it’s a folder 🙂

The let’s make sure the ‘boot variable‘ in the device is set to use the new image;

[box]

Petes-Switch# show boot
BOOT path-list      : flash:/c2960x-universalk9-mz.152-7.E0a/c2960x-universalk9-mz.152-7.E0a.bin
Config file         : flash:/config.text
Private Config file : flash:/private-config.text
Enable Break        : yes
Manual Boot         : no
Allow Dev Key         : yes
HELPER path-list    :
Auto upgrade        : yes
Auto upgrade path   :
Boot optimization   : disabled
NVRAM/Config file
      buffer size:   524288
Timeout for Config
          Download:    0 seconds
Config Download
       via DHCP:       disabled (next boot: disabled)
-------------------
Switch 2
-------------------
BOOT path-list      : flash:/c2960x-universalk9-mz.152-7.E0a/c2960x-universalk9-mz.152-7.E0a.bin
Config file         : flash:/config.text
Private Config file : flash:/private-config.text
Enable Break        : yes
Manual Boot         : no
Allow Dev Key         : yes
HELPER path-list    :
Auto upgrade        : no
Auto upgrade path   :

[/box]

All looks good save the config and reload the stack.

[box]

Petes-Switch# write mem
Petes-Switch# reload
Proceed with reload? [confirm] {Enter}

Dec 19 17:38:50.952: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload command.

[/box]

Time for another coffee while it’s reloading the stack, when it’s back up you can check it was successful like so;

[box]

Petes-Switch# show version
---LOTS OF OUTPUT OMITTED FOR THE SAKE OF BREVITY---
Switch Ports Model                     SW Version            SW Image
------ ----- -----                     ----------            ----------
*    1 54    WS-C2960X-48TS-L          15.2(7)E0a            C2960X-UNIVERSALK9-M
     2 54    WS-C2960X-48TS-L          15.2(7)E0a            C2960X-UNIVERSALK9-M

[/box]

Related Articles, References, Credits, or External Links

NA

Cisco IOS – How To Find VLAN IPs (SVI’s)

KB ID 0001258 

Problem

If you have a complicated network, you can spend more time finding out how it’s configured, than actually doing any work on it!

Today I had a client that needed some changes made on their LAN, I knew their name, and their network address, and common sense told me which of the core switches they were connected to.

Solution

A quick search on the client name told me what VRF they were in, and what VLAN they were in (3000), let’s have a look at that;

[box]

Petes-Core-SW#show run vlan 3000
Building configuration...

Current configuration:
!
vlan 3000
 name CORP:NET
end

[/box]

That doesn’t yield much more than I already know, so I can either do this and get a LOT of information;

[box]

Petes-Core-SW#show interfaces vlan 3000
Vlan3000 is up, line protocol is up
 Hardware is EtherSVI, address is c062.6be3.3000 (bia c062.6be3.9d40)
 Description: CORP:NET
 Internet address is 192.168.1.100/24
 MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
 reliability 255/255, txload 1/255, rxload 1/255
 Encapsulation ARPA, loopback not set
 Keepalive not supported
 ARP type: ARPA, ARP Timeout 04:00:00
 Last input 00:00:00, output never, output hang never
 Last clearing of "show interface" counters never
 Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
 Queueing strategy: fifo
 Output queue: 0/40 (size/max)
 5 minute input rate 254000 bits/sec, 115 packets/sec
 5 minute output rate 504000 bits/sec, 119 packets/sec
 L2 Switched: ucast: 22179333 pkt, 1561846492 bytes - mcast: 0 pkt, 0 bytes
 L3 in Switched: ucast: 471521755 pkt, 367932934560 bytes - mcast: 0 pkt, 0 bytes
 L3 out Switched: ucast: 493390206 pkt, 464908773459 bytes - mcast: 0 pkt, 0 bytes
 475554223 packets input, 366284328453 bytes, 0 no buffer
 Received 0 broadcasts (1116 IP multicasts)
 0 runts, 0 giants, 0 throttles
 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
 493591347 packets output, 462947525840 bytes, 0 underruns
 0 output errors, 0 interface resets
 0 output buffer failures, 0 output buffers swapped out

[/box]

Or a more sensible;

[box]

Petes-Core-SW#show run interface vlan 3000
Building configuration...



Current configuration : 160 bytes
!
interface Vlan3000
 description CORP:NET
 mac-address c062.6be3.3000
 vrf forwarding CORP:NET
 ip address 192.168.1.100 255.255.255.0
end

[/box]

Find What VLAN An IP Address Is In

If you have the opposite problem, i.e. you know the IP, (or a part of the IP). You can get the VLAN number like so;

[box]

Petes-Core-SW#show ip int br | incl 192.168.1.100
Vlan3000               192.168.1.100     YES NVRAM  up                    up

[/box]

 

Related Articles, References, Credits, or External Links

NA

Cisco IOS ‘Crypto’ Unrecognized Command?

KB ID 0001246

Problem

I was working on a Cisco 3750-G last week, and I was in the process of setting up SSH access. When I went to generate the crypto key and enable SSH, It fired an error at me. In fact it wouldn’t execute any crypto commands;

[box]

Core-SW(config)#crypto ?
% Unrecognized command

[/box]

 

Now I have seen this before, (but not for a while). You need to be running a K9 version of the code. A quick ‘show version‘ will tell you.

[box]

Core-SW#show version
Cisco IOS Software, C3750 Software (C3750-IPSERVICES-M), Version 12.2(25)SEB2, RELEASE SOFTWARE (fc1)

[/box]

Solution

So you can either, just use TELNET to manage the switch, or upgrade it to a K9 version of the code, (in my case c3750-ipserviceslmk9-tar.122-55.SE11). I chose to upgrade.

Upgrade Cisco Catalyst 3750G

First I tried to TFTP in the bin file, but I kept getting a lot of ‘O’ (‘out of sequence’ errors,) and the process failed. After discussions with a colleague, he recommended I simply use the archive-download command and use the TAR upgrade file instead.

WARNING: These old G series switches only have a 16MB flash in them, the TAR file is about 13.5MB you will need to delete the boot file and folder from flash to upgrade the IOS, but I suggest you copy the bin file out to TFTP in case theres a drama, and you need top copy it back in, before you continue. (In fact backup the switch config as well to be on the safe side!)

Setup your TFTP server, and download your image (c3750-ipserviceslmk9-tar.122-55.SE11.tar).

Delete the .bin file from the flash on the switch, and any associated folders (Note: to delete a folder, the syntax is different). Obviouly you may have different files and folders.

[box]

Core-SW delete flash:/c3750-ipservices-mz.122-25.SEB2.bin
Core-SW delete /force /recursive flash:/c3750-i5-mz.121-19.EA1d

[/box]

Perform the upgrade;

[box]

Core-SW archive download-sw /overwrite tftp://192.168.254.250/c3750-ipserviceslmk9-tar.122-55.SE11.tar

[/box]

It can take 10 minutes or so, but when complete, check the boot variable is set to the new image, and then reload the switch (It may restart a couple of times that’s OK).

[box]

Core-SW(config)#do show boot
BOOT path-list : flash:c3750-ipservicesk9-mz.122-55.SE11/c3750-ipservicesk9-mz.122-55.SE11.bin
Config file : flash:/config.text
Private Config file : flash:/private-config.text
Enable Break : no
Manual Boot : no
HELPER path-list :
Auto upgrade : yes
Core-SW(config)#do reload
[CONFIMRM] Y

[/box]

Related Articles, References, Credits, or External Links

 NA

Cisco IOS – DHCP Helper (DHCP Relay) – IP-Helper Setup

KB ID 0001168 

Problem

Cisco documentation calls this a ‘DHCP Relay’, and uses the command IP-Helper, and I usually call this DHCP Helper, just to confuse everyone. To be fair the term DHCP Relay is an industry standard, it’s not particular to Cisco (as you will see later when I Wireshark the traffic).

So If you are reading this you have a DHCP server and you want to use it to lease addresses to clients that are on a different network segment (layer 2, or layer3).

To do that you need an agent to be on the same network segment as the client listening for DHCP requests, when it receives one it talks to the DHCP server on the clients behalf and gets the correct address.

Solution

Example 1 Cisco Router

Here we need to lease two different DHCP scopes to two different network segments, R1 will act as the IP-Helper for both of those networks, R2 and R3 will get their IP addresses from the correct DHCP scope.

This works because each (client facing) interface on R1 has an IP-Helper address defined that points to the DHCP server.

So How Does It Know Which Scope To Lease From? This is because the Router supplies the IP address of a RELAY AGENT, which is just the IP address of the physical interface that intercepted the DHCP request. When it asks for an IP address from the DHCP server, the Server leases an address from the same range, (again I’ve tracked all this in Wireshark below).

IP-Helper Router Configuration

[box]

R1 Config

!
interface GigabitEthernet0/0
 description Uplink to DHCP Server
 ip address 10.2.2.254 255.255.255.0
 negotiation auto
!
interface GigabitEthernet2/0
 description Uplink to 192_168_2_0
 ip address 192.168.2.1 255.255.255.0
 ip helper-address 10.2.2.10
 negotiation auto
!
interface GigabitEthernet3/0
 description Uplink to 192_168_3_0
 ip address 192.168.3.1 255.255.255.0
 ip helper-address 10.2.2.10
 negotiation auto
!
ip route 0.0.0.0 0.0.0.0 10.2.2.10
!


R2 Config

!
interface GigabitEthernet2/0
 description Uplink to R1
 ip address dhcp
 negotiation auto
!
ip route 0.0.0.0 0.0.0.0 GigabitEthernet2/0
!

R3 Config

!
interface GigabitEthernet3/0
 description Uplink to R1
 ip address dhcp
 negotiation auto
!
ip route 0.0.0.0 0.0.0.0 GigabitEthernet3/0
!

[/box]

You can see this works because the DHCP server has matching scopes for both network segments. (Yes one of my test servers is 2003, you’re going to see some Windows XP in a minute!

Well that’s fine for routers, but what about machines? They send a DHCP Discover just like any other client. I’ve replaced one of the routers with an actual machine.

With its network card set to DHCP you will again get a lease from the correct scope, because the Router brokered it for us.

Back on the DHCP server you can see the lease to the windows XP machine entered in the current scope leases, It knows the name of the client because (as you will see below) the relay agent (Router) passed that information (along with the MAC address of the client) to the DHCP server.

Example 2 Cisco Switches

OK, I did the routers first because I find it easier to explain things at layer 3. Not that you can’t create sub interfaces on the router, add those sub interfaces to VLANs, and run DHCP relays from them. But in most cases you will be setting up DHCP helpers on switches. Here the principle is the same but you define the ip-helper on the VLAN, (unless it’s routed port then treat it the same as a router interface). Let’s modern things up a bit, and use a 2012 R2 DHCP server, and some Windows 8 clients.

I need to lease addresses from my second scope to clients in VLAN 200, (the other client and server are in the same VLAN, so that will just work. (Remember a VLAN is a broadcast domain, and DHCP is using broadcasts).

Here’s the two scopes setup on the 2012 server;

And my client, (DHCP Client in VLAN 200) gets the correct IP.

IP-Helper Switch Configuration (VLANS)

[box]

SW1 Config

interface FastEthernet1/0/1
 description Uplink to DHCP Server
 switchport access vlan 100
 switchport mode access
 spanning-tree pordtfast
!
interface FastEthernet1/0/4
 description Uplink 192_168_200_0
 switchport access vlan 200
 switchport mode access
 spanning-tree pordtfast
!
interface FastEthernet1/0/5
 description Uplink 192_168_100_0
 switchport access vlan 100
 switchport mode access
 spanning-tree portfast
!
interface Vlan200
 ip address 192.168.200.1 255.255.255.0
 ip helper-address 192.168.100.10
!

IF YOU HAVE MULTIPLE/FAILOVER IP-HELPERS OR SPLIT SCOPES YOU CAN ADD A SECOND 
ADDRESS LIKE SO;

!
interface Vlan200
 ip address 192.168.200.1 255.255.255.0
 ip helper-address 192.168.100.10
 ip helper-address 192.168.100.15
!

[/box]

Analysing (Packet-Sniffing) DHCP Relay Sequence with Wireshark

Other packet sniffers are available, but I’ve got a soft spot for Wireshark. To filter DHCP traffic you can use the following ‘filter’.

bootp.option.type == 53

DHCP works by using four messages, (which I remember using the acronym DORA: Discover, Offer, Request, Acknowledge). If you sniff the traffic on the DHCP server, you can watch this process being brokered by your DHCP Relay Agent.

Discover

Offer

Request

Acknowledge

And just to prove it’s not all ‘smoke and mirrors’, here’s the client with the leased address, showing a matching MAC address, and hostname.

Related Articles, References, Credits, or External Links

NA

Using OSPF over DMVPN

KB ID 0001151 Dtd 03/02/16

Problem

This article is a supplement to the earlier one on Setting Up DMVPN. It covers how to use OSPF over the top of DMVPN.

This is the topology I’m going to use;

As I’ve said (above) this is not a run though on setting up DMVPN, but if you want to spin it up in GNS3, or on the test bench, here’s the DMVPN config;

[box]

Hub Site

configure terminal
interface Tunnel10
ip address 192.168.254.1 255.255.255.0
ip nhrp map multicast dynamic
ip nhrp network-id 1
tunnel source 10.10.10.10
tunnel mode gre multipoint
exit
!
crypto isakmp policy 10
authentication pre-share
encryption aes
group 2
crypto isakmp key DMVPNdf3fdc829fj2iw2ndh2ds82 address 0.0.0.0
crypto ipsec transform-set TFS-PNL esp-aes esp-sha-hmac
crypto ipsec profile PF-PNL
set transform-set TFS-PNL
interface Tunnel10
tunnel protection ipsec profile PF-PNL
exit

Spoke1

configure terminal
interface Tunnel10
ip address 192.168.254.2 255.255.255.0
ip nhrp map 192.168.254.1 10.10.10.10
ip nhrp map multicast 10.10.10.10
ip nhrp network-id 1
ip nhrp nhs 192.168.254.1
tunnel source 11.11.11.11
tunnel mode gre multipoint
exit
!
configure terminal
crypto isakmp policy 10
authentication pre-share
encryption aes
group 2
crypto isakmp key DMVPNdf3fdc829fj2iw2ndh2ds82 address 0.0.0.0
crypto ipsec transform-set TFS-PNL esp-aes esp-sha-hmac
crypto ipsec profile PF-PNL
 set transform-set TFS-PNL
interface Tunnel10
tunnel protection ipsec profile PF-PNL
exit

Spoke2

configure terminal
interface Tunnel10
ip address 192.168.254.3 255.255.255.0
ip nhrp map 192.168.254.1 10.10.10.10
ip nhrp map multicast 10.10.10.10
ip nhrp network-id 1
ip nhrp nhs 192.168.254.1
tunnel source 21.21.21.21
tunnel mode gre multipoint
exit
!
configure terminal
crypto isakmp policy 10
authentication pre-share
encryption aes
group 2
crypto isakmp key DMVPNdf3fdc829fj2iw2ndh2ds82 address 0.0.0.0
crypto ipsec transform-set TFS-PNL esp-aes esp-sha-hmac
crypto ipsec profile PF-PNL
 set transform-set TFS-PNL
interface Tunnel10
tunnel protection ipsec profile PF-PNL
exit

[/box]

Solution

To add OSPF over the top, here’s the additional config;

[box]

Hub Site

interface tunnel 10
ip mtu 1400
ip ospf network broadcast
ip ospf hello-interval 30
ip ospf priority 255
!
router ospf 1
router-id 192.168.254.1
network 192.168.254.0 0.0.0.255 area 52
network 192.168.0.0 0.0.0.255 area 52

Spoke 1

interface tunnel 10
ip mtu 1400
ip ospf network non-broadcast
ip ospf priority 0
!
router ospf 1
router-id 192.168.254.2
network 192.168.254.0 0.0.0.255 area 52
network 192.168.1.0 0.0.0.255 area 52

Spoke 2

interface tunnel 10
ip mtu 1400
ip ospf network non-broadcast
ip ospf priority 0
!
router ospf 1
router-id 192.168.254.3
network 192.168.254.0 0.0.0.255 area 52
network 192.168.2.0 0.0.0.255 area 52

[/box]

Related Articles, References, Credits, or External Links

Cisco – Configuring Dynamic Multipoint Virtual Private Networks DMVPN

Implementing GDOI into DMVPN

PPTP VPN – Enable Split Tunneling

 

KB ID 0000997 

Problem

I was asked yesterday, “When you get five minutes, I need split tunneling setup, when I VPN into a network I lose Internet connectivity”. On inspection he was using the Microsoft VPN client, I jumped on the VPN device to discover it was a Cisco IOS router.

What I discovered was, unlike the firewall VPN’s I’m used to, you DONT set split tunneling up on the VPN device, you set it up on the client, (and its a bit clunky – sorry!)

Solution

1. Windows Key + R > ncpa.cpl {Enter} > Locate the VPN connection > Right Click > Properties > Networking > Internet Protocol Version 4 (TCP/IPv4) > Properties > Advanced.

2. Untick “Use default gateway on remote network” > OK > OK > OK.

BE AWARE: There is a downside to doing this, as site visitor Clayton Webb points out;

“Unchecking that default gateway is a godsend, until end users use their laptops for torrents, malware, etc. If you have the time I’d recommend a direct access setup for company equipment. VPN w/ NPS health validators for non-company equipment.”

I agree, I would only ever see this as a temporary solution for the ‘technically savvy’.

3. WARNING: At this point you may find you can connect to the VPN, and your Internet now works, (hooray!) But you can no longer talk to any servers or systems on the site you are VPN’d into. This is a Windows routing problem, lets take a look at what IP address I’m getting from the VPN Device.

Above you can see Ive got an IP address of 192.168.2.207, and in my case I don’t have a default gateway (this is not unusual, yours may be the same or you may have a default gateway as well).

4. If you open a command window and issue a ‘route print’ command, you can see the reason I don’t have a default gateway is my gateway is may actual IP address (again this is not unusual, In my case I need to remember 192.168.2.207, if you have a different gateway listed thats the one you need to take notice of).

5. Run a command window (as administrator) and issue a ‘route add‘, command like below.

Note: -P Adds the route persistently (will remain after a reboot). The network you are trying to get to will probably be a different network, to the network IP you are being leased to you by the VPN device. If you have multiple networks you will need a ‘route add’ for each one.

6. To demonstrate; below I can’t get to 192.168.1.1, I then enter the ‘route add’ command, and after that I can get to 192.168.1.1.

Note: I’m not adding my route as persistent!

 

Related Articles, References, Credits, or External Links

Cisco ASA – Enable Split Tunnel for IPSEC / SSLVPN / WEBVPN Clients

NDES – Fails to Issue Certificates (Signature Algorithm)

KB ID 0001021 

Problem

I was trying to enroll some ASA firewalls to NDES to get some certificates. Each time the process failed with the following error.

[box]

% Error in receiving Certificate Authority certificate: status = FAIL, cert length = 0[/box]

That’s a pretty generic error, and does not give me a lot to go on. So I thought I would try from another network device, (a Cisco Catalyst switch). It’s a little easier to ‘debug’ the process in IOS rather than on the ASA, so that’s what I did.

 

[box]

Enable NDES Debugging 

Petes-Router# debug crypto pki messages
Crypto PKI Msg debugging is on
Petes-Router# debug crypto pki transactions
Crypto PKI Trans debugging is on
Petes-Router#

[/box]

The switch failed with the same error as the firewall but at least now I had some debugging information.

[box]

Petes-Router# show logg

Jan 4 10:31:11.818: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/21, 
changed state to up
Jan 4 10:32:40.648: CRYPTO_PKI: pki request queued properly
Jan 4 10:32:40.648: CRYPTO_PKI: Sending CA Certificate Request:
GET /CertSrv/mscep/mscep.dll/pkiclient.exe?operation=GetCACert&message=PNL-Trustpoint HTTP/1.0
User-Agent: Mozilla/4.0 (compatible; MSIE 5.0; Cisco PKI)
Host: 192.168.1.100

Jan 4 10:32:40.648: CRYPTO_PKI: locked trustpoint PNL-Trustpoint, refcount is 1
Jan 4 10:32:40.656: CRYPTO_PKI: http connection opened
Jan 4 10:32:40.656: CRYPTO_PKI: Sending HTTP message

Jan 4 10:32:40.656: CRYPTO_PKI: Reply HTTP header:
HTTP/1.0
User-Agent: Mozilla/4.0 (compatible; MSIE 5.0; Cisco PKI)
Host: 192.168.1.100

Jan 4 10:32:40.656: CRYPTO_PKI: unlocked trustpoint PNL-Trustpoint, refcount is 0
Jan 4 10:32:40.656: CRYPTO_PKI: locked trustpoint PNL-Trustpoint, refcount is 1
Jan 4 10:32:40.673: CRYPTO_PKI: unlocked trustpoint PNL-Trustpoint, refcount is 0
Jan 4 10:32:40.673: CRYPTO_PKI: Reply HTTP header:
HTTP/1.1 200 OK
Content-Length: 7946
Content-Type: application/x-x509-ca-ra-cert
Server: Microsoft-IIS/8.5
Date: Wed, 07 Jan 2015 10:30:36 GMT
Connection: close

Content-Type indicates we have received CA and RA certificates.

Jan 4 10:32:40.673: CRYPTO_PKI:crypto_process_ca_ra_cert(trustpoint=PNL-Trustpoint)

Jan 4 10:32:40.673: CRYPTO_PKI: status = 0x722(E_SIGNATURE_ALG_NOT_SUPPORTED : 
signature algorithm not supported): crypto_certc_pkcs7_extract_certs_and_crls failed
Jan 4 10:32:40.673: CRYPTO_PKI: status = 0x722(E_SIGNATURE_ALG_NOT_SUPPORTED : 
signature algorithm not supported): crypto_pkcs7_extract_ca_cert returned
Jan 4 10:32:40.673: CRYPTO_PKI: Unable to read CA/RA certificates.
Jan 4 10:32:40.673: %PKI-3-GETCARACERT: Failed to receive RA/CA certificates.
Jan 4 10:32:40.673: CRYPTO_PKI: transaction GetCACert completed
Petes-Router#
[/box]

So we are getting the CA cert and the RA cert from the NDES server but we can’t read them.

Here’s the slightly less descriptive debug from the ASA firewall.

[box]
Petes-ASA(config)# debug crypto ca transactions
Petes-ASA(config)# crypto ca authenticate PNL-Trustpoint

ERROR: receiving Certificate Authority certificate: status = FAIL, cert length = 0
Petes-ASA(config)# show logg
crypto_certc_pkcs7_extract_certs_and_crls failed (1826):
crypto_certc_pkcs7_extract_certs_and_crls failed
CRYPTO_PKI:crypto_pkcs7_extract_ca_cert returned 1826

Petes-ASA(config)#
[/box]

Solution

I really struggled with this one, the bottom line is the Cisco device can’t read the certificates, and the reason it can’t is actually shown above;

E_SIGNATURE_ALG_NOT_SUPPORTED

What this is telling us is that the signature algorithm that Windows Certificate Services is using can not be understood by the Cisco network devices. At first I thought It might be because I was using Windows Server 2012 R2, and it might have some new security feature.

So I built a test Server in VMware Workstation, and presented an ASA and router to it from GNS3 and it worked first time, (annoyingly). When I looked at the certificates and compared them, and took into account the debug above, I spotted the difference.

If the signature algorithm is set to sha1RSA, it works if it’s set to RSASSA-PSS it fails. To compound my problem even further I have a three tier PKI deployment with an offline root, intermediate (Sub CA), and an issuing CA (Sub CA). And the signature algorithm needs to be correct for EVERY CERTFICIATE IN THE CERTIFICATE PATH (CHAIN).

Why Has This Happened?

Basically when the offline root was created, I followed the instructions for deploying an offline CA as per the instructions on Technet. Before you even install the role, Microsoft recommend you create a CApolicy.inf file with the following line in it;

[box]AlternateSignatureAlgorithm=1[/box]

I says that this signature algorithm is more secure, but it’s not compatible with Windows XP. What IT DOES NOT SAY, is it’s incompatible with Cisco devices wanting to get certificates from NDES!

Note: Executing the following command also enables this;

[box]

Certutil -setreg CAcspAlternateSignatureAlgorithm 1

[/box]

What this does is change a registry key, you can revert back by carrying out the following steps;

1. Open regedit and Navigate to;

[box]HKEY_LOCAL_MACHINE >SYSTEM > CurrentControlSet > Services > CertSvc > Configuration > {SERVER-NAME} > CSP[/box]

2. Locate the AlternateSignatureAlgorithm value and change it to 0 (zero).

3. Open a command windows as administrator > Restart certificate services.

From this point forward, all new certificates issued by this CA will use the older signature algorithm. So if you renew the CA Certificate the new one will be fine.

WARNING: When renewing the CA Cert MAKE SURE YOU DO NOT generate new keys (or previously issued certificates may stop working!)

If you only have one certificate server you can then simply remove NDES.

Then delete the RA certificates used for NDES.

When NDES is reinstalled the new RA certs will use the correct signature algorithm.

What If You Have a Two or Three Tier PKI Deployment

If like me you have a multi tiered PKI deployment, you need to go all the way back to the Root CA > Fix that > Reissue all the Sub CA certs down the certificate path fixing each tier as you go.

Here’s the process I used, (Use at you own risk and I accept no responsibility if you trash your PKI environment).

Related Articles, References, Credits, or External Links

NA

GNS3 – Error ‘ghostsize is to small for device’

KB ID 0000935 

Problem

While doing a quick lab in GNS3, I tried to add NAT to a router, and it fell over with the following error;

[box]

R3(config-if)#ip nat outside
% NBAR ERROR: parsing stopped
% NBAR Error : Activation failed due to insufficient dynamic memory
% NBAR Error: Stile could not add protocol node
%NAT: Error activating CNBAR on the interface FastEthernet0/0
R3(config-if)#
*Mar 1 00:01:11.655: %SYS-2-MALLOCFAIL: Memory allocation of 10260 bytes failed
from 0x62915CD4, alignment 0
Pool: Processor Free: 28660 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "Exec", ipl= 0, pid= 93, -Traceback= 0x6148BFF8 0x60016604 0x6001C564 0x6001CBBC 0x636756E4 0x62915CDC 0x628F468C 0x628F9DA0 0x628F5968 0x628FA474 0x628F5968 0x628F8344 0x628F5968 0x628F5B2C 0x62928FBC 0x62933A20
*Mar 1 00:01:11.659: %NBAR-2-NOMEMORY: No memory available for StILE lmalloc, -Traceback= 0x6148BFF8 0x62915CF8 0x628F468C 0x628F9DA0 0x628F5968 0x628FA474 0x628F5968 0x628F8344 0x628F5968 0x628F5B2C 0x62928FBC 0x62933A20 0x62920BD0 0x6293DF70 0x6293E2F0 0x61C77C70
R3(config-if)#
*Mar 1 00:01:12.231: %LINEPROTO-5-UPDOWN: Line protocol on Interface NVI0, changed state to up
R3(config-if)#

[/box]

A quick Google told me, this was because I didn’t have enough memory assigned to the router, and I was supposed to change this, save the new setting, and reload the router. But as soon as I did that, this happened. (Note: Seen in GNS3, I could not console to the router).

[box]

GNS3 management console. Running GNS3 version 0.8.6.
Copyright (c) 2006-2013GNS3 Project.

=> *** Warning: ghostsize is to small for device R3. Increase it with the ghostsize option.

[/box]

Solution

1. Locate the filename.net file for your project, and open it.

Note: Usually in C:Users{user-name}GNS3Projects

2. Locate the section that contains your router settings, (you should see the IOS image name). Below that change the RAM value, here I changed it from 128 to 256, then save the file and reopen your GNS3 project.

3. To stop this happening again, whilst in GNS3 > Edit > IOS Images and Hypervisors > Locate the router image, and set the default RAM figure here > Save > Click Test settings to make sure.

Related Articles, References, Credits, or External Links

NA

Cisco Catalyst – Upgrading ‘Stacked’ Switches

KB ID 0001002

Problem

The following procedure was carried out on two Cisco Catalyst 3750 switches.

Solution

1. We can see (above) that we have two switches, but if your connected remotely, best make sure.

[box]

Petes-Stack#show switch
Switch/Stack Mac Address : 0018.7347.a000
                                           H/W   Current
Switch#  Role   Mac Address     Priority Version  State
----------------------------------------------------------
*1       Master 0018.7347.a000     1      0       Ready
 2       Member 0024.f79b.9b00     1      0       Ready

[/box]

2. Lets see what IOS files are in the flash memory on both switches.

[box]

Petes-Stack#dir flash1:
Directory of flash:/

    2  -rwx        5514  Sep 25 2014 14:28:06 +01:00  private-config.text
    3  drwx         192   Mar 1 1993 00:10:57 +00:00  c3750-ipservicesk9-mz.122-55.SE8
   84  -rwx         856  Sep 23 2014 13:24:52 +01:00  vlan.dat
   85  -rwx       15354  Sep 25 2014 14:28:06 +01:00  config.text
   87  -rwx        3096  Sep 25 2014 14:28:06 +01:00  multiple-fs

15998976 bytes total (2406400 bytes free)
Petes-Stack#dir flash2:
Directory of flash2:/

    2  -rwx        5514  Sep 25 2014 14:28:06 +01:00  private-config.text
    3  drwx         192   Mar 1 1993 00:23:02 +00:00  c3750-ipservicesk9-mz.122-55.SE8
   84  -rwx       15354  Sep 25 2014 14:28:06 +01:00  config.text
   85  -rwx        3096   Mar 1 1993 00:04:19 +00:00  multiple-fs
   86  -rwx         856  Sep 23 2014 13:24:52 +01:00  vlan.dat

15998976 bytes total (2406400 bytes free)
Petes-Stack#

[/box]

3. Well there’s only one IOS file in there but let’s make sure anyway, by seeing what version is loaded.

[box]

Petes-Stack#show version
----output ommitted for the sake of brevity----
Switch Ports Model              SW Version            SW Image
------ ----- -----              ----------            ----------
*    1 52    WS-C3750-48P       12.2(55)SE8           C3750-IPSERVICESK9-M
     2 52    WS-C3750-48P       12.2(55)SE8           C3750-IPSERVICESK9-M
----output ommitted for the sake of brevity----

[/box]

4. Lets delete the IOS file from flash1, and make sure it’s gone.

[box]

Petes-Stack#delete /f /r flash1:c3750-ipservicesk9-mz.122-55.SE8
Petes-Stack#dir flash1:
Directory of flash:/

    2  -rwx        5514  Sep 25 2014 14:28:06 +01:00  private-config.text
   84  -rwx         856  Sep 23 2014 13:24:52 +01:00  vlan.dat
   85  -rwx       15354  Sep 25 2014 14:28:06 +01:00  config.text
   87  -rwx        3096  Sep 25 2014 14:28:06 +01:00  multiple-fs

15998976 bytes total (15972352 bytes free)
Petes-Stack#

[/box]

5. Now I’ve setup my TFTP server and downloaded the new IOS file. I need to copy it into the flash1 memory.

[box]

Petes-Stack#copy tftp flash1:
Address or name of remote host? 192.168.1.38
Source filename? c3750-ipservicesk9-mz.122-55.SE9.bin
Destination filename? c3750-ipservicesk9-mz.122-55.SE9.bin
Accessing tftp://192.168.1.38/c3750-ipservicesk9-mz.122-55.SE9.bin...
Loading c3750-ipservicesk9-mz.122-55.SE9.bin from 192.168.1.38 (via Vlan1): !!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[OK - 13009047 bytes]

13009047 bytes copied in 214.044 secs (60777 bytes/sec)
Petes-Stack#

[/box]

6. Repeat the process of deleting the IOS, and copying the new one onto flash2. It will remember your answers from earlier so just hit enter.

[box]

Petes-Stack#delete /f /r flash2:c3750-ipservicesk9-mz.122-55.SE8
Petes-Stack#show flash2:

Directory of flash2:/

    2  -rwx        5514  Sep 25 2014 14:28:06 +01:00  private-config.text
   84  -rwx       15354  Sep 25 2014 14:28:06 +01:00  config.text
   85  -rwx        3096   Mar 1 1993 00:04:19 +00:00  multiple-fs
   86  -rwx         856  Sep 23 2014 13:24:52 +01:00  vlan.dat

15998976 bytes total (15972352 bytes free)
Petes-Stack#copy tftp flash2:
Address or name of remote host [192.168.1.38]? {Enter}
Source filename [c3750-ipservicesk9-mz.122-55.SE9.bin]? {Enter}
Destination filename [c3750-ipservicesk9-mz.122-55.SE9.bin]? {Enter}
Accessing tftp://192.168.1.38/c3750-ipservicesk9-mz.122-55.SE9.bin...
Loading c3750-ipservicesk9-mz.122-55.SE9.bin from 192.168.1.38 (via Vlan1): !!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[OK - 13009047 bytes]

13009047 bytes copied in 245.945 secs (52894 bytes/sec)
Petes-Stack#

[/box]

7. Now let’s make sure the new file is in both switches flash memory.

[box]

Petes-Stack#show flash1:

Directory of flash:/

    2  -rwx        5514  Sep 25 2014 14:28:06 +01:00  private-config.text
    3  -rwx    13009047  Sep 26 2014 15:46:10 +01:00  c3750-ipservicesk9-mz.122-55.SE9.bin
   84  -rwx         856  Sep 23 2014 13:24:52 +01:00  vlan.dat
   85  -rwx       15354  Sep 25 2014 14:28:06 +01:00  config.text
   87  -rwx        3096  Sep 25 2014 14:28:06 +01:00  multiple-fs

15998976 bytes total (2962944 bytes free)
Petes-Stack#show flash2:

Directory of flash2:/

    2  -rwx        5514  Sep 25 2014 14:28:06 +01:00  private-config.text
    3  -rwx    13009047  Sep 26 2014 15:52:03 +01:00  c3750-ipservicesk9-mz.122-55.SE9.bin
   84  -rwx       15354  Sep 25 2014 14:28:06 +01:00  config.text
   85  -rwx        3096   Mar 1 1993 00:04:19 +00:00  multiple-fs
   86  -rwx         856  Sep 23 2014 13:24:52 +01:00  vlan.dat

15998976 bytes total (2962944 bytes free)
Petes-Stack#

[/box]

8. Even though it’s been deleted, the boot variable will be set to the OLD version of the IOS, to demonstrate issue the following command.

[box]

Petes-Stack#show boot
BOOT path-list      : flash:/c3750-ipservicesk9-mz.122-55.SE8
Config file         : flash:/config.text
Private Config file : flash:/private-config.text
Enable Break        : no
Manual Boot         : no
HELPER path-list    :
Auto upgrade        : yes
Auto upgrade path   :
NVRAM/Config file
      buffer size:   524288
Timeout for Config
          Download:    0 seconds
Config Download
       via DHCP:       disabled (next boot: disabled)
-------------------
Switch 2
-------------------
BOOT path-list      : flash:/c3750-ipservicesk9-mz.122-55.SE8
Config file         : flash:/config.text
Private Config file : flash:/private-config.text
Enable Break        : no
Manual Boot         : no
HELPER path-list    :
Auto upgrade        : no
Auto upgrade path   :
Petes-Stack#

[/box]

9. So change the boot variable to the new one, and check again.

[box]

Petes-Stack# configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
Petes-Stack(config)# boot system switch all flash:c3750-ipservicesk9-mz.122-55.SE9.bin

Petes-Stack#show boot
BOOT path-list      : flash:c3750-ipservicesk9-mz.122-55.SE9.bin
Config file         : flash:/config.text
Private Config file : flash:/private-config.text
Enable Break        : no
Manual Boot         : no
HELPER path-list    :
Auto upgrade        : yes
Auto upgrade path   :
NVRAM/Config file
      buffer size:   524288
Timeout for Config
          Download:    0 seconds
Config Download
       via DHCP:       disabled (next boot: disabled)
-------------------
Switch 2
-------------------
BOOT path-list      : flash:c3750-ipservicesk9-mz.122-55.SE9.bin
Config file         : flash:/config.text
Private Config file : flash:/private-config.text
Enable Break        : no
Manual Boot         : no
HELPER path-list    :

Auto upgrade        : no
Auto upgrade path   :
Petes-Stack#

[/box]

10. Save the changes, and reload the switch.

[box]

Petes-Stack#write mem
Building configuration...
[OK]
Petes-Stack#reload
Proceed with reload? [confirm] {Enter}
Switch 2 reloading...

[/box]

11. Post reboot, log in and check that the stack is running the new code.

[box]

Petes-Stack#show version
Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(55)SE9, RELEASE SOFTWARE
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Mon 03-Mar-14 22:45 by prod_rel_team
Image text-base: 0x01000000, data-base: 0x02F00000

ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)

Petes-Stack uptime is 5 minutes
System returned to ROM by power-on
System image file is "flash:c3750-ipservicesk9-mz.122-55.SE9.bin"

[/box]

Option 2

You can also carry out the following procedure on the switch ‘stack master’ that will automate the entire procedure for you. Note: This requires the IOS in .tar format not .bin (as above);

[box]archive download-sw /safe /allow-feature-upgrade /reload tftp:{ip-of-TFTP-Server}/{IOS-File-Name.tar}[/box]

Related Articles, References, Credits, or External Links

NA