Tag Archives: 10Gb

VMware ‘Disable DelayedAck’ Does Not Work?

Posted on by
Reply

KB ID 0001525

Problem

I’ve got a client that’s been having some performance issues with their VMs. Their storage vendor, (EMC) said that as a result of finding this in the logs;

[box]

B       02/28/19 09:50:53.953 scsitarg          117000e [INFO] System: iSCSI Logout Initiator Data: IP=192.168.200.161 Name=...-ec-21 Target Data: Port=2 Flags=0x00002002 Info=0x01200801
B       02/28/19 09:50:53.969 scsitarg          117000e [INFO] System: iSCSI Logout Initiator Data: IP=192.168.201.161 Name=...-ec-21 Target Data: Port=3 Flags=0x00002002 Info=0x01200801
B       02/28/19 09:51:16.413 Health              608fe [WARN] User: Host ESXi-01.petenetlive.com does not have any initiators logged into the storage system.
A       02/28/19 10:04:25.968 scsitarg          117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.200.161 Name=...-ec-21 Target Data: Port=2 Flags=0x00002002 Info=0x00000000 [Target]
B       02/28/19 10:04:26.034 scsitarg          117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.200.161 Name=...-ec-21 Target Data: Port=2 Flags=0x00002002 Info=0x00000000
A       02/28/19 10:04:31.996 scsitarg          117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.201.161 Name=...-ec-21 Target Data: Port=3 Flags=0x00002002 Info=0x00000000 [Target]
B       02/28/19 10:04:32.055 scsitarg          117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.201.161 Name=...-ec-21 Target Data: Port=3 Flags=0x00002002 Info=0x00000000
B       02/28/19 10:04:57.438 Health              608fc [INFO] User: Host ESXi-01.petenetlive.com is operating normally.
Host Host ESXi-01.petenetlive.com is accessing lun Datastore_3 as HLU 3, After the initiators for this host start logging in/logging,  unit attention update events will be logged as the paths to the luns have changed this is expected
2019/02/28-09:50:41.607527 ~~~~     7F3C92369703      std:TCD:   Unit Attention update from 0000001A to 0001030D for LUN 0x3.
2019/02/28-10:02:55.860669 ~~~~     7FE476E61702      std:TCD:   Unit Attention update from 00010149 to 00010157 for LUN 0x3.

[/box]

We should disable DelayedAck and they kindly gave me the VMware KB that outlined the procedure.

Solution

The procedure outlined (for VMware 6.x) is to put the host in maintenance mode, then edit the properties of the iSCSI controller(s), untick the DelayedAck options, reboot the Host, and everything will be peachy. However, even though (post reboot) everything looks good in the the vSphere Web console. If you look on the host you may find something like this;

[box]

vmkiscsid --dump-db | grep Delayed

[/box]

DelayedAck = ‘1’ means ENABLED, DelayedAck = ‘0’ means DISABLED

So half my iSCSI entries in the iSCSI database still have DelayedAck ENABLED?

Some Internet searching told me this was quite common, and that the best way to ‘fix‘ it was to, disable the iSCSI initiator, remove the iSCSI database, reboot and then setup iSCSI again;

[box]

cd /etc/vmware/vmkiscsid
esxcfg-swiscsi -d
rm -f vmkiscsid.db
reboot

[/box]

Which is fine IF YOU ARE USING A SOFTWARE iSCSI INITIATOR, I however was not, I had 2x dedicated hardware iSCSI HBAs on each host!

After many hours of messing about and trial and error, it became clear, I had to do things in a certain order, or DelayedAck would simply just be enabled whether I liked it or not. 🙁

Disable DelayedAck With Hardware iSCSI NICs / HBAs

MAKE SURE THE HOST IS IN MAINTENANCE MODE FIRST

Then take a note of your iSCSI setup, Port Groups, VMKernel Ports, and Physical NICs, you are going to delete the iSCSI database in a minute, and you will need to ‘rebind’ the VMKernel Ports and add the iSCSI targets back in again.

Manually remove your iSCSI target(s) for ALL the iSCSI NIC/HBA’s

Below if you re-run the command, vmkiscsid –dump-db | grep Delayed you will see there’s still some entries in the database with DelayedAck enabled! So unlike above (see example for software iSCSI) we are going to remove the iSCSI database, only here we don’t need to disable the software iSCSI initiator (because we are not using one!) Finally reboot the host.

[box]

cd /etc/vmware/vmkiscsid
rm -f vmkiscsid.db
reboot

[/box]

When the host is back online ADD in the Network Port Binding for the appropriate VMkernel adaptor.

Like so;

DON’T RESCAN THE CONTROLLER AS PROMPTED TO DO SO!

On the Advanced Settings of EACH hardware iSCSI NIC/HBA > Edit > UNTICK ‘DelayedAck’.

Double check they are both still unticked (I’ve seen them re-tick themselves for no discernible reason!) Then rescan the controller(s).

Target > Add.

Re-add the iSCSI target back in, (that you took note of above).

Select the Target > Advanced > Untick the DelayedAck option (Note: This time it’s not inherited). Repeat for any additional iSCSI targets.

When they are all added, rescan the storage controllers again.

Finally recheck all the database entries are set to DISABLED.

[box]

vmkiscsid --dump-db | grep Delayed

[/box]

Related Articles, References, Credits, or External Links

Thanks to Russell and Iain for their patience while I worked all that out!

HPe Synergy 12000 MPIO QSFP to 10GB SFP Setup

Posted on by
Reply

KB ID 0001495

Problem

I was recently involved in deploying an HPe Synergy 12000 Frame. And the network connections from it were ‘a little unusual’ so I thought I’d document that here, to save anyone else the problems I had.

I was connecting to an HP/Aruba 5412 switch so my cables were all HP/Aruba (to be on the safe side).

What you can see (above) is the MPIO Cable (K2Q46A P/N 800867-001) fixed onto the left (and above boxed,) there is a QSFP (P/N 817040-B21.) Note: this can be used either as 4 x 10Gbe or 4 x 8GbFC). On the right you can see the cable ends in 4x Standard LC fibre connectors, so you will also need 4x 10GB SR SFP+ Modules (Aruba P/N J1950D) – shown bottom right.

So what does it do? (Apart from cost a fortune!) Well the QSFP connects at 40Gb and splits the traffic down into 4 x 10Gb 

Cabling and Configuring MPIO QSFP

Connecting up is pretty straight forward, REMEMBER when you connect the 40GB QSFP to the Synergy it will light purple if its connected, and flash purple when it sees activity.

Connecting to the switch is also easy enough, (WARNING: All the ports need to be Trunked (HP) or Ether Channelled (Cisco,)) with LACP enabled. You don’t need to worry about configuring LACP on the Synergy, that’s handled automatically by the ‘Uplink set’.

So the back of the ‘Frame’ has two interconnect links (If you are from a switch background think of these like stacking cables). And two MPIO uplink cables.

HPe/Aruba Switch Config For MPIO

As previously stated, the switch I’m using is an Aruba 5412, with two 8 Port 1Gb/10Gb modules (J9993-A). Here’s the relevant switch config;

[box]

Firstly give the interfaces a sensible name;
!
interface A2
   name "Trunk Link to Synergy VC1 Port Q1"
   exit
interface A3
   name "Trunk Link to Synergy VC1 Port Q1"
   exit
interface A4
   name "Trunk Link to Synergy VC1 Port Q1"
   exit
interface A5
   name "Trunk Link to Synergy VC1 Port Q1"
   exit
!
interface B2
   name "Trunk Link to Synergy VC2 Port Q1"
   exit
interface B3
   name "Trunk Link to Synergy VC2 Port Q1"
   exit
interface B4
   name "Trunk Link to Synergy VC2 Port Q1"
   exit
interface B5
   name "Trunk Link to Synergy VC2 Port Q1"
   exit
!

Show any 'already configured' Trunk links with a 'show trunk' command
In my case two existed, (Trk1 and Trk2). So I used Trk3;

!
trunk A2-A5,B2-B5 Trk3 LACP
!

Now UNTAG vlan 1 (assuming that's your default VLAN) And TAG and VLANS that 
need to be used in the Synergy Deployment. (Note on an HP switch simply add
the Trk3 to the existing settings like so;

!
vlan1
untagged A6-A8,B6-B8,E1-E24,F1-F24,G3-G12,H3-H12,Trk1-Trk3
!
vlan 100
   tagged Trk1-Trk3
   exit
vlan 101
   tagged Trk1-Trk3
   exit
vlan 102
   tagged Trk1-Trk3
   exit
vlan 103
   tagged Trk1-Trk3
   exit
etc.

[/box]

Cisco Switch Config For MPIO

If you have a Cisco Switch then instead of ‘Trunking’ you will be ‘Ether Channelling’ for a more detailed explanation see the following post

HP and Cisco – VLANs and Trunks Confusion!

[box]

!
interface port-channel 1
switchport trunk encapsulation dot1q
switchport mode trunk
!
interface range Te1/1 - 4 , Te2/1 - 4
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active
!

[/box]

Configuring Synergy 12000 Networking

The process is, you add  Networks, then collect Networks together in Network Sets, Then you create Logical Interconnect  Groups. Part of creating a Logical Interconnect Groups, involves creating an Uplink Set, which consists of both your Networks, and the The Uplink ports.

Note: A Network Set is used by a Server Profile, (or a Server Profile Template).

Create Networks

One View > Networking > Networks > Create Network

Create Network Sets

One View > Networking > Networks Sets  > Create Network Set  > Give it a name > Add Networks > Create.

Create Logical Interconnect Group

One View > Networking > Logical Interconnect Group  > Create Logical Interconnect Group > Give it a name > Select the correct  Interconnect Bay Set (see diagram above) > Select Interconnects > Add Uplink Set.

Give the set a name > Select the Type > Add in the Networks > Add in the Uplinks > Create.

Note: You only need to add in ALL the LOGICAL interfaces i.e. Q1:1, Q1:2,Q1:3,Q1:4 for EACH Interconnect module. .

After a few minutes if you look under One View > Networking > Logical Interconnects > You will see one listed that has the name of your Logical Interconnect group (with a divide symbol on the end!) Make sure ALL the logical uplinks are connected. (If not you will see LACP errors on the switch).

 

Related Articles, References, Credits, or External Links

NA

Stacking (VSF) Aruba Switches

Posted on by
4

KB ID 0001492

I noticed some shiny Aruba switches on the bench today, they were for a job my colleague is working on. (Note: Each switch in a stack should be the same model, so these will need two stacks!)

I work on the occasional HP/Aruba core switch, but it’s been a while since I did any work on distribution switches like these. The first thing I learned, was there’s no dedicated stacking cable for them. They simply use a 10Gb (Twinax / DAC) cable. Which I suppose is pretty straight forward, but it means you lose an SFP+ port (which is a bit pants).*

*Note: You can stack with 1GB cables, but you can’t mix and match!

So I said “Give me a shoult when you stack them and I’ll take a nosey!”

Solution

In the ‘land of Aruba’ this is called creating a VSF (Virtual Switching Fabric). As you can see from the photo, these are 2930F Switches, and you can stack up to four switches in a VSF. The same stacking method is used on the 5400R (v3) and 5412, where you can link two 5400R or 5412’s).

Also this method is NOT to be confused with ‘Fabric Stacking’ which is available on the 2920,2930M,3800,3810M models, (that is more like Cisco FlexStack, with a dedicated 100Gb stack cable).

So, assuming you have your switch new and fresh, connect in with your console cable, and dedicate a port to use for VSF.

[box]

Aruba-2930F-24G-PoEP-4SFPP# conf t
Aruba-2930F-24G-PoEP-4SFPP(config)# vsf member 1 link 1 ethernet 25
All configuration on this port has been removed and port is placed in VSF mode.

[/box]

Then place the switch into a VSF domain

[box]

Aruba-2930F-24G-PoEP-4SFPP(config)# vsf enable domain 1
This will save the current configuration and reboot the switch.

[/box]

The switch will ask for a reboot, let it do so.

Repeat the procedure on the second switch, (but this will be member 2).

[box]

Aruba-2930F-24G-PoEP-4SFPP# conf t
Aruba-2930F-24G-PoEP-4SFPP(config)# vsf member 1 link 1 ethernet 25
All configuration on this port has been removed and port is placed in VSF mode.
Aruba-2930F-24G-PoEP-4SFPP(config)# vsf enable domain 1
This will save the current configuration and reboot the switch.

[/box]

Once again let the switch reboot. 

Post reboot you will see the ports are ‘re-numbered’ 1/{port-number} on vsf member 1, 2/{port-number} on vsf member 2 etc.

[box]

Aruba-2930F-24G-PoEP-4SFPP# show interfaces
Status and Counters - Port Counters

                                                                 Flow Bcast
  Port         Total Bytes    Total Frames   Errors Rx Drops Tx  Ctrl Limit
  ------------ -------------- -------------- --------- --------- ---- -----
  1/1          0              0              0         0         off  0    
  1/2          0              0              0         0         off  0    
  1/3          0              0              0         0         off  0    
  1/4          0              0              0         0         off  0    
<---------------Output Removed For The Sake Of Brevity-------------->   
  1/10         0              0              0         0         off  0    
  1/11         0              0              0         0         off  0    
  1/12         0              0              0         0         off  0    
  1/13         0              0              0         0         off  0  
<---------------Output Removed For The Sake Of Brevity--------------> 
  1/19         0              0              0         0         off  0    
  1/20         0              0              0         0         off  0    
  1/21         0              0              0         0         off  0       
  1/25         1,496,823,949  23,354,845     0         0         off  0
<---------------Output Removed For The Sake Of Brevity--------------> 
  2/1          0              0              0         0         off  0    
  2/2          0              0              0         0         off  0    
  2/3          0              0              0         0         off  0    
  2/4          0              0              0         0         off  0    
<---------------Output Removed For The Sake Of Brevity--------------> 
  2/22         0              0              0         0         off  0    
  2/23         0              0              0         0         off  0    
  2/24         0              0              0         0         off  0    
  2/25         1,536,016,322  23,966,915     0         0         off  0    
  2/26         0              0              0         0         off  0    
  2/27         0              0              0         0         off  0    
  2/28         0              0              0         0         off  0    
 

[/box]

If you need to Stack 3 or 4 Switches then you need to add a second link, and create a ring;

i.e.

  • Switch 2 (2nd link now to switch 3) vsf member 2 link 2 ethernet 26
  • Switch 3 (1st link to switch 2 ) vsf member 2 link 1 ethernet 25
  • Switch 3 (2nd link to switch 4 ) vsf member 2 link 2 ethernet 26
  • Switch 4 (1st link to switch 3 ) vsf member 4 link 1 ethernet 25
  • Switch 4 (2nd link to switch 1 ) vsf member 4 link 2 ethernet 26

Useful Aruba VSF Commands

show vsf or show vsf detail :  Shows the list of provisioned chassis members.

show vsf link or show vsf link detail : Shows the state of vsf links for all members.

show vsf lldp-mad status : Shows LLDP MAD (Multi-Active Detection).

show vsftrunk-designated-forwarder : Shows designated forwarders for each trunk.

Related Articles, References, Credits, or External Links

NA

Cisco IOS – Configuring Switch to Switch MACSEC

Posted on by

KB ID 0001000 

Problem

My colleague had to set this up on the test bench today, and it looked infinitely more interesting that what I was doing, so I grabbed my console cable, and offered to ‘help’.

This was done on two Cisco Catalyst 3560-X switches, each with a 10G Service Module (C3KX-SM-10G), and 1Gb SFP modules (Note: Not 10Gb ones, this will become important later).

Solution

1. First hurdle was, when we tried to add the first command to the interface ‘cts man’ it would not accept the command, you need to make sure you are running either IP Base, or the IP Services feature set.

Note: We are running the universal IOS image this allows us to do the following;

[box]

Switch(config)#license boot level ipbase
PLEASE READ THE FOLLOWING TERMS CAREFULLY. INSTALLING THE LICENSE OR

LICENSE KEY PROVIDED FOR ANY CISCO PRODUCT FEATURE OR USING SUCH
PRODUCT FEATURE CONSTITUTES YOUR FULL ACCEPTANCE OF THE FOLLOWING
TERMS. YOU MUST NOT PROCEED FURTHER IF YOU ARE NOT WILLING TO BE BOUND
BY ALL THE TERMS SET FORTH HEREIN.

You hereby acknowledge and agree that the product feature license
is terminable and that the product feature enabled by such license
may be shut down or terminated by Cisco after expiration of the
applicable term of the license (e.g., 30-day trial period). Cisco
reserves the right to terminate or shut down any such product feature
electronically or by any other means available. While alerts or such
messages may be provided, it is your sole responsibility to monitor
your terminable usage of any product feature enabled by the license
and to ensure that your systems and networks are prepared for the shut
down of the product feature. You acknowledge and agree that Cisco will
not have any liability whatsoever for any damages, including, but not
limited to, direct, indirect, special, or consequential damages related
to any product feature being shutdown or terminated. By clicking the
"accept" button or typing "yes" you are indicating you have read and
agree to be bound by all the terms provided herein.

ACCEPT? (yes/[no]): yes
Switch(config)#
Mar 30 01:43:18.513: %IOS_LICENSE_IMAGE_APPLICATION-6-LICENSE_LEVEL: Module name
= c3560x Next reboot level = ipbase and License = ipbase

[/box]

Then reload the switch.

2. Then this jumped up and bit us;

[box]

Mar 30 01:32:07.400: %CTS-6-PORT_UNAUTHORIZED: Port unauthorized for int(Te1/1)
Mar 30 01:32:19.379: %PLATFORM_SM10G-3-SW_VERSION_MISMATCH: The FRULink 10G Service Module (C3KX-SM-10G) in switch 1 has a software version that is incompatible
with the IOS software version. Please update the software. Module is in pass-thru mode.

[/box]

3. If you issue the following command, you can see the difference (highlighted).

[box]

Switch#show switch service-modules
Switch/Stack supports service module CPU version: 03.00.76
                          Temperature                     CPU
Switch#  H/W Status       (CPU/FPGA)      CPU Link      Version
-----------------------------------------------------------------
 1       OK               41C/47C         ver-mismatch  03.00.41


Switch#

[/box]

4. So a quick download from Cisco later, with the file on a FAT32 formatted USB drive.

[box]

Switch#archive download-sw usbflash0:/c3kx-sm10g-tar.150-2.SE6.tar
examining image...
extracting info (100 bytes)
extracting c3kx-sm10g-mz.150-2.SE6/info (499 bytes)
extracting info (100 bytes)

System Type: 0x00010002
Ios Image File Size: 0x017BDA00
Total Image File Size: 0x017BDA00
Minimum Dram required: 0x08000000
Image Suffix: sm10g-150-2.SE6
Image Directory: c3kx-sm10g-mz.150-2.SE6
Image Name: c3kx-sm10g-mz.150-2.SE6.bin
Image Feature: IP|LAYER_3|MIN_DRAM_MEG=128
FRU Module Version: 03.00.76

Updating FRU Module on switch 1...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!
!!!!!!!!!!!!!!!!!!!

Updating FRU FPGA image...

FPGA image update complete.
All software images installed.
Switch#

[/box]

Configuring One Switch Uplink for MACSEC

1. Notice I’m configuring GigabitEthernet 1/2 NOT TenGigabitEthernet 1/1, this is because I’m using 1Gb SFP’s, both interfaces are listed in the config! (This confused us for about twenty minutes). We are not using dot1x authentication, we are simply using a shared secret password (abc123). Note: This has to be a hexedecimal password i.e numbers 0-9 and letters a-f.

[box]

Switch(config)#interface GigabitEthernet 1/2
Switch(config-if)#cts man
% Enabling macsec on Gi1/2 (may take a few seconds)...

Switch(config-if-cts-manual)#no propagate sgt
Switch(config-if-cts-manual)#sap pmk abc123 mode-list gcm-encrypt
Switch(config-if-cts-manual)#no shut
Switch(config-if)#
Mar 30 01:59:03.800: %CTS-6-PORT_UNAUTHORIZED: Port unauthorized for int(Gi1/2)
Mar 30 01:59:04.799: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthe
rnet1/2, changed state to down
Mar 30 01:59:05.805: %LINK-3-UPDOWN: Interface GigabitEthernet1/2, changed state
to down
Mar 30 01:59:08.339: %LINK-3-UPDOWN: Interface GigabitEthernet1/2, changed state
to up
Mar 30 01:59:09.329: %CTS-6-PORT_UNAUTHORIZED: Port unauthorized for int(Gi1/2)
Mar 30 01:59:10.016: %CTS-6-PORT_AUTHORIZED_SUCCESS: Port authorized for int(Gi1/2)
Mar 30 01:59:11.023: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthe
rnet1/2, changed state to up

[/box]

Configuring A Port-Channel Switch Uplink for MACSEC

1. Configure MACSEC on both physical interfaces, before you ‘port-channel’ them. The second interface (when using 1GB SFP’s), is GigabitEthernet 1/4.

[box]

!
interface Port-channel1
switchport trunk encapsulation dot1q
switchport mode trunk
!
!
interface GigabitEthernet1/2
switchport trunk encapsulation dot1q
switchport mode trunk
cts manual
no propagate sgt
sap pmk abc123 mode-list gcm-encrypt
channel-group 1 mode on
!

interface GigabitEthernet1/4
switchport trunk encapsulation dot1q
switchport mode trunk
cts manual
no propagate sgt
sap pmk abc123 mode-list gcm-encrypt
channel-group 1 mode on
!

[/box]

 

Related Articles, References, Credits, or External Links

Thanks to Steve Housego (www.linevty.com) for doing 97% of the hard work, whilst being slowed down by my ‘help’.

Cisco Catalyst – Upgrading IOS (via USB)

Posted on by

KB ID 0001056

Problem

Had a stack of 3560-X Switches to update today, and when I went looking for the notes I used last time, I could not find them. So This time I took the time to document the procedure.

Solution

Now I could load in the IOS image from TFTP like this, but last time I did this I used a spare USB drive and the image ‘tar’ file, and found it a lot less hassle.

1. Make sure you have formatted your dive at FAT32, download you image file to it and put it in the switch.

At console you should see something like this;

[box]Apr 22 13:13:18.466: %USBFLASH-5-CHANGE: usbflash0 has been inserted![/box]

2. Update the switch like so;

[box]

Petes-Switch#archive download-sw usbflash0:/c3560e-universalk9-tar.150-2.SE6.tar
examining image...
extracting info (110 bytes)
extracting c3560e-universalk9-mz.150-2.SE6/info (581 bytes)
extracting info (110 bytes)

System Type:             0x00000002
  Ios Image File Size:   0x0135B200
  Total Image File Size: 0x0187BA00
  Minimum Dram required: 0x08000000
  Image Suffix:          universalk9-150-2.SE6
  Image Directory:       c3560e-universalk9-mz.150-2.SE6
  Image Name:            c3560e-universalk9-mz.150-2.SE6.bin
  Image Feature:         IP|LAYER_3|PLUS|SSH|3DES|MIN_DRAM_MEG=128

Old image for switch 1: flash:/c3560e-universalk9-mz.122-55.SE8
  Old image will be deleted before download.

Deleting `flash:/c3560e-universalk9-mz.122-55.SE8' to create required space

————output removed for the sake of brevity————

extracting c3560e-universalk9-mz.150-2.SE6/dc_default_profiles.txt (66292 bytes)
extracting c3560e-universalk9-mz.150-2.SE6/c3560e-universalk9-mz.150-2.SE6.bin (20288000 bytes)
extracting info (110 bytes)

Installing (renaming): `flash:update/c3560e-universalk9-mz.150-2.SE6' ->
                                       `flash:/c3560e-universalk9-mz.150-2.SE6'
New software image installed in flash:/c3560e-universalk9-mz.150-2.SE6


All software images installed.
Petes-Switch#reload
Proceed with reload? [confirm]

*Mar  1 00:09:14.243: %SYS-5-RELOAD: Reload requested by console. Reload reason: Reload command

[/box]

3. At this point when the switch reloads, it will take a long time to boot as it performs a lot of updates and code rewrites when it restarts.

Upgrading The Catalyst Service Module

These switches have a 10Gb Service module in them that also needs updating, once the switch reboots you will have to wait a few minutes before the service module boots as well, if you don’t wait then you will see this; [box]

Petes-Switch#show switch service-modules
Switch/Stack supports service module CPU version: 03.00.76
                          Temperature                     CPU
Petes-Switch#  H/W Status       (CPU/FPGA)      CPU Link      Version
-----------------------------------------------------------------
 1             OK               48C/43C         notconnected  N/A

[/box]

You may also see an error like this (I’ve blogged this before).

[box]

Mar 30 01:29:55.128: POST: Macsec Uplink Loopback Tests : Passed Decryption Mode
Mar 30 01:29:57.594: POST: Macsec Uplink Loopback Tests : End
Mar 30 01:29:57.594: %PLATFORM-6-FRULINK_INSERTED: FRULink 10G SM module inserted.
Mar 30 01:32:13.188: %PLATFORM_SM10G-3-SW_VERSION_MISMATCH: The FRULink 10G Service Module
(C3KX-SM-10G) in switch 1 has a software version that is incompatible with the IOS software
 version. Please update the software. Module is in pass-thru mode.
Petes-Switch#show switch service-modules
Switch/Stack supports service module CPU version: 03.00.76
                          Temperature                     CPU
Petes-Switch#  H/W Status       (CPU/FPGA)      CPU Link      Version
-----------------------------------------------------------------
 1             OK               54C/54C         ver-mismatch  03.00.41

[/box]

Or it may simply look like this;

[box]

Mar 30 01:32:29.403: %PLATFORM_SM10G-6-LINK_UP: The FRULink 10G Service Module (C3KX-SM-10G)
communication has been established.
Petes-Switch#
Petes-Switch#show switch service-modules
Switch/Stack supports service module CPU version: 03.00.76
                          Temperature                     CPU
Petes-Switch#  H/W Status       (CPU/FPGA)      CPU Link      Version
-----------------------------------------------------------------
 1             OK               50C/48C         connected     03.00.76

[/box]

To perform the upgrade, you will need a matching image for the service module.

[box]

Petes-Switch#archive download-sw usbflash0:/c3kx-sm10g-tar.150-2.SE6.tar
examining image...
extracting info (100 bytes)
extracting c3kx-sm10g-mz.150-2.SE6/info (499 bytes)
extracting info (100 bytes)

System Type:             0x00010002
  Ios Image File Size:   0x017BDA00
  Total Image File Size: 0x017BDA00
  Minimum Dram required: 0x08000000
  Image Suffix:          sm10g-150-2.SE6
  Image Directory:       c3kx-sm10g-mz.150-2.SE6
  Image Name:            c3kx-sm10g-mz.150-2.SE6.bin
  Image Feature:         IP|LAYER_3|MIN_DRAM_MEG=128
  FRU Module Version:    03.00.76


Updating FRU Module on switch 1...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Updating FRU FPGA image...

FPGA image update complete.

All software images installed.
Petes-Switch#reload
Proceed with reload? [confirm]

Mar 30 01:47:19.459: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload command.

[/box]

Related Articles, References, Credits, or External Links

Cisco Catalyst – Upgrading ‘Stacked’ Switches