Stacking (VSF) Aruba Switches

KB ID 0001492

I noticed some shiny Aruba switches on the bench today, they were for a job my colleague is working on. (Note: Each switch in a stack should be the same model, so these will need two stacks!)

I work on the occasional HP/Aruba core switch, but it’s been a while since I did any work on distribution switches like these. The first thing I learned, was there’s no dedicated stacking cable for them. They simply use a 10Gb (Twinax / DAC) cable. Which I suppose is pretty straight forward, but it means you lose an SFP+ port (which is a bit pants).*

*Note: You can stack with 1GB cables, but you can’t mix and match!

So I said “Give me a shoult when you stack them and I’ll take a nosey!”

Solution

In the ‘land of Aruba’ this is called creating a VSF (Virtual Switching Fabric). As you can see from the photo, these are 2930F Switches, and you can stack up to four switches in a VSF. The same stacking method is used on the 5400R (v3) and 5412, where you can link two 5400R or 5412’s).

Also this method is NOT to be confused with ‘Fabric Stacking’ which is available on the 2920,2930M,3800,3810M models, (that is more like Cisco FlexStack, with a dedicated 100Gb stack cable).

So, assuming you have your switch new and fresh, connect in with your console cable, and dedicate a port to use for VSF.

[box]

Aruba-2930F-24G-PoEP-4SFPP# conf t
Aruba-2930F-24G-PoEP-4SFPP(config)# vsf member 1 link 1 ethernet 25
All configuration on this port has been removed and port is placed in VSF mode.

[/box]

Then place the switch into a VSF domain

[box]

Aruba-2930F-24G-PoEP-4SFPP(config)# vsf enable domain 1
This will save the current configuration and reboot the switch.

[/box]

The switch will ask for a reboot, let it do so.

Repeat the procedure on the second switch, (but this will be member 2).

[box]

Aruba-2930F-24G-PoEP-4SFPP# conf t
Aruba-2930F-24G-PoEP-4SFPP(config)# vsf member 1 link 1 ethernet 25
All configuration on this port has been removed and port is placed in VSF mode.
Aruba-2930F-24G-PoEP-4SFPP(config)# vsf enable domain 1
This will save the current configuration and reboot the switch.

[/box]

Once again let the switch reboot. 

Post reboot you will see the ports are ‘re-numbered’ 1/{port-number} on vsf member 1, 2/{port-number} on vsf member 2 etc.

[box]

Aruba-2930F-24G-PoEP-4SFPP# show interfaces
Status and Counters - Port Counters

                                                                 Flow Bcast
  Port         Total Bytes    Total Frames   Errors Rx Drops Tx  Ctrl Limit
  ------------ -------------- -------------- --------- --------- ---- -----
  1/1          0              0              0         0         off  0    
  1/2          0              0              0         0         off  0    
  1/3          0              0              0         0         off  0    
  1/4          0              0              0         0         off  0    
<---------------Output Removed For The Sake Of Brevity-------------->   
  1/10         0              0              0         0         off  0    
  1/11         0              0              0         0         off  0    
  1/12         0              0              0         0         off  0    
  1/13         0              0              0         0         off  0  
<---------------Output Removed For The Sake Of Brevity--------------> 
  1/19         0              0              0         0         off  0    
  1/20         0              0              0         0         off  0    
  1/21         0              0              0         0         off  0       
  1/25         1,496,823,949  23,354,845     0         0         off  0
<---------------Output Removed For The Sake Of Brevity--------------> 
  2/1          0              0              0         0         off  0    
  2/2          0              0              0         0         off  0    
  2/3          0              0              0         0         off  0    
  2/4          0              0              0         0         off  0    
<---------------Output Removed For The Sake Of Brevity--------------> 
  2/22         0              0              0         0         off  0    
  2/23         0              0              0         0         off  0    
  2/24         0              0              0         0         off  0    
  2/25         1,536,016,322  23,966,915     0         0         off  0    
  2/26         0              0              0         0         off  0    
  2/27         0              0              0         0         off  0    
  2/28         0              0              0         0         off  0    
 

[/box]

If you need to Stack 3 or 4 Switches then you need to add a second link, and create a ring;

i.e.

  • Switch 2 (2nd link now to switch 3) vsf member 2 link 2 ethernet 26
  • Switch 3 (1st link to switch 2 ) vsf member 2 link 1 ethernet 25
  • Switch 3 (2nd link to switch 4 ) vsf member 2 link 2 ethernet 26
  • Switch 4 (1st link to switch 3 ) vsf member 4 link 1 ethernet 25
  • Switch 4 (2nd link to switch 1 ) vsf member 4 link 2 ethernet 26

Useful Aruba VSF Commands

show vsf or show vsf detail :  Shows the list of provisioned chassis members.

show vsf link or show vsf link detail : Shows the state of vsf links for all members.

show vsf lldp-mad status : Shows LLDP MAD (Multi-Active Detection).

show vsftrunk-designated-forwarder : Shows designated forwarders for each trunk.

Related Articles, References, Credits, or External Links

NA

Cisco Error ‘%PHY-4-SFP_NOT_SUPPORTED’

KB ID 0001347 

Problem

This is another question I see getting asked a lot in forums!

You see something like the following;

[box]

000032: *Sep 28 09:35:32.507 UTC: %PHY-4-SFP_NOT_SUPPORTED: The SFP in Gi3/0/50 is not supported (PNL-3750-Stack)
000033: *Sep 28 09:35:32.507 UTC: %PM-4-ERR_DISABLE: gbic-invalid error detected  on Gi3/0/50, putting Gi3/0/50 in err-disable state (PNL-3750-Stack)

[/box]

The usual response is ‘Enable unsupported SFP’s’, and while that sometimes is the answer, it’s not always the answer!

 

Solution

1. Firstly Check the Modules and the Switches, Are you tying to plug a 10GB SFP+ into a slot that only supports SFP, (that includes plugging a twinax cable into an old switch!) In your ‘show run’ you should see TenGigabitEthernet (if your using SFP+ modules). Some switches with network modules list the same interface twice (once as 10GB interfaces and once at 1GB modules, I’ve blogged about that before see THIS ARTICLE, and to confuse things even further, the four interface versions, are grouped as two pairs with each pair consisting of one SFP slot and one SFP+ slot.)

2. Make sure your cable is NOT a CAB-SFP-50CM, (unless you are connecting a 3560 to ANOTHER 3560).

3. Are you using a 2960-S? If so you may need to update the IOS to use SFP+ (assuming your model supports SFP+ not all 2960-S models do).

4. Are you plugging into a Nexus switch with a 1GB connection? If so check the other end for the following error;

Description: Gi1/1/15: This port has been disabled because Non Compliant Gigabit Interface Converter (GBIC) connector detected.

If so, you may need to Manually set the speed on the 5K to 1000, (it wont auto-sense).

5. Is it a non-cisco branded SFP? If so it may still work, (but you will get no joy if you log a TAC call) with the following commands;

[box]

Petes-SW(config)#service internal
Petes-SW(config)#no errdisable detect cause gbic-invalid
Petes-SW(config)#service unsupported-transceiver

[/box]

If you are still in doubt check the Compatibility Matrix.

Related Articles, References, Credits, or External Links

NA

Cisco IOS – Configuring Switch to Switch MACSEC

KB ID 0001000 

Problem

My colleague had to set this up on the test bench today, and it looked infinitely more interesting that what I was doing, so I grabbed my console cable, and offered to ‘help’.

This was done on two Cisco Catalyst 3560-X switches, each with a 10G Service Module (C3KX-SM-10G), and 1Gb SFP modules (Note: Not 10Gb ones, this will become important later).

Solution

1. First hurdle was, when we tried to add the first command to the interface ‘cts man’ it would not accept the command, you need to make sure you are running either IP Base, or the IP Services feature set.

Note: We are running the universal IOS image this allows us to do the following;

[box]

Switch(config)#license boot level ipbase
PLEASE READ THE FOLLOWING TERMS CAREFULLY. INSTALLING THE LICENSE OR

LICENSE KEY PROVIDED FOR ANY CISCO PRODUCT FEATURE OR USING SUCH
PRODUCT FEATURE CONSTITUTES YOUR FULL ACCEPTANCE OF THE FOLLOWING
TERMS. YOU MUST NOT PROCEED FURTHER IF YOU ARE NOT WILLING TO BE BOUND
BY ALL THE TERMS SET FORTH HEREIN.

You hereby acknowledge and agree that the product feature license
is terminable and that the product feature enabled by such license
may be shut down or terminated by Cisco after expiration of the
applicable term of the license (e.g., 30-day trial period). Cisco
reserves the right to terminate or shut down any such product feature
electronically or by any other means available. While alerts or such
messages may be provided, it is your sole responsibility to monitor
your terminable usage of any product feature enabled by the license
and to ensure that your systems and networks are prepared for the shut
down of the product feature. You acknowledge and agree that Cisco will
not have any liability whatsoever for any damages, including, but not
limited to, direct, indirect, special, or consequential damages related
to any product feature being shutdown or terminated. By clicking the
"accept" button or typing "yes" you are indicating you have read and
agree to be bound by all the terms provided herein.

ACCEPT? (yes/[no]): yes
Switch(config)#
Mar 30 01:43:18.513: %IOS_LICENSE_IMAGE_APPLICATION-6-LICENSE_LEVEL: Module name
= c3560x Next reboot level = ipbase and License = ipbase

[/box]

Then reload the switch.

2. Then this jumped up and bit us;

[box]

Mar 30 01:32:07.400: %CTS-6-PORT_UNAUTHORIZED: Port unauthorized for int(Te1/1)
Mar 30 01:32:19.379: %PLATFORM_SM10G-3-SW_VERSION_MISMATCH: The FRULink 10G Service Module (C3KX-SM-10G) in switch 1 has a software version that is incompatible
with the IOS software version. Please update the software. Module is in pass-thru mode.

[/box]

3. If you issue the following command, you can see the difference (highlighted).

[box]

Switch#show switch service-modules
Switch/Stack supports service module CPU version: 03.00.76
                          Temperature                     CPU
Switch#  H/W Status       (CPU/FPGA)      CPU Link      Version
-----------------------------------------------------------------
 1       OK               41C/47C         ver-mismatch  03.00.41


Switch#

[/box]

4. So a quick download from Cisco later, with the file on a FAT32 formatted USB drive.

[box]

Switch#archive download-sw usbflash0:/c3kx-sm10g-tar.150-2.SE6.tar
examining image...
extracting info (100 bytes)
extracting c3kx-sm10g-mz.150-2.SE6/info (499 bytes)
extracting info (100 bytes)

System Type: 0x00010002
Ios Image File Size: 0x017BDA00
Total Image File Size: 0x017BDA00
Minimum Dram required: 0x08000000
Image Suffix: sm10g-150-2.SE6
Image Directory: c3kx-sm10g-mz.150-2.SE6
Image Name: c3kx-sm10g-mz.150-2.SE6.bin
Image Feature: IP|LAYER_3|MIN_DRAM_MEG=128
FRU Module Version: 03.00.76

Updating FRU Module on switch 1...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!
!!!!!!!!!!!!!!!!!!!

Updating FRU FPGA image...

FPGA image update complete.
All software images installed.
Switch#

[/box]

Configuring One Switch Uplink for MACSEC

1. Notice I’m configuring GigabitEthernet 1/2 NOT TenGigabitEthernet 1/1, this is because I’m using 1Gb SFP’s, both interfaces are listed in the config! (This confused us for about twenty minutes). We are not using dot1x authentication, we are simply using a shared secret password (abc123). Note: This has to be a hexedecimal password i.e numbers 0-9 and letters a-f.

[box]

Switch(config)#interface GigabitEthernet 1/2
Switch(config-if)#cts man
% Enabling macsec on Gi1/2 (may take a few seconds)...

Switch(config-if-cts-manual)#no propagate sgt
Switch(config-if-cts-manual)#sap pmk abc123 mode-list gcm-encrypt
Switch(config-if-cts-manual)#no shut
Switch(config-if)#
Mar 30 01:59:03.800: %CTS-6-PORT_UNAUTHORIZED: Port unauthorized for int(Gi1/2)
Mar 30 01:59:04.799: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthe
rnet1/2, changed state to down
Mar 30 01:59:05.805: %LINK-3-UPDOWN: Interface GigabitEthernet1/2, changed state
to down
Mar 30 01:59:08.339: %LINK-3-UPDOWN: Interface GigabitEthernet1/2, changed state
to up
Mar 30 01:59:09.329: %CTS-6-PORT_UNAUTHORIZED: Port unauthorized for int(Gi1/2)
Mar 30 01:59:10.016: %CTS-6-PORT_AUTHORIZED_SUCCESS: Port authorized for int(Gi1/2)
Mar 30 01:59:11.023: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthe
rnet1/2, changed state to up

[/box]

Configuring A Port-Channel Switch Uplink for MACSEC

1. Configure MACSEC on both physical interfaces, before you ‘port-channel’ them. The second interface (when using 1GB SFP’s), is GigabitEthernet 1/4.

[box]

!
interface Port-channel1
switchport trunk encapsulation dot1q
switchport mode trunk
!
!
interface GigabitEthernet1/2
switchport trunk encapsulation dot1q
switchport mode trunk
cts manual
no propagate sgt
sap pmk abc123 mode-list gcm-encrypt
channel-group 1 mode on
!

interface GigabitEthernet1/4
switchport trunk encapsulation dot1q
switchport mode trunk
cts manual
no propagate sgt
sap pmk abc123 mode-list gcm-encrypt
channel-group 1 mode on
!

[/box]

 

Related Articles, References, Credits, or External Links

Thanks to Steve Housego (www.linevty.com) for doing 97% of the hard work, whilst being slowed down by my ‘help’.

Cisco Catalyst – ‘Daughtercard inserted in this switch may not have been manufactured by Cisco’

KB ID 0001018 

Problem

In a newly deployed switch, the MACSEC link refused to establish, when I consoled in I was greeted with this;

[box]

Dec 06 01:30:07.023: %ILET-1-DEVICE_AUTHENTICATION_FAIL: The FRULink SM Daughtercard inserted in this switch may not have been manufactured by Cisco or with Cisco's authorization. If your use of this product is the cause of a support issue, Cisco may deny operation of the product, support under your warranty or under a Cisco technical support program such as Smartnet. Please contact Cisco's Technical Assistance Center for more information.
Dec 06 01:30:07.023: %PLATFORM_SM10G-3-AUTHENTICATION: The FRULink 10G Service Module (C3KX-SM-10G) may not have been entirely manufactured by Cisco. Module is in pass-through mode.
Dec 06 01:30:07.090: %FRNTEND_CTRLR-2-SUB_INACTIVE: The front end controller 1 is inactive
-Traceback= 4E12E0z 250AA2Cz 26A1BE8z 269C198z
Dec 06 01:30:18.079: %FRNTEND_CTRLR-2-SUB_INACTIVE: The front end controller 0 is inactive
-Traceback= 4E12E0z 250AA2Cz 26A1BE8z 269C198z
[/box]

Solution

At first I assumed the C3KX-SM-10G was faulty, so I put it in another 3560-X switch and the problem moved, problem solved (or so I thought).

However I moved the service module in it’s entirety (SFP’s as well). After some more troubleshooting it turns out the service module was fine, the entire problem was caused by a faulty SFP (GLC-SX-MMD).

Related Articles, References, Credits, or External Links

Cisco IOS – Configuring Switch to Switch MACSEC

 

Cisco ASA 5585-X Port Numbering

KB ID 0001004 

Problem

Back at the beginning of the year I had to do a firewall design that included an ASA5585-X, I did some searching to find out how the ports were numbered but came up blank. So I took an (incorrect) educated guess.

I unboxed and fired one up today, and ran though the port numbering and orientation, and discovered the correct numbering.

Solution

Note: This ASA5585-X also has a CX module fitted. The bottom ‘blade’ is the ASA firewall, and the one at the TOP is the CX module. With the CX module fitted, we have an extra eight gigabit Ethernet ports, and two more ten gigabit Ethernet ports.

Port Numbering

Click for larger image

Related Articles, References, Credits, or External Links

NA