FortiGate High Availability (Active / Passive)

KB ID 0001730

So my aim was to setup FortiGate High Availability failover in Active / Passive mode. I’m setting this up in EVE-NG and here’s what my lab looks like;

Note: Im using TWO connections for Heartbeat/Failover, you can simply use one if you prefer.

FortiGate High Availability (Pre-Requisites)

Obviously the firewalls need to be the same! For physical firewalls that’s straightforward, but be careful if you are using virtual FortiGates, make sure they are the same hardware and licence versions!

As you can see in my topology (above), each port needs to be connected to the same network on BOTH firewalls.

Make sure both firewalls are running the SAME firmware.

Make sure the interfaces are not getting their IP addresses from DHCP, or PPPoE.

I’m assuming the primary firewall is all setup and configured properly, the secondary firewall then needs to be built, licensed, and internet connected (so it can update etc). Note: It will take it’s config from the primary firewall when configured.

FortiGate High Availability (Active Passive) From Command Line

I know, Fortinet like to say that there’s no need to be working at command line these days, but when you see how easy it is to setup from CLI, you might choose to do it that way instead.

Primary FortiGate High Availability Setup

FortiGate uses priority to set the primary firewall, by default it sets the value to 128. So I’m going to set my Primary firewall to 200 and my Secondary firewall to 100.

Here are the commands you can simply copy and paste;

[box]

config system ha
set group-id 10
set group-name HA-GROUP
set mode a-p
set password Password123
set hbdev port3 0 port4 0
set session-pickup enable
set override enable
set override-wait-time 10
set priority 200
end

[/box]

What does all that mean? The Group ID and Group Name, are shared by all firewalls in the cluster. (Yes cluster, you can scale this up to more firewalls). Mode simply sets the mode to a-p (active passive). Hbdev denotes the interfaces used for the heartbeat network, (the zeros denote that they will be used equally). Session-pickup allows the firewalls to share the session state table for your clients, (i.e. provides seamless TCP session failover). Override is enabled, this will fail back the the primary firewall when it becomes available.  Note: wait-time is enabled and set to 10 seconds to avoid any ‘flap / stutter’ that may cause disconnections when executing the override. Then finally the priority is set to 200.

Secondary FortiGate Hight Availability Setup

Practically the same, but you don’t need to set override, and the priority will be lower (100).

[box]

config system ha
set group-id 10
set group-name HA-GROUP
set mode a-p
set password Password123
set hbdev port3 0 port4 0
set session-pickup enable
set priority 100
end

[/box]

FortiGate HA Monitor and TroubleShooting

At this point go and have a coffee, the config needs replicating from the primary to the secondary, and this can take a few minutes. Use the following command to check;

[box]

get system ha status

[/box]

You want to see them both ‘in-sync‘.

To troubleshoot, use;

[box]

diagnose system ha status

[/box]

FortiGate Failover (Active Passive) From GUI

On the Primary (pre configured) firewall, System > HA  > Change the drop down to Active-Passive.

  • Device Priority: 200
  • Group name: HA-GROUP {or something sensible}.
  • Password: {needs to match on both firewalls}.
  • Sesión pickup: Enabled {replicates client session data}.
  • Monitor Interfaces: {you can leave this blank, unless you only want to monitor certain interfaces}.
  • Heartbeat Interfaces: {enter one or more interfaces}.

What’s Management interface Reservation? : You can use this to have separate management for each firewall, (usually you can’t get to the standby/secondary firewall for GUI/SSH management etc).

Clock OK when complete

You should see something like this;

Jump onto the Secondary firewall, and set it the same, (with the exception of the priority).

Note: Override (Fail Back) cannot currently be set in the GUI.

Wait a while and hit refresh, you should see both firewalls in Sync

Possible Bug in 6.4.4 build 1803 (GA)

When carrying out the above procedure (from the web management console), my firewalls did not sync, while trouble shooting at command line they were both set as primary and appeared not to be able to see the other, they were saying;

{Firewall Serial Number}is selected as the master because it’s the only member in the cluster.

Despite many hours of troubleshooting/rebuilding, this kept happening, it worked fine when configured from command line, but not from the GUI. When I checked the commands after using the GUI I noticed that “set group-id {number}” was missing from the config. If I added this manually it started syncing straight away? 

Related Articles, References, Credits, or External Links

Thanks to the good folk at Fortinet for sending me some licences to test this (much appreciated).

Configuring Cisco HSRP

KB ID 0000946 

Problem

Cisco HSRP: Normally your client machines have one route off the network, (their default gateway). But what if that goes down? HSRP aims to solve this problem by assigning a ‘Virtual IP address’ to your default gateway (or default route). So that IP can be shared amongst two or more possible devices (routers, or layer 3 switches).

Above, we have a client 192.168.1.10 that has two possible routes off the network, (.254 and .253). We will setup a virtual IP of .250 and both routers can use that IP, (if they are the active gateway). Below is a brief overview of how to set it up.

Deploy Cisco HSRP

Setting up Cisco HSRP

1. On the first router (Router0), add the standby IP address (192.168.1.250) the ‘1’ denotes the standby group (a number from 0 to 4096). It comes up as standby, then after it has checked (via multicast address 224.0.0.2 on UDP port 1985). It finds no other live HSRP devices using that IP address so it becomes ‘Active’.

[box]

Router0>enable
Router0#configure terminal 
Enter configuration commands, one per line. End with CNTL/Z.
Router0(config)#interface GigabitEthernet0/0
Router0(config-if)#standby 1 ip 192.168.1.250
Router0(config-if)#

%HSRP-6-STATECHANGE: GigabitEthernet0/0 Grp 1 state Speak -> Standby

%HSRP-6-STATECHANGE: GigabitEthernet0/0 Grp 1 state Standby -> Active

Router0(config-if)#

[/box]

2. Repeat this on the second Router, this one discovers the ‘Active’ router and sets itself up as ‘Standby’.

[box]

Router1>
Router1>enable
Router1#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
Router1(config)#interface GigabitEthernet0/0
Router1(config-if)#standby 1 ip 192.168.1.250
Router1(config-if)#
%HSRP-6-STATECHANGE: GigabitEthernet0/0 Grp 1 state Speak -> Standby

Router1(config-if)#

[/box]

3. You can prove this by running show standby (or do show standby in configure terminal mode).

[box]

Router0

Router0#show standby
GigabitEthernet0/0 - Group 1 (version 2)
State is Active
8 state changes, last state change 00:02:02
Virtual IP address is 192.168.1.250
Active virtual MAC address is 0000.0C9F.F001
Local virtual MAC address is 0000.0C9F.F001 (v2 default)
Hello time 3 sec, hold time 10 sec
Next hello sent in 0.055 secs
Preemption disabled
Active router is local
Standby router is 192.168.1.253
Priority 100 (default 100)
Group name is hsrp-Gig0/0-1 (default)
Router0#

Router1

Router1#show standby
GigabitEthernet0/0 - Group 1 (version 2)
State is Standby
3 state changes, last state change 00:10:44
Virtual IP address is 192.168.1.250
Active virtual MAC address is unknown
Local virtual MAC address is 0000.0C9F.F001 (v2 default)
Hello time 3 sec, hold time 10 sec
Next hello sent in 2.125 secs
Preemption disabled
Active router is 192.168.1.254
Standby router is local
Priority 100 (default 100)
Group name is hsrp-Gig0/0-1 (default)
Router1#

[/box]

4. That is HSRP configured! However there are a few changes you might want to make, for example, what if one router had a 100MB leased line, and the other was a 2MB ADSL line, you would want the fastest one to be in use, (as long as it was up). To achieve that, give the router with the fastest connection a higher priority (you may notice above, that by default the priority is 100). Be aware, even if a router has the highest priority, it wont ‘seize’ the virtual IP, it just sits and waits until it’s available. For our 100MB and 2MB example that’s not good. We would want Router1 to seize the virtual IP as soon as it can. To do that we need to set it to preempt. (Note: This process is called ‘launching a coup’).

[box]

Router1(config)#interface GigabitEthernet0/0
Router1(config-if)#standby 1 priority 105
Router1(config-if)#standby 1 preempt

[/box]

5. At this point it’s important to say, that in our scenario we would also need to setup a virtual IP for the ‘other side’ of the routers (i.e their GigabitEthernet 0/1 interfaces), or the remote client (172.16.1.10) would not be able to return our ‘pings’ or get any traffic back to us. So lets setup a virtual HSRP address on that side as well. Notice I just use another standby group number.

Note: To work the remote host 172.16.1.10 will need its default gateway changing to the HSRP Virtual IP of 172.16.1.250.

[box]

Router0 

Router0(config)#interface GigabitEthernet0/1
Router0(config-if)#standby 2 ip 172.16.1.250
Router0(config-if)#

Router1

Router1(config)#interface GigabitEthernet0/1
Router1(config-if)#standby 2 ip 172.16.1.250
Router1(config-if)#

[/box]

6. Finally we have set Router0 with the highest priority and set it to seize the virtual IP as soon as it can. But what if another interface on Router1 goes down? e.g. If the GigabitEthernet 0/1 interface were to go down, HSRP would not do anything because it’s tracking both the GigabitEthernet 0/0 interfaces, so communications would fail.

To solve the problem we need to tell it which interfaces to ‘Track’. In our example we need to track GigabitEthernet 0/1, if that goes down we need to give the virtual IP address to the standby ‘router’. This works because once we tell it to ‘track’ the GigabitEthernet 0/1 interface, if that were to fail it will DECREMEMT the routers priority by 10. So for Router0 its priority would drop to 95, this is five less than the default value of 100 (on Router1). But Remember, at the moment that fail-over would still fail, unless you allow Router1 to preempt and launch a coup.

[box]

Router0 

Router0(config)#interface GigabitEthernet0/0
Router0(config-if)#standby 1 track GigabitEthernet0/1
Router0(config-if)#

Router1

Router1(config)#interface GigabitEthernet0/0
Router1(config-if)#standby 1 preempt
Router1(config-if)#

[/box]

7. We can see that by running a ‘show standby’ on Router0.

[box]

Router0#show standby
GigabitEthernet0/0 - Group 1 (version 2)
State is Active
7 state changes, last state change 00:00:31
Virtual IP address is 192.168.1.250
Active virtual MAC address is 0000.0C9F.F001
Local virtual MAC address is 0000.0C9F.F001 (v2 default)
Hello time 3 sec, hold time 10 sec
Next hello sent in 2.81 secs
Preemption enabled
Active router is local
Standby router is 192.168.1.253, priority 115 (expires in 7 sec)
Priority 115 (configured 115)
Track interface GigabitEthernet0/1 state Up decrement 10
Group name is hsrp-Gig0/0-1 (default)
GigabitEthernet0/1 - Group 2 (version 2)
State is Active
6 state changes, last state change 00:00:28
Virtual IP address is 172.16.1.250
Active virtual MAC address is 0000.0C9F.F002
Local virtual MAC address is 0000.0C9F.F002 (v2 default)
Hello time 3 sec, hold time 10 sec
Next hello sent in 0.81 secs
Preemption disabled
Active router is local
Standby router is 172.16.1.253
Priority 100 (default 100)
Group name is hsrp-Gig0/1-2 (default)
Router0#

[/box]

Related Articles, References, Credits, or External Links

NA

Cisco Stacking 2960-X Catalyst Switches

KB ID 0001444

Problem

You can stack up to 8 2960-X Switches*, you will require the stack modules and cables, (shown below). 

*Note: If you are studying for an exam, and the question is StackWize the answer is 9.

Solution

Stack Modules: Power down the switch, remove the blanking plate and fit the module, then when powered on you can use a show inventory command to make sure the module has been detected correctly.

[box]

Switch#show inventory
NAME: "1", DESCR: "WS-C2960X-48FPD-L"
PID: WS-C2960X-48FPD-L , VID: V07 , SN: FCW2213B2XX
NAME: "Switch 1 - FlexStackPlus Module", DESCR: "Stacking Module"
PID: C2960X-STACK , VID: V02 , SN: FOC221410XX

Switch#
[/box]

Setting a Stack Master: If you do nothing and simply connect up a stack, then power it on, the switches have an ‘election’ and the one with the lowest MAC address becomes the master. This is a bit bobbins, so I prefer to configure a switch to be master by increasing its priority. This is a number between 1 and 15 (Highest wins) and all switches have a priority of 1 out of the box.

I typically set the ‘top’ switch as the stack master and set its priority to 15.

[box]

Switch#configure terminal
Switch(config)#switch 1 priority 15
Changing the Switch Priority of Switch Number 1 to 15
Do you want to continue?[confirm]{Enter}	
New Priority has been set successfully

TO CONFIRM
Switch#show switch
Switch/Stack Mac Address : 5061.bf51.dd80
                                           H/W   Current
Switch#  Role   Mac Address     Priority Version  State
----------------------------------------------------------
*1       Master 5061.bf51.dd80     15     4       Ready


Switch#
[/box]

Then cable the switches together with the stack cables. If you are cabling two switches together then cable like so;

If cabling more switches , then cable stack port 1 of a switch, to port 2 of the switch below it, and keep going, then on the bottom switch cable port 1 back up to port 2 on the top switch (to make a ring).

Then power everything on, check the stack is up 

[box]

Switch#show switch detail
Switch/Stack Mac Address : 5061.bf51.dd80
                                           H/W   Current
Switch#  Role   Mac Address     Priority Version  State
----------------------------------------------------------
*1       Master 5061.bf51.dd80     15     4       Ready
 2       Member 5061.bf51.de00     1      4       Ready

         Stack Port Status             Neighbors
Switch#  Port 1     Port 2           Port 1   Port 2
--------------------------------------------------------
  1        Ok         Ok                2        2
  2        Ok         Ok                1        1

[/box]

Note: You can use ‘show switch stack-ports‘ to get the second half of that output only.

Check the switch stack ring;

[box]

Switch#show switch stack-ring speed

Stack Ring Speed        : 20G
Stack Ring Configuration: Full
Stack Ring Protocol     : FlexStack

[/box]

My Ring speed is only 20G? Yes thats correct 🙂

Cisco Say it’s supposed to be 80G? Yeah, marketing piffle sorry, with two rings at full duplex it’s rated at four times the bandwidth that’s where they get this figure from.

You can now configure all the interfaces from one management IP, note to select all the interfaces use the following syntax

[box]

Switch(config)#interface range gi1/0/1-48,gi2/0/1-48

[/box]

Related Articles, References, Credits, or External Links

Cisco Stacking 3750 Switches

Sync Microsoft Domain Time To A Cisco NTP Device

KB ID 0001038

Problem

I’ve been posting domain time articles for a long time, and on more than one occasion I’ve really needed to take my Windows time from a Cisco Device and failed miserably. I’ve even used third party NTP software to solve this problem on my own test network.

On a client network, my colleague deployed ACS5 this week, I secured the ASA5585-X for AAA and it failed authentication. Logging revealed a clock skew error, so we manually set the time on the domain PDC. Within half an hour it was failing. The network topology prevented me syncing to a public NTP server from the domain PDC.

We did however have all the network devices syncing from a public time source, if only we could use one of those?

Solution

Step 1 Configure NTP on your Cisco Device.

Here I’m using a 7200 Router in GNS3, the NTP IP addresses I use are UK based NTP servers, I suggest you replace them with some public NTP servers on your own continent. I’m using two for redundancy.

[box]

Petes-Router#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
Petes-Router(config)#ntp server 130.88.202.49 prefer
Petes-Router(config)#ntp server 194.35.252.7

[/box]

NOTE: You need to force the Cisco device to advertise itself with a low stratum, typically the lower the stratum, the closer to atomic time you are supposed to be, (so we are actually forcing the device to lie, but if we don’t, Windows wont trust it!)

[box]

 Petes-Router(config)#ntp master 5 

[/box]

It can take a while for NTP, (go and have a coffee), then check it’s synchronised, DO NOT proceed until the Cisco device has synchronised.

[box]

R1#show ntp status
Clock is synchronized, stratum 5, reference is 127.127.1.1
nominal freq is 250.0000 Hz, actual freq is 250.0000 Hz, precision is 2**19
ntp uptime is 364600 (1/100 of seconds), resolution is 4000
reference time is D898D3A0.319A96D4 (23:05:04.193 GMT Wed Feb 25 2015)
clock offset is 0.0000 msec, root delay is 0.00 msec
root dispersion is 0.26 msec, peer dispersion is 0.23 msec
loopfilter state is 'CTRL' (Normal Controlled Loop), drift is -0.000000000 s/s
system poll interval is 16, last update was 3 sec ago.

[/box]

Step 2 Configure Windows to use Cisco NTP Time

In the past I’ve said “Windows Does not use NTP, it uses Win32 Time” This is not strictly true, it does use NTP, but by default it uses ‘Symmetric Active Mode NTP’ and your Cisco Device expects its NTP requests to be submitted via ‘Client Mode NTP‘. (See MS KB 875424 for more info).

Note: By default Windows Domains take their time from the PDC emulator, carry this procedure out on that server!

Locate your FSMO Role Servers

Open an elevated command prompt and execute the following commands (the Cisco device IP is shown in red, change accordingly);

[box]

w32tm /config /manualpeerlist:"123.123.123.148",0x8 /syncfromflags:MANUAL
net stop "windows time"
net start "windows time"
w32tm /resync

Note: If you want to specify TWO Cisco devices, use the following syntax

w32tm /config /manualpeerlist:"123.123.123.148,123.123.123.149",0x8 /syncfromflags:MANUAL

[/box]

Now in the Servers System log, you should see the following two events logged.

Event ID 37

Log Name: System
Source: Microsoft-Windows-Time-Service
Date: 25/02/2015 22:33:19
Event ID: 37
Task Category: None
Level: Information
Keywords:
User: LOCAL SERVICE
Computer: 2012-DC-CA.petenetlive.com
Description:
The time provider NtpClient is currently receiving valid time data from 123.123.123.148,
0x8 (ntp.m|0x8|0.0.0.0:123->123.123.123.148:123).

Event ID 35

Log Name: System
Source: Microsoft-Windows-Time-Service
Date: 25/02/2015 22:21:17
Event ID: 35
Task Category: None
Level: Information
Keywords:
User: LOCAL SERVICE
Computer: 2012-DC-CA.petenetlive.com
Description:
The time service is now synchronizing the system time with the time source 123.123.123.148,
0x8 (ntp.m|0x8|0.0.0.0:123->123.123.123.148:123).

 

Windows and Cisco NTP Problems and Errors

Event ID 47

Log Name: System
Source: Microsoft-Windows-Time-Service
Date: 25/02/2015 22:11:07
Event ID: 47
Task Category: None
Level: Warning
Keywords:
User: LOCAL SERVICE
Computer: 2012-DC-CA.petenetlive.com
Description:
Time Provider NtpClient: No valid response has been received from manually configured 
peer 123.123.123.148 after 8 attempts to contact it. This peer will be discarded as a 
time source and NtpClient will attempt to discover a new peer with this DNS name. The 
error was: The peer is unreachable.

On your Cisco Device you will see debug output like so, (it will repeat 8 times);

[box]

Petes-Router#debug ntp all
NTP events debugging is on
NTP core messages debugging is on
NTP clock adjustments debugging is on
NTP reference clocks debugging is on
NTP packets debugging is on
Petes-Router#
000031: Feb 25 22:07:45.831: NTP message received from 123.123.123.151 on interface 'GigabitEthernet0/0' (123.123.123.148).
000032: Feb 25 22:07:45.835: NTP Core(DEBUG): ntp_receive: message received
000033: Feb 25 22:07:45.835: NTP Core(DEBUG): ntp_receive: peer is 0x67A57898, next action is 1.
Petes-Router#
000034: Feb 25 22:07:54.967: NTP message received from 123.123.123.151 on interface 'GigabitEthernet0/0' (123.123.123.148).
000035: Feb 25 22:07:54.967: NTP Core(DEBUG): ntp_receive: message received
000036: Feb 25 22:07:54.971: NTP Core(DEBUG): ntp_receive: peer is 0x67A57898, next action is 1.
Petes-Router#

[/box]

Causes:

This is a pretty generic error, but in this case, one of the following situations can cause this;

1. UDP Port 123 is blocked between Windows and the Cisco NTP device.

2. The Cisco NTP device has not synchronised form a reliable NTP source.

3. The stratum of the Cisco NTP device is to high.

4. Windows is attempting to sync time using ‘Symmetric Active Mode NTP‘ See my comments above.

Related Articles, References, Credits, or External Links

Windows – Setting Domain Time

Cisco ASA – Configuring for NTP

VMware – Setting up ESX NTP Time Sync