Cisco ASA - 'Prove it's Not The Firewall!'

KB ID 0001049

Problem

Yeah, it’s funny because it’s true! The article title might not sound like the most professional approach, but when the ‘Well it’s not working now’ finger gets pointed at the ‘firewall guy/girl’, they need to ascertain two things;

1. Is the problem actually the firewall, if not then help the frustrated party track down the actual problem.

2. If your problem IS the firewall, fix it!

I’m just coming out of a major network greenfield site build, all the individual technologies that have been getting planned and designed are now starting to come online and require comms though the firewall solution that I’ve been working on. So my days are pretty much filled with conversations like this;

Consultant/Engineer: Pete I need some ports opening on the firewall.
Me: OK let me know the IP addresses, host-names, ports, protocols etc, and I’ll open them for you.
I then open the requested ports/protocols.
Consultant/Engineer: You know those ports you opened? They don’t work.

At this point one of the following has occurred;

1. I’ve made an error, (it happens I’m human), I might have entered the wrong information, or not applied an ACL, or put the rule on the wrong firewall. Always assume you have done something wrong, until you are 100% sure that’s not the case.

2. The person who asked for traffic to be allowed, asked for the wrong thing, either they didn’t RTFM, or someone has given them the wrong IP addresses, or because they are human too, they’ve made a mistake.

3. The traffics not even getting to the firewall, because either it’s getting blocked before it gets to you, or there is a routing problem stopping the traffic hitting the firewall. (Remember routing works by Unicorns and Magic, so routing people are not to be trusted!)

4. The traffic needs some kind of special inspection to work through the firewall i.e. ICMP, FTP, or PPTP etc.

5. Some annoying bug in the ASA code is stopping you, which either requires a lot of Internet and forum searching or a call to TAC to confirm.

If I’ve forgotten another reason – feel free to contact me. (Link at the bottom of the page).

Solution

Step 1: Make sure you are not blocking the Traffic

Packet tracer is your friend! Use it to simulate traffic going though the firewall, and the firewall will tell you what it will do with that traffic. I prefer to use command line, but you can also run packet tracer graphically in the ASDM.

Packer Tracer Graphically

1. From the ASDM > Tools Packet Tracer.

2. Enter the details and click start, if the firewall is blocking the traffic this should tell you where and why.

Packer-Tracer From Command Line (v 7.21 and upwards)

Syntax

packet-tracer input source_interface protocol source_address source_port destination_adress destination_port [detailed] [xml]

Syntax Description

Source_interface: Specifies the source interface name for the packet trace.
Protocol: Protocol type for the packet trace. e.g. icmp, tcp, or udp.
Source_address: The IP address for the host thats sending the traffic to be tested.
Source_port: Source port (can be and random port usually it’s the destination port that’s usually important).
Destination_address: The IP address for the host that traffic is being sent to.
Destination_port: The port that you are testing.
Detailed: (Optional) Provides detailed packet trace information.
Xml: (Optional) Displays the trace capture in XML format.

Example

Below I’m checking that an internal host (10.254.254.5) can get access to a public web server (123.123.123.123) via http (TCP port 80). Note: As mentioned above I just picked a random source port (1024).

OK, so if packet-tracer shows the firewall is not blocking the traffic. Then either there’s other ports we don’t know about that may need opening, or the traffic is not getting to the firewall. Normally at this point I’d test to see if the traffic is getting to the firewall. To do that I would do a packet capture.

To demonstrate, below someone has requested that we open https from Server A on our LAN, to an Internet server Server B.

Petes-ASA# configure terminal
Petes-ASA(config)# capture capout interface inside match tcp host 10.0.0.1 host 123.123.123.123 eq 443

Send some traffic!

Petes-ASA(config)# show capture capout

0 packet captured

0 packet shown Petes-ASA(config)#

Above the traffic is not getting to the firewall as there’s a problem between Server A and the Firewall, either something is blocking the traffic downstream, or Server A cannot route traffic to the firewall.

Below we can see traffic hitting the firewall, in fact 10.0.0.1 sends out three packets on TCP port 443 (https). What we CANNOT SEE is any traffic coming back, in this case Server B is not replying to us, either its down or it cannot route traffic back to us.

Petes-ASA(config)# show capture capout

3 packets captured

1: 20:28:47.165976 10.0.0.1.1061 > 123.123.123.123.443: S 1762767908:1762767908(0) win 64240 <mss 1460,nop,nop,sackOK> 2: 20:28:50.214649 10.0.0.1.1061 > 123.123.123.123.443: S 1762767908:1762767908(0) win 64240 <mss 1460,nop,nop,sackOK> 3: 20:28:56.168951 10.0.0.1.1061 > 123.123.123.123.443: S 1762767908:1762767908(0) win 64240 <mss 1460,nop,nop,sackOK> 3 packets shown Petes-ASA(config)#

Here is an example of what you should see;

Petes-ASA(config)# show capture capout

11 packets captured

1: 20:34:49.575806 10.0.0.1.1063 > 123.123.123.123.443: S 4084340501:4084340501(0) win 64240 <mss 1460,nop,nop,sackOK> 2: 20:34:49.576828 123.123.123.123.443 > 10.0.0.1.1063: S 4235939008:4235939008(0) ack 4084340502 win 64240 <mss 1380,nop,nop,sackOK> 3: 20:34:49.577820 10.0.0.1.1063 > 123.123.123.123.443: . ack 4235939009 win 64240 4: 20:34:49.578812 10.0.0.1.1063 > 123.123.123.123.443: P 4084340502:4084340579(77) ack 4235939009 win 64240 5: 20:34:49.582825 123.123.123.123.443 > 10.0.0.1.1063: P 4235939009:4235940127(1118) ack 4084340579 win 64163 6: 20:34:49.583816 10.0.0.1.1063 > 123.123.123.123.443: P 4084340579:4084340761(182) ack 4235940127 win 63122 7: 20:34:49.584823 123.123.123.123.443 > 10.0.0.1.1063: P 4235940127:4235940170(43) ack 4084340761 win 63981 8: 20:34:49.804783 10.0.0.1.1063 > 123.123.123.123.443: . ack 4235940170 win 63079 9: 20:35:20.378322 10.0.0.1.1063 > 123.123.123.123.443: F 4084340761:4084340761(0) ack 4235940170 win 63079 10: 20:35:20.379344 123.123.123.123.443 > 10.0.0.1.1063: . ack 4084340762 win 63981 11: 20:35:20.379405 123.123.123.123.443 > 10.0.0.1.1063: R 4235940170:4235940170(0) ack 4084340762 win 0 11 packets shown Petes-ASA(config)#

Now the port(s) we want to allow, we can see are actually working, so if theres still a problem, theres probably another port / protocol that’s being blocked. To find out we need to enable logging and see if any packets are being denied.

Petes-ASA#configure terminal Petes-ASA(config)# logg buffer-size 4096 Petes-ASA(config)# logg buffered 7 Petes-ASA(config)# logg on

Try the connection again, then view the log, (here I’m filtering it on 10.0.0.1, as the log can be quite sizable);

Petes-ASA(config)# show logg | incl 10.0.0.1 %ASA-7-609001: Built local-host inside:10.0.0.1 %ASA-6-302013: Built outbound TCP connection 15 for outside:123.123.123.123/443 (123.123.123.123/443) to inside:10.0.0.1/1070 (10.0.0.1/1070) %ASA-4-106023: Deny tcp src inside:10.0.0.1/1073 dst outside:123.123.123.123/21 by access-group “outbound” [0x0, 0x0] %ASA-4-106023: Deny tcp src inside:10.0.0.1/1073 dst outside:123.123.123.123/21 by access-group “outbound” [0x0, 0x0] %ASA-4-106023: Deny tcp src inside:10.0.0.1/1073 dst outside:123.123.123.123/21 by access-group “outbound” [0x0, 0x0] %ASA-6-302014: Teardown TCP connection 15 for outside:123.123.123.123/443 to inside:10.0.0.1/1070 duration 0:00:30 bytes 1420 TCP FINs %ASA-7-609002: Teardown local-host inside:10.0.0.1 duration 0:00:30 Petes-ASA(config)#

As we can see traffic is being denied and it’s on TCP port 21 (That’s FTP if your interested). So let’s open that port, and try again;

Petes-ASA(config)# show logg | incl 10.0.0.1 %ASA-5-111008: User ‘enable_15’ executed the ‘access-list outbound extended permit tcp host 10.0.0.1 host 123.123.123.123 eq 21’ command. %ASA-5-111010: User ‘enable_15’, running ‘CLI’ from IP 0.0.0.0, executed ‘access-list outbound extended permit tcp host 10.0.0.1 host 123.123.123.123 eq 21’ %ASA-7-609001: Built local-host inside:10.0.0.1 %ASA-6-302013: Built outbound TCP connection 16 for outside:123.123.123.123/443 (123.123.123.123/443) to inside:10.0.0.1/1077 (10.0.0.1/1077) %ASA-6-302013: Built outbound TCP connection 17 for outside:123.123.123.123/21 (123.123.123.123/21) to inside:10.0.0.1/1080 (10.0.0.1/1080) %ASA-6-302014: Teardown TCP connection 16 for outside:123.123.123.123/443 to inside:10.0.0.1/1077 duration 0:00:30 bytes 1420 TCP FINs Petes-ASA(config)#

And we are working!

If we have got this far and you are still not working, then check the traffic you are trying to send does not need any special inspection enabling. Or the port number you are using may have been reserved for a particular type of traffic (like this).

Failing that, upgrade the ASA, then open a TAC call.

Cisco ASA – ‘Prove it’s Not The Firewall!’

Problem

Solution

Step 1: Make sure you are not blocking the Traffic

Related Articles, References, Credits, or External Links