Cisco ASA – I Cannot Ping External Addresses? (Troubleshooting ICMP)

KB ID 0000914 

Problem

Considering we use ICMP to test connectivity, the fact that it is not a stateful protocol can be a major pain! Last week one of my colleagues rang me up and said, “Can you jump on this firewall, I’ve got no comms, and I cant ping external IP addresses. I can ping the internet from the firewall and I can ping internal IP addresses form the firewall”.

Solution

1. Before we start, lets get the basics out of the way, does the client you are pinging from have a firewall turned on? Can you ping the inside interface of the firewall?

2. Pinging will never work unless you have ICMP inspection turned on on the firewall. See the following article.

Cisco Firewalls and PING

Using Packet-Tracer to Test Ping/ICMP

3. At this point we troubleshoot as we would for any other traffic through the firewall. To do this we use packet-tracer, the syntax is slightly different for ICMP, than it is for TCP and UDP though. We need to specify an ICMP Type and an ICMP code, to make sure the traffic leaves the firewall we trace ICMP type 8 (echo), with ICMP code 0 (none). And ICMP type 0 (Echo-reply), with ICMP code 0 (none), for traffic inbound.

ICMP Types and Codes

[box]

Test Outbound Ping

Petes-ASA# packet-tracer input inside icmp 192.168.1.1 8 0 4.2.2.2

Testing Inbound Ping (where 123.123.123.123 is the public IP you are mapped to)

Petes-ASA# packet-tracer input outside icmp 4.2.2.2 0 0 123.123.123.123

[/box]

Note: You need to use the public addresses or this will happen.

4. Make sure the client you are on is getting NATTED or PATTED through the firewall. Below we will assume my internal IP address is 192.168.1.1.

Note: If you have names enabled and 192.168.1.1 has a name, you will get no results! issue a no names command from configure terminal mode to check.

[box]

Petes-ASA# show xlate | incl 192.168.1.1

If this machine was being NATTED to another public IP address it would look like..

NAT from inside:192.168.1.1 to outside:123.123.123.124

If this machine was being PATTED to a public IP address it would look like..

ICMP PAT from inside:192.168.1.1/1 to outside:123.123.123.123/1 flags ri idle 0:00:07 timeout 0:00:30

[/box]

If it fails at this stage then check you network translation configuration on the firewall.

5. If all appears normal so far you can capture the traffic as it passes though the firewall, below I’m successfully capturing the ICMP traffic though the firewall.

[box]

Petes-ASA# configure terminal
Petes-ASA(config)# capture capout interface inside match icmp host 192.168.1.1 any
Petes-ASA(config)# capture capin interface outside match icmp host 4.2.2.2 any

At this point attempt to ping, so some traffic is captured

Petes-ASA(config)# show capture capout

8 packets captured

1: 12:56:51.089244 192.168.1.1 > 4.2.2.2: icmp: echo request
2: 12:56:51.104410 4.2.2.2 > 192.168.1.1: icmp: echo reply
3: 12:56:52.092829 192.168.1.1 > 4.2.2.2: icmp: echo request
4: 12:56:52.108926 4.2.2.2 > 192.168.1.1: icmp: echo reply
5: 12:56:53.098688 192.168.1.1 > 4.2.2.2: icmp: echo request
6: 12:56:53.113809 4.2.2.2 > 192.168.1.1: icmp: echo reply
7: 12:56:54.105463 192.168.1.1 > 4.2.2.2: icmp: echo request
8: 12:56:54.120263 4.2.2.2 > 192.168.1.1: icmp: echo reply

Petes-ASA(config)# show capture capin

8 packets captured

1: 12:57:28.170981 123.123.123.123 > 4.2.2.2: icmp: echo request
2: 12:57:28.185949 4.2.2.2 > 123.123.123.123: icmp: echo reply
3: 12:57:29.175238 123.123.123.123 > 4.2.2.2: icmp: echo request
4: 12:57:29.190084 4.2.2.2 > 123.123.123.123: icmp: echo reply
5: 12:57:30.180212 123.123.123.123 > 4.2.2.2: icmp: echo request
6: 12:57:30.195500 4.2.2.2 > 123.123.123.123: icmp: echo reply
7: 12:57:31.186101 123.123.123.123 > 4.2.2.2: icmp: echo request
8: 12:57:31.201680 4.2.2.2 > 123.123.123.123: icmp: echo reply
8 packets shown

[/box]

Note: If your capout capture looks like the following, then you didn’t have inspect icmp enabled on your policy-map.

[box]

Petes-ASA(config)# show capture capout

4 packets captured

1: 13:02:33.285309 192.168.1.1 > 4.2.2.2: icmp: echo request
2: 13:02:37.886596 192.168.1.1 > 4.2.2.2: icmp: echo request
3: 13:02:42.886672 192.168.1.1 > 4.2.2.2: icmp: echo request
4: 13:02:47.888198 192.168.1.1 > 4.2.2.2: icmp: echo request
4 packets shown

[/box]

How Do I Clear or Delete a Cisco ASA Capture?

[box]

To clear a Capture, but leave it running;

Petes-ASA(config)# clear capture capin
Petes-ASA(config)# clear capture capout
Petes-ASA(config)# show capture capin

0 packet captured

0 packet shown
Petes-ASA(config)# show capture capout

0 packet captured

0 packet shown

To Delete a Capture;

Petes-ASA(config)# no capture capout
Petes-ASA(config)# no capture capin  

[/box]

Related Articles, References, Credits, or External Links

Cisco Firewalls and PING

Cisco ASA 5500 Allowing Tracert

 

Cisco ASA – ‘Prove it’s Not The Firewall!’

KB ID 0001049 

Problem

Yeah, it’s funny because it’s true! The article title might not sound like the most professional approach, but when the ‘Well it’s not working now’ finger gets pointed at the ‘firewall guy/girl’, they need to ascertain two things;

1. Is the problem actually the firewall, if not then help the frustrated party track down the actual problem.

2. If your problem IS the firewall, fix it!

I’m just coming out of a major network greenfield site build, all the individual technologies that have been getting planned and designed are now starting to come online and require comms though the firewall solution that I’ve been working on. So my days are pretty much filled with conversations like this;

Consultant/Engineer: Pete I need some ports opening on the firewall.
Me: OK let me know the IP addresses, host-names, ports, protocols etc, and I’ll open them for you.
I then open the requested ports/protocols.
Consultant/Engineer: You know those ports you opened? They don’t work.

At this point one of the following has occurred;

1. I’ve made an error, (it happens I’m human), I might have entered the wrong information, or not applied an ACL, or put the rule on the wrong firewall. Always assume you have done something wrong, until you are 100% sure that’s not the case.

2. The person who asked for traffic to be allowed, asked for the wrong thing, either they didn’t RTFM, or someone has given them the wrong IP addresses, or because they are human too, they’ve made a mistake.

3. The traffics not even getting to the firewall, because either it’s getting blocked before it gets to you, or there is a routing problem stopping the traffic hitting the firewall. (Remember routing works by Unicorns and Magic, so routing people are not to be trusted!)

4. The traffic needs some kind of special inspection to work through the firewall i.e. ICMP, FTP, or PPTP etc.

5. Some annoying bug in the ASA code is stopping you, which either requires a lot of Internet and forum searching or a call to TAC to confirm.

If I’ve forgotten another reason – feel free to contact me. (Link at the bottom of the page).

Solution

Step 1: Make sure you are not blocking the Traffic

Packet tracer is your friend! Use it to simulate traffic going though the firewall, and the firewall will tell you what it will do with that traffic. I prefer to use command line, but you can also run packet tracer graphically in the ASDM.

Packer Tracer Graphically

1. From the ASDM > Tools Packet Tracer.

2. Enter the details and click start, if the firewall is blocking the traffic this should tell you where and why.

Packer-Tracer From Command Line (v 7.21 and upwards)

Syntax

[box]packet-tracer input source_interface protocol source_address source_port destination_adress destination_port [detailed] [xml] [/box]

Syntax Description

  • Source_interface: Specifies the source interface name for the packet trace.
  • Protocol: Protocol type for the packet trace. e.g. icmp, tcp, or udp.
  • Source_address: The IP address for the host thats sending the traffic to be tested.
  • Source_port: Source port (can be and random port usually it’s the destination port that’s usually important).
  • Destination_address: The IP address for the host that traffic is being sent to.
  • Destination_port: The port that you are testing.
  • Detailed: (Optional) Provides detailed packet trace information.
  • Xml: (Optional) Displays the trace capture in XML format.

Example

Below I’m checking that an internal host (10.254.254.5) can get access to a public web server (123.123.123.123) via http (TCP port 80). Note: As mentioned above I just picked a random source port (1024).

OK, so if packet-tracer shows the firewall is not blocking the traffic. Then either there’s other ports we don’t know about that may need opening, or the traffic is not getting to the firewall. Normally at this point I’d test to see if the traffic is getting to the firewall. To do that I would do a packet capture.

To demonstrate, below someone has requested that we open https from Server A on our LAN, to an Internet server Server B.

 

[box]

Petes-ASA# configure terminal
Petes-ASA(config)# capture capout interface inside match tcp host 10.0.0.1 host 123.123.123.123 eq 443

[/box]

Send some traffic!

[box] Petes-ASA(config)# show capture capout

0 packet captured

0 packet shown
Petes-ASA(config)#

[/box]

Above the traffic is not getting to the firewall as there’s a problem between Server A and the Firewall, either something is blocking the traffic downstream, or Server A cannot route traffic to the firewall.

Below we can see traffic hitting the firewall, in fact 10.0.0.1 sends out three packets on TCP port 443 (https). What we CANNOT SEE is any traffic coming back, in this case Server B is not replying to us, either its down or it cannot route traffic back to us.

[box] Petes-ASA(config)# show capture capout

3 packets captured

1: 20:28:47.165976 10.0.0.1.1061 > 123.123.123.123.443: S 1762767908:1762767908(0) win 64240 <mss 1460,nop,nop,sackOK>
2: 20:28:50.214649 10.0.0.1.1061 > 123.123.123.123.443: S 1762767908:1762767908(0) win 64240 <mss 1460,nop,nop,sackOK>
3: 20:28:56.168951 10.0.0.1.1061 > 123.123.123.123.443: S 1762767908:1762767908(0) win 64240 <mss 1460,nop,nop,sackOK>
3 packets shown
Petes-ASA(config)#

[/box]

Here is an example of what you should see;

[box]Petes-ASA(config)# show capture capout

11 packets captured

1: 20:34:49.575806 10.0.0.1.1063 > 123.123.123.123.443: S 4084340501:4084340501(0) win 64240 <mss 1460,nop,nop,sackOK>
2: 20:34:49.576828 123.123.123.123.443 > 10.0.0.1.1063: S 4235939008:4235939008(0) ack 4084340502 win 64240 <mss 1380,nop,nop,sackOK>
3: 20:34:49.577820 10.0.0.1.1063 > 123.123.123.123.443: . ack 4235939009 win 64240
4: 20:34:49.578812 10.0.0.1.1063 > 123.123.123.123.443: P 4084340502:4084340579(77) ack 4235939009 win 64240
5: 20:34:49.582825 123.123.123.123.443 > 10.0.0.1.1063: P 4235939009:4235940127(1118) ack 4084340579 win 64163
6: 20:34:49.583816 10.0.0.1.1063 > 123.123.123.123.443: P 4084340579:4084340761(182) ack 4235940127 win 63122
7: 20:34:49.584823 123.123.123.123.443 > 10.0.0.1.1063: P 4235940127:4235940170(43) ack 4084340761 win 63981
8: 20:34:49.804783 10.0.0.1.1063 > 123.123.123.123.443: . ack 4235940170 win 63079
9: 20:35:20.378322 10.0.0.1.1063 > 123.123.123.123.443: F 4084340761:4084340761(0) ack 4235940170 win 63079
10: 20:35:20.379344 123.123.123.123.443 > 10.0.0.1.1063: . ack 4084340762 win 63981
11: 20:35:20.379405 123.123.123.123.443 > 10.0.0.1.1063: R 4235940170:4235940170(0) ack 4084340762 win 0
11 packets shown
Petes-ASA(config)#

[/box]

Now the port(s) we want to allow, we can see are actually working, so if theres still a problem, theres probably another port / protocol that’s being blocked. To find out we need to enable logging and see if any packets are being denied.

[box]Petes-ASA#configure terminal
Petes-ASA(config)# logg buffer-size 4096
Petes-ASA(config)# logg buffered 7
Petes-ASA(config)# logg on [/box]

Try the connection again, then view the log, (here I’m filtering it on 10.0.0.1, as the log can be quite sizable);

[box]Petes-ASA(config)# show logg | incl 10.0.0.1
%ASA-7-609001: Built local-host inside:10.0.0.1
%ASA-6-302013: Built outbound TCP connection 15 for outside:123.123.123.123/443 (123.123.123.123/443) to inside:10.0.0.1/1070 (10.0.0.1/1070)
%ASA-4-106023: Deny tcp src inside:10.0.0.1/1073 dst outside:123.123.123.123/21 by access-group “outbound” [0x0, 0x0]
%ASA-4-106023: Deny tcp src inside:10.0.0.1/1073 dst outside:123.123.123.123/21 by access-group “outbound” [0x0, 0x0]
%ASA-4-106023: Deny tcp src inside:10.0.0.1/1073 dst outside:123.123.123.123/21 by access-group “outbound” [0x0, 0x0]

%ASA-6-302014: Teardown TCP connection 15 for outside:123.123.123.123/443 to inside:10.0.0.1/1070 duration 0:00:30 bytes 1420 TCP FINs
%ASA-7-609002: Teardown local-host inside:10.0.0.1 duration 0:00:30
Petes-ASA(config)# [/box]

As we can see traffic is being denied and it’s on TCP port 21 (That’s FTP if your interested). So let’s open that port, and try again;

[box]Petes-ASA(config)# show logg | incl 10.0.0.1
%ASA-5-111008: User ‘enable_15’ executed the ‘access-list outbound extended permit tcp host 10.0.0.1 host 123.123.123.123 eq 21’ command.
%ASA-5-111010: User ‘enable_15’, running ‘CLI’ from IP 0.0.0.0, executed ‘access-list outbound extended permit tcp host 10.0.0.1 host 123.123.123.123 eq 21’
%ASA-7-609001: Built local-host inside:10.0.0.1
%ASA-6-302013: Built outbound TCP connection 16 for outside:123.123.123.123/443 (123.123.123.123/443) to inside:10.0.0.1/1077 (10.0.0.1/1077)
%ASA-6-302013: Built outbound TCP connection 17 for outside:123.123.123.123/21 (123.123.123.123/21) to inside:10.0.0.1/1080 (10.0.0.1/1080)
%ASA-6-302014: Teardown TCP connection 16 for outside:123.123.123.123/443 to inside:10.0.0.1/1077 duration 0:00:30 bytes 1420 TCP FINs
Petes-ASA(config)# [/box]

And we are working!

If we have got this far and you are still not working, then check the traffic you are trying to send does not need any special inspection enabling. Or the port number you are using may have been reserved for a particular type of traffic (like this).

Failing that, upgrade the ASA, then open a TAC call.

Related Articles, References, Credits, or External Links

NA