If you look after a firewall, sooner or later something will fail, and the blame (rightly or wrongly), will be leveled at the firewall. I came back from holiday this week to find a client had got a problem with secure POP email. The problem had been fixed (temporarily) by dropping the affected users into a group, and opening all ports. As this had fixed the problem then it’s fair to say that the ASA was the root cause of the problem.
So I was asked to take a look and open the correct ports and lock the firewall back down again.
Solution
Step 1 – Setting up logging on the ASA
I’m going to do some real time testing, so the internal buffer on the ASA will hold enough logs for me, if you have an intermittent problem you might want to setup an external syslog server. I’m going to set the log buffer size, and the logging level, and finally turn logging on.
[box]User Access Verification
Password:
Type help or ‘?’ for a list of available commands.
PetesASA> enable
Password: *******
PetesASA# conf t
PetesASA(config) logg buffer-size 4096
PetesASA(config)# logg buffered 7
PetesASA(config)# logg on[/box]
Step 2 – Attempt communication
At this point I got the client to attempt connection to the secure POP server, then had a look at the logs. I could view the whole log with ‘show logg’, but I filtered it down just to include traffic to and from this client (192.168.1.2).
Note: The ports being used are highlighted in red, (YES I know that these are the ports required for secure POP, but your application could be using anything!)
[box]PetesASA(config)# show logg | inc 192.168.1.2
There are a few ways of doing this. I just created some network objects, then if any other hosts need secure POP, I can simply add them to the object group.
WARNING: This assumes you DON’T have an outbound traffic access list. If you DO replace the word ‘outbound’ with the name of yours. Also remember as soon as you allow traffic like this all other traffic gets blocked!
Step 4 – Disable Logging
Simply prefix your earlier command with the word ‘no’.
[box]PetesASA(config)# no logg on[/box]
Related Articles, References, Credits, or External Links
Yeah, it’s funny because it’s true! The article title might not sound like the most professional approach, but when the ‘Well it’s not working now’ finger gets pointed at the ‘firewall guy/girl’, they need to ascertain two things;
1. Is the problem actually the firewall, if not then help the frustrated party track down the actual problem.
2. If your problem IS the firewall, fix it!
I’m just coming out of a major network greenfield site build, all the individual technologies that have been getting planned and designed are now starting to come online and require comms though the firewall solution that I’ve been working on. So my days are pretty much filled with conversations like this;
Consultant/Engineer: Pete I need some ports opening on the firewall.
Me: OK let me know the IP addresses, host-names, ports, protocols etc, and I’ll open them for you. I then open the requested ports/protocols. Consultant/Engineer: You know those ports you opened? They don’t work.
At this point one of the following has occurred;
1. I’ve made an error, (it happens I’m human), I might have entered the wrong information, or not applied an ACL, or put the rule on the wrong firewall. Always assume you have done something wrong, until you are 100% sure that’s not the case.
2. The person who asked for traffic to be allowed, asked for the wrong thing, either they didn’t RTFM, or someone has given them the wrong IP addresses, or because they are human too, they’ve made a mistake.
3. The traffics not even getting to the firewall, because either it’s getting blocked before it gets to you, or there is a routing problem stopping the traffic hitting the firewall. (Remember routing works by Unicorns and Magic, so routing people are not to be trusted!)
4. The traffic needs some kind of special inspection to work through the firewall i.e. ICMP, FTP, or PPTP etc.
5. Some annoying bug in the ASA code is stopping you, which either requires a lot of Internet and forum searching or a call to TAC to confirm.
If I’ve forgotten another reason – feel free to contact me. (Link at the bottom of the page).
Solution
Step 1: Make sure you are not blocking the Traffic
Packet tracer is your friend! Use it to simulate traffic going though the firewall, and the firewall will tell you what it will do with that traffic. I prefer to use command line, but you can also run packet tracer graphically in the ASDM.
Xml: (Optional) Displays the trace capture in XML format.
Example
Below I’m checking that an internal host (10.254.254.5) can get access to a public web server (123.123.123.123) via http (TCP port 80). Note: As mentioned above I just picked a random source port (1024).
OK, so if packet-tracer shows the firewall is not blocking the traffic. Then either there’s other ports we don’t know about that may need opening, or the traffic is not getting to the firewall. Normally at this point I’d test to see if the traffic is getting to the firewall. To do that I would do a packet capture.
To demonstrate, below someone has requested that we open https from Server A on our LAN, to an Internet server Server B.
Above the traffic is not getting to the firewall as there’s a problem between Server A and the Firewall, either something is blocking the traffic downstream, or Server A cannot route traffic to the firewall.
Below we can see traffic hitting the firewall, in fact 10.0.0.1 sends out three packets on TCP port 443 (https). What we CANNOT SEE is any traffic coming back, in this case Server B is not replying to us, either its down or it cannot route traffic back to us.
Now the port(s) we want to allow, we can see are actually working, so if theres still a problem, theres probably another port / protocol that’s being blocked. To find out we need to enable logging and see if any packets are being denied.
Try the connection again, then view the log, (here I’m filtering it on 10.0.0.1, as the log can be quite sizable);
[box]Petes-ASA(config)# show logg | incl 10.0.0.1
%ASA-7-609001: Built local-host inside:10.0.0.1
%ASA-6-302013: Built outbound TCP connection 15 for outside:123.123.123.123/443 (123.123.123.123/443) to inside:10.0.0.1/1070 (10.0.0.1/1070) %ASA-4-106023: Deny tcp src inside:10.0.0.1/1073 dst outside:123.123.123.123/21 by access-group “outbound” [0x0, 0x0]
%ASA-4-106023: Deny tcp src inside:10.0.0.1/1073 dst outside:123.123.123.123/21 by access-group “outbound” [0x0, 0x0]
%ASA-4-106023: Deny tcp src inside:10.0.0.1/1073 dst outside:123.123.123.123/21 by access-group “outbound” [0x0, 0x0]
%ASA-6-302014: Teardown TCP connection 15 for outside:123.123.123.123/443 to inside:10.0.0.1/1070 duration 0:00:30 bytes 1420 TCP FINs
%ASA-7-609002: Teardown local-host inside:10.0.0.1 duration 0:00:30
Petes-ASA(config)# [/box]
As we can see traffic is being denied and it’s on TCP port 21 (That’s FTP if your interested). So let’s open that port, and try again;
[box]Petes-ASA(config)# show logg | incl 10.0.0.1
%ASA-5-111008: User ‘enable_15’ executed the ‘access-list outbound extended permit tcp host 10.0.0.1 host 123.123.123.123 eq 21’ command.
%ASA-5-111010: User ‘enable_15’, running ‘CLI’ from IP 0.0.0.0, executed ‘access-list outbound extended permit tcp host 10.0.0.1 host 123.123.123.123 eq 21’
%ASA-7-609001: Built local-host inside:10.0.0.1
%ASA-6-302013: Built outbound TCP connection 16 for outside:123.123.123.123/443 (123.123.123.123/443) to inside:10.0.0.1/1077 (10.0.0.1/1077)
%ASA-6-302013: Built outbound TCP connection 17 for outside:123.123.123.123/21 (123.123.123.123/21) to inside:10.0.0.1/1080 (10.0.0.1/1080)
%ASA-6-302014: Teardown TCP connection 16 for outside:123.123.123.123/443 to inside:10.0.0.1/1077 duration 0:00:30 bytes 1420 TCP FINs
Petes-ASA(config)# [/box]
And we are working!
If we have got this far and you are still not working, then check the traffic you are trying to send does not need any special inspection enabling. Or the port number you are using may have been reserved for a particular type of traffic (like this).
Failing that, upgrade the ASA, then open a TAC call.
Related Articles, References, Credits, or External Links
