I’ve been setting up a VPN solution on the test bench as I’m looking at Always On VPN. When I noticed that I had a problem with my remote VPN connections on Windows. They would connect fine but I could not resolve any FQDNs for my domain?
VPN no DNS Solution
By default, all (Windows) VPN connections are ‘Force Tunnel’ (this means they have the option ‘Use default gateway on remote network’ selected). This also means that, (unless your RAS server is the default Gateway for your network,) you usually don’t have internet access when connected to the VPN.
Now I connected fine, and I could ping IP addresses on my corporate network, but I could not ping my servers by their domain name, in fact Windows was trying to resolve my domain name to a public IP?
Google this problem and you’re simply told to ‘Disable IPv6 on your network card, and this works, (if you want to keep your remote users Force-Tunnelled). But disabling IPv6 is hardly a fix is it?
Also If you want internet access for your remote clients, (Commonly referred to as ‘Split Tunnel’), then even with IPv6 disabled, the problem comes back!
Why is this happening? Well even with Force Tunnel enabled, you can still use your local LAN (Connect to your VPN, and ping your home gateway, or printer or wireless access point if you don’t believe me!) This connection takes precedence over your remote VPN connection, to prove it run a netstat -rn command.
From the above you can see my Ethernet Adaptor has a metric of 6, and my VPN connector, (in this case called Connection Template) has metric of 23. AND THE LOWEST ONE WINS, so your DNS queries are going out of your local internet connection NOT down the VPN tunnel!
How Do I Fix this VPN no DNS?
Well until Microsoft fixes this in Windows 10, (it’s fine on Windows 8 and earlier), you have to manipulate the metrics yourself, like so;
VPN no DNS On Your Physical Adapter;
Start > ncpa.cpl {enter} > Right click your NIC > Properties > Internet Protocol Version 4 > Properties.
Advanced > Untick ‘Automatic Metric’ > Set the Interface Metric to 20 > OK > OK >OK.
On Your VPN Connector;
Start > ncpa.cpl {enter} > Right click your VPN Connector > Properties > Internet Protocol Version 4 > Properties.
Advanced > Untick ‘Automatic Metric’ > Set the Interface Metric to 10 > OK > OK >OK.
Now your DNS look-ups should behave!
Related Articles, References, Credits, or External Links
I use EVE-NG a lot, it’s an awesome tool. Also I’m lucky enough to have my own ESX servers, so that’s where it lives. I’ve noticed this problem before, but I’ve either given up, and done something else, or it’s manifested itself in an ‘odd’ way that I can work around.
If you’re new to connecting EVE-NG to a live network you might want to read the following post first;
When setting up a new lab, I connected a Ciso IOL router to my cloud object, and it successfully got an IP from DHCP, but could not communicate with the outside world. So I replaced it with a Cisco (Dynamips) router, same thing! So I replaced with with a Cisco ASAv, same thing!
I moved the EVE-NG machine onto its own standard vSwitch, (no difference).
I hadn’t committed the ‘schoolboy error‘ of forgetting to allow promiscuous mode on the port group.
I could also see from my physical network, that there was layer 2 connectivity happening, as it was in in the ARP cache of my core switch.
I did notice that if I waited for a long time, it would start working, so (on the Core Switch) I flushed the ARP cache, and pinged the EVE-NG device and got a response, then it worked fine from EVE-NG, (for a while, in a manner of speaking!) If I tried to NAT any other traffic, or do anything else, then the problem returned. I could keep flushing the ARP cache on the switch, but that’s a bit annoying?
Solution
Well, (in my case) the problem turned out to be a problem with the fact I have ‘teamed‘ physical NICs on the vSwitch, which you can see above (vmnic0, and vmnic7). AS SOON as I removed one, and only had one physical uplink it worked faultlessly?
Everything works now.
Note: I tried changing the load balancing algorithms to ‘Route based on IP hash‘, ‘Route based on source MAC hash, and even ‘Use Explicit failover order‘, none of these worked.
I did see other people in forums that were saying, ‘I only have one physical uplink‘, I’m suspecting that in their case, it’s promiscuous mode was missing, but feel free to comment below, if any one manages a better work-around / fix / explanation.
Related Articles, References, Credits, or External Links
This is a strange one? I was deploying FirePOWER to a pair of ASA 5550-8-X firewalls in Active / Standby failover last week. After each SFR was updated (via ASDM.) I could no longer ‘ping it’, the SFR itself could ping everything on the same VLAN, APART from its own default gateway, (which was an SVI on the Cisco 3750 switch it was connected to).
This happened every time I updated the SFR, (or re-imaged it.) Then after an hour or so it was fine?
Solution
If I connected to the switch that the SFR, (and firewall) was connected to, I could NOT ping the SFR. The interface was up/up on the switch, and the firewalls Management interface was also up/up.
[box]
Petes-3750#ping 10.2.1.252
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.2.1.252, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
[/box]
I did notice it was in the ARP table though, (with the correct MAC address), So I manually removed it;
[box]
Petes-3750#clear ip arp 10.2.1.252
[/box]
Then it was fine?
[box]
Petes-3750#ping 10.2.1.252
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.2.1.252, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
With regards to Ping, out of the box a Cisco firewall will allow you to ping the interface you are connected to, so in a normal setup inside clients can ping the inside interface, and the firewalls outside interface can be pinged from outside.
OK – to understand pinging through a Cisco Firewall you need to understand that Ping is part of the ICMP protocol suite, and unlike other protocols is not “connection orientated” what that means is, (on a new firewall that has no rules applied outbound) the firewall happily lets ping traffic out but it won’t let ping traffic back in – this results in a failure on the client.
Cisco ASA and Cisco PIX (version 7 and above) From CLI
Version 7 introduced an ICMP inspection engine so that it could track ICMP requests like other protocols. It’s NOT turned on by default. And the command is “inspect icmp” but you need to enter the default map first (this assumes you have the standard policy-map). Connect to the firewall and use the following commands from config terminal mode. then save the changes with a “write mem” command.
[box]
PetesASA>
PetesASA> enable
Password:********
PetesASA# conf t
PetesASA(config)# policy-map global_policy
PetesASA(config)# (config-pmap)#class inspection_default
PetesASA(config)#inspect icmp
PetesASA(config)# write mem
Building configuration...
Cryptochecksum: 5c8dfc45 ee6496db 8731d2d5 fa945425
8695 bytes copied in 3.670 secs (2898 bytes/sec)
[OK]
PetesASA(config)#
[/box]
Cisco ASA and Cisco PIX (version 7 and above) FromASDM
Connect to the ASDM > Configuration > Firewall > Service Policy Rules > Select “inspection_default” > Edit > Rule Actions > Tick ICMP > OK > Apply > File > Save running configuration to flash.
Older firewalls do not have an inspection map, nor was there a “fixup” for ICMP and ping traffic, so you need to explicitly allow the return icmp traffic back in. Note: this assumes you already have an inbound access-list called “inbound”, and we are adding some more lines to it, change the works inbound to match the name/number of your inbound access list (the “show access-group” will tell you what is applied to the outside interface),
e.g.
[box]
PetesASA>
PetesASA> enable
Password: ********
PetesASA# show access-group
access-group inbound in interface outside <<This ones called inbound.
[/box]
If you already have an access-list applied then simply substitute the name word “inbound” for the name on your ACL.
[box]
PetesASA# conf t
PetesASA(config)# access-list inbound permit icmp any any echo-reply
PetesASA(config)# access-list inbound permit icmp any any time-exceeded
PetesASA(config)# access-list inbound permit icmp any any unreachablePetesASA(config)# access-list inbound permit icmp any any source-quench
[/box]
If you HAD an inbound ACL skip this step, If you didn’t have one you need to apply the ACL with an access-group command.
[box]
PetesASA(config)# access-group inbound in interface outside
[/box]
Lastly save your work with a write mem command.
[box]
PetesASA# write mem
Building configuration...
Cryptochecksum: 4d7f7ccd 5c55a9e1 6ced12c4 46728bc7
[OK]
PetesASA#
Connect to the PDM > Configuration > Access Rules > Rules > Add > Permit > Outside Inside > Tick ICMP > Select “echo-reply”> OK > Apply > File > Save running configuration to flash.
Then repeat for time-exceeded, unreachable and source-quench
Stop Interfaces replying to Ping traffic
As stated above all firewall interfaces will respond to pings if they are on the network you are connected to. To stop this you use the “icmp” command.
[box]
Syntax
icmp {permit or deny} {any or IP and subnet}{type of icmp traffic}{interface name}
Example
icmp deny any echo outside
Connect to the PDM > Configuration > system Properties > Administration > ICMP > Add > > Specify the Type, Interface, Source etc > OK > Apply > File > Save running configuration to flash.
If you are still having problems, see the following article;
Note: May also be asked as, Client VPN connects but cannot ping anything behind the Firewall.
KB ID 0000199
Problem
If I had a pound for every time I’ve seen this either in the wild, or asked in a forum, I would be minted! In nearly every case the problem is NAT related.
In most cases, If the person launching the VPN client is behind a device that is performing NAT, (Home Router, Access Point, Firewall, etc) then the device will BREAK the NO NAT, or “nat 0” on pre 8.3 firewalls. (that’s the command that says “DONT change the address of my remote VPN client as it passes up and down the VPN tunnel).
Update 08/09/16: Due to a bug, I found an exception to this problem being broken NAT (see below)
Solution
Enable nat-traversal, this is a global configuration setting and will not affect any other site to site, or client to gateway VPN’s you are currently running.
Then go to enable mode > Configure Terminal mode > and issue a “crypto isakmp nat-traversal 20” command >Then save the change with a “write mem” command.
[box]
User Access Verification
Password:
Type help or '?' for a list of available commands.
Petes-ASA> enable
Password: ********Petes-ASA# configure terminal
Petes-ASA(config)# crypto isakmp nat-traversal 20
Petes-ASA# write mem
Building configuration...
Cryptochecksum: 79745c0a 509726e5 b2c66028 021fdc7d
7424 bytes copied in 1.710 secs (7424 bytes/sec)
[OK]
Petes-ASA#
If you can find this in the ASDM post version 7 – You are better than me!
Navigate to > Configuration > Remote Access VPN > Advanced > IKE Parameters > Tick “Enable IPSec over NAT-T” option > Set the “NAT Keepalive” to 20 seconds > Apply > File > Save running configuration to flash.
I’ve done that and its still not working?
On a Firewall Running 8.3 (or Newer)
1. On the firewall issue a “show run nat” command > Make sure there is a NAT statement that has static (the network behind the ASA) to static (the remote VPN network). I’ve highlighted it below.
[box]
User Access Verification
Password:
Type help or '?' for a list of available commands.
Petes-ASA>enable
Password: ********
Petes-ASA# show run natnat (inside,any) source static obj-10.254.254.0 obj-10.254.254.0 destination static obj-10.253.253.0 obj-10.253.253.0 route-lookup
!
object network obj_any
nat (inside,outside) dynamic interface
object network Media_PC
nat (inside,outside) static interface service tcp 123 123
!
nat (outside,outside) after-auto source dynamic VPN_Pool interface
PetesASA#
[/box]
2. Make sure the correct network(s) are in the correct groups.
On the firewall issue a “show run nat 0” command > take note of the access-list name.
[box]
User Access Verification
Password:
Type help or '?' for a list of available commands.
Petes-ASA> enable
Password: ********
Petes-ASA# show run nat 0
nat (inside) 0 access-list NO-NAT-TRAFFIC
nat (inside) 1 0.0.0.0 0.0.0.0
[/box]
In this example mines called NO-NAT-TRAFFIC (cause I like to keep things simple) yours can be called anything (inside_nat0_outbound is the norm if you used the ASDM to set up the VPN).
Now make sure that you have the correct addresses in that access-list, issue a show run access-list {name} command.
[box]
Petes-ASA#show run access-list NO-NAT-TRAFFIC
access-list NO-NAT-TRAFFIC extended permit ip 10.254.254.0 255.255.255.0 10.253.253.0 255.255.255.0
access-list NO-NAT-TRAFFIC extended permit ip 10.254.254.0 255.255.255.0 10.252.252.0 255.255.255.0
Petes-ASA#
[/box]
Above we have two subnets that are going to be exempt from NAT, they are 10.253.253.0/24 and 10.252.252.0/24, if the range of IP addresses your remote clients are using is NOT on this list you need to add them.
If you don’t know what addresses they are supposed to be using, then issue a “show run ip local pool” command.
[box]
Petes-ASA#show run ip local pool
ip local pool IPSEC-VPN-DHCP-POOL 10.253.253.1-10.253.253.5
ip local pool SSL-VPN-DHCP-POOL 10.252.252.1-10.252.252.5
Petes-ASA#
[/box]
Again I’ve got a sensible naming policy – so we can see what my pools are for, to see what pools are being used for what, issue a “show run tunnel-group” command.
[box]
Petes-ASA# show run tunnel-group
tunnel-group IPSEC-VPN-GROUP type remote-access <<< Here's my IPSEC VPN's
tunnel-group IPSEC-VPN-GROUP general-attributes
address-pool IPSEC-VPN-DHCP-POOL <<< And here's my matching DHCP scope (IPSEC)
authentication-server-group PNL-KERBEROS
default-group-policy IPSEC-VPN-POLICY
tunnel-group IPSEC-VPN-GROUP ipsec-attributes
pre-shared-key *****
tunnel-group SSL-VPN-POLICY type remote-access <<< Here's my SSL VPN's
tunnel-group SSL-VPN-POLICY general-attributes
address-pool SSL-VPN-DHCP-POOL <<< And here's my matching DHCP scope (SSL)
authentication-server-group PNL-KERBEROS
default-group-policy SSL-VPN-GROUP-POLICY
tunnel-group SSL-VPN-POLICY webvpn-attributes
group-alias PNL enable
Petes-ASA#
[/box]
If any of yours are missing then change accordingly.
BUG (08/09/16)
Had this problem again recently, and after staying on the phone to TAC until 03:00, it turned out to be a bug in the SFR (FirePOWER service module) code. That was causing the firewall to silently drop the AnyConnect traffic. So debugs showed nothing, and packet captures were empty. Fixed by removing ‘sfr fail-open’ from the firewall and upgrading the code by re-imaging the SFR module.
Related Articles, References, Credits, or External Links
On a newly deployed FirePOWER service module I wanted to test connectivity and attempted to ping a public IP address. This is what happened;
> expert admin@Petes-SFR:~$ ping 8.8.8.8 ping: icmp open socket: Operation not permitted
My first though was, “Well you have to set a default gateway on the SFR when you set it up, so the firewall is probably blocking ICMP”. So I checked the default policy and indeed there was no ICMP inspection turned on.
UPDATE: The correct way to do this, (as pointed out below) is the use;
[box]system support ping {IP or Hostname}[/box]
-=-=-=- Back to the original post -=-=-=-
On checking the Firewall, (Note: The decrement-ttl-tcl isn’t really needed, I do that to make tracerts work).
[box]
Petes-FW(config)# policy-map global_policy
Petes-FW(config-pmap)# class inspection_default
Petes-FW(config-pmap-c)# inspect icmp
Petes-FW(config-pmap-c)# class class-default
Petes-FW(config-pmap-c)# set connection decrement-ttl
Petes-FW(config-pmap-c)# exit
Petes-FW(config-pmap)# exit
Petes-FW(config)# write mem
Building configuration...
Cryptochecksum: ae83c992 52a79538 b8370d6f 36125070
3120 bytes copied in 0.290 secs
[OK]
Petes-FW(config)#
[/box]
Unfortunately it still didn’t work!
Solution
Some forum searching yielded the same error on other flavours of Linux, the fix there was to change some permissions, so I gave that a try, and it worked.
[box]
admin@Petes-SFR:~$ sudo chmod u+s /bin/ping
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
Password:{Your-SFR-Admin-Password}
admin@Petes-SFR:~$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_req=1 ttl=52 time=12.8 ms
64 bytes from 8.8.8.8: icmp_req=2 ttl=52 time=13.7 ms
64 bytes from 8.8.8.8: icmp_req=3 ttl=52 time=12.6 ms
64 bytes from 8.8.8.8: icmp_req=4 ttl=52 time=13.2 ms
[/box]
Related Articles, References, Credits, or External Links
Warning: If your ASA is running version 8.3(4) or above you are going to have problems assigning public IP addresses from your allocated BT Range (jump to the bottom of the article for a resolution).
You have a pool of public IP addresses and you wish to allocate one of these IP addresses to your Cisco ASA Firewall. Note: This is for customers using BOTH ADSL and BT Infinity
Solution
For this procedure I was running an ASA5505 (Unlimited) with version 8.4(5). You will need to know the public IP address range allocated to you by BT (and the IP allocated to the router/hub).
Allocating a Public IP address to an Internal Client with the BT Business Hub
1. Log into the router, (the password initially is on the pull out plastic tab on top of the router). Set the IP to the one allocated to the router by BT (from the IP range they have given you). Note: The router actually gets a different IP address externally, this is normal, don’t panic.
2. Apply > Wait for the changes to apply.
3. Under business Network > Devices> You should see your device listed > Select it.
4. Assign the public IP as shown, you need to select the two radio buttons before the drop-down list of IP addresses will work > Apply.
5. Note: additionally if you are installing a firewall you might want to disable the Business Hubs internal firewall. Settings >Port Forwarding > Firewall > “Allow all traffic…” > Apply.
Problem with Cisco ASA (Now Resolved: See below)
My firewall (after a reload) picked up the correct IP address, but was unable to connect to the internet. My laptop (also connected to the BT Business Hub) connected fine to the internet (both with an allocated public address, and using the public address of the router). The ASA could not get out at all, nor could it ping the IP address of the Business Hub. The ASA showed as disconnected for a while, then disappeared from the ‘Devices’ tab, even though it continued to get the correct IP address leased to it from the Business Hub, this persisted after a reload of the firewall – so the hub COULD see it. I tried giving the ASA the correct IP address statically, I also locked the speed and duplex of the ethernet interface (in case it was simply an auto-negotiation error), this did not resolve the problem. BT told me they had no record of anyone having the same problem, but that they would take a note in case it came up again. Luckily the client had his old 2Wire router as soon as I plugged that in everything worked fine.
Update 210414 (and resolution)
Got an email from Nate Morris this week who had been working on this very problem, while debugging the ARP traffic he saw;
[box]
arp-in: request at external from 192.168.1.254 c0ac.54e4.d8d8 for 123.123.123.123
0000.0000.0000 arp-in: Arp packet received from 192.168.1.254 which is in different subnet
than the connected interface 123.123.123.123/255.255.255.248
[/box]
This pointed to a known problem with Cisco ASA introduced in version 8.3(4). Cisco identified this as bug CSCty95468 (Cisco CCO Login required to view). To resolve this problem you need to allow the ASA to populate its ARP table from a non connected subnet. To do this you need to issue an arp permit-nonconnected command.
[box]
User Access Verification Password:
Type help or '?' for a list of available commands.
Petes-ASA> enable
Password: ********
Petes-ASA# configure terminal
Petes-ASA(config)# arp permit-nonconnected
Petes-ASA(config)# exit
Petes-ASA# write mem
Building configuration... Cryptochecksum: 28790e0e 91da681e 7cf92e8a 85efb7ea 9449 bytes copied in 1.310 secs (9449 bytes/sec) [OK]
Petes-ASA#
[/box]
Update 260213
Got an Email from Andrew Joubert, to say that he had the same problem, and he was using the BT business hub via BT Infinity not ADSL.
Related Articles, References, Credits, or External Links
Original Article Written 26/02/13
Credit to: Nate Morris, for finding the resolution to the original problem.
Special thanks to Steve at BT, who rang me back on my mobile so I didn’t have wait in a queue, and then followed up afterwards to see what the outcome was, if I knew his surname I would publish it! He did a grand job, and does not get paid enough!
Also thanks to Chris at BT who pitched in and did as much as he could.
While trying to deploy Solarwinds to monitor a Juniper SRX failover cluster, we were having no joy connecting to the management interface of the secondary/standby firewall. The management (fxp0) interface on the primary (node0) firewall we could get to OK.
]
After jumping on the secondary firewall (via the console connection) we observed the following;
error: the routing subsystem is not running
Solution
As you can see (above) I couldn’t get the routing services started. And I soon found out, this is quite normal, the primary (active) firewall maintains the routing instance, the secondary firewall does not.
Well that fine but what about out Solarwinds box, what happens if the secondary firewall goes down? No one would know, and we also can’t take daily backups of its config.
To fix this problem you need to use the ‘backup-router’ command. This lets the appliance maintain some routes in the event that it is not hosting the live routing instance.
1. I’ll connect to to the primary firewall from this console session on the secondary firewall with the following command;
[box]request routing-engine login node0[/box]
2. Now I can add the backup-router routes, but assign them to the secondary (node1) firewall. Note: Where 192.168.100.1 is the next hop.
[box]
To get traffic back to the Solarwinds Management Server
set groups node1 system backup-router 192.168.100.1 destination 10.1.20.10/32
To get traffic back to the Cisco ACS Appliance
set groups node1 system backup-router 192.168.100.1 destination 10.1.20.10/32
[/box]
3. Don’t forget if the firewalls failover you will have the same problem (but the opposite way round), so I need the same to the primary node as well.
[box]
To get traffic back to the Solarwinds Management Server
set groups node0 system backup-router 192.168.100.1 destination 10.1.20.10/32
To get traffic back to the Cisco ACS Appliance
set groups node0 system backup-router 192.168.100.1 destination 10.1.20.10/32
[/box]
3. Save the config with a ‘commit’ command.
Related Articles, References, Credits, or External Links
Considering we use ICMP to test connectivity, the fact that it is not a stateful protocol can be a major pain! Last week one of my colleagues rang me up and said, “Can you jump on this firewall, I’ve got no comms, and I cant ping external IP addresses. I can ping the internet from the firewall and I can ping internal IP addresses form the firewall”.
Solution
1. Before we start, lets get the basics out of the way, does the client you are pinging from have a firewall turned on? Can you ping the inside interface of the firewall?
2. Pinging will never work unless you have ICMP inspection turned on on the firewall. See the following article.
3. At this point we troubleshoot as we would for any other traffic through the firewall. To do this we use packet-tracer, the syntax is slightly different for ICMP, than it is for TCP and UDP though. We need to specify an ICMP Type and an ICMP code, to make sure the traffic leaves the firewall we trace ICMP type 8 (echo), with ICMP code 0 (none). And ICMP type 0 (Echo-reply), with ICMP code 0 (none), for traffic inbound.
Test Outbound Ping
Petes-ASA# packet-tracer input inside icmp 192.168.1.1 8 0 4.2.2.2
Testing Inbound Ping(where 123.123.123.123 is the public IP you are mapped to)
Petes-ASA# packet-tracer input outside icmp 4.2.2.2 0 0 123.123.123.123
4. Make sure the client you are on is getting NATTED or PATTED through the firewall. Below we will assume my internal IP address is 192.168.1.1.
Note: If you have names enabled and 192.168.1.1 has a name, you will get no results! issue a no names command from configure terminal mode to check.
[box]
Petes-ASA# show xlate | incl 192.168.1.1
If this machine was being NATTED to another public IP address it would look like..
NAT from inside:192.168.1.1 to outside:123.123.123.124
If this machine was being PATTED to a public IP address it would look like..
ICMP PAT from inside:192.168.1.1/1 to outside:123.123.123.123/1 flags ri idle 0:00:07 timeout 0:00:30
[/box]
If it fails at this stage then check you network translation configuration on the firewall.
5. If all appears normal so far you can capture the traffic as it passes though the firewall, below I’m successfully capturing the ICMP traffic though the firewall.
