Mac OSX – Testing Packet Fragmentation Over VPN

KB ID 0001204 

Problem

Many moons ago I wrote a post about a problem where I had no RDP over a VPN connection, and all the hoops I jumped though to troubleshoot and fix the problem.

Today I had a similar problem, I was connected to a client via Cisco AnyConnect, and I had hair-pinned that traffic, from the client site, over an IPSEC VPN to their servers in the Data Center. Pings were successful, but not RDP. 

To be honest this affects various types of TCP traffic, but it only ever seems to trip me up with RDP! In the past I used to ping and set the traffic to ‘not fragment’ and manually set the packet size, then I increased/decreased the packet size until I found the optimal MTU size like this;

But these are Windows options, how can I do the same thing on my Mac?

Solution

Well things on the Mac are even easier! It will even find the value for you, (in a manner of sorts). You set the minimum packet size, and the maximum packet size, and ask it to increment each ping packet by one. Below I’ve narrowed the scope to fit it on one screenshot;

Syntax;

ping -g {lowest-value} -G {highest-value} -h 1 -D {IP-address}

As you can see above, I started at 1370 and the last ‘working’ ping was sequence 8, so the last working value was 1386

Related Articles, References, Credits, or External Links

NA

VMware VI Client Error ‘Call “ServiceInstance.RetrieveContent” for object “ServiceInstance” on Server “IP-Address” failed’

KB ID 0000870 

Problem

This is a pretty generic error. It basically means “I cant connect to what you are asking me to connect to, on TCP Port 443 (https)”.

Solution

Internet searching for this error is very frustrating, everyone who was posting this error was seeing it because, instead of putting the IP address or name in the box (that actually tells you to put in the IP address or name (see image above)). If you put in https://{Name or IP Address}, you will see this error. However this was NOT MY PROBLEM.

This is happening because there is no communication between you and the ESX/vCenter you are trying to connect to. The first thing you need to do is see if HTTPS is open. On the affected machine open a web browser and point it to the same target and make sure you see the web console of the ESX/vCenter server. If you can’t see this, check firewalls (and proxies) and make sure HTTPS is not getting blocked.

In my case I could see this but it still did not work! Then I was reminded we have had strange comms problems on this site before, which I have documented here. Sure enough, when I dropped the MTU on the server I was trying to connect from (which was over a site to site VPN tunnel). It started to work fine.

Related Articles, References, Credits, or External Links

NA

Cannot Remote Desktop over VPN connection

KB ID 0000845 

Problem

This one had me well and truly stumped! The client has two sites and from their remote site they could not open a remote Desktop connection to a server at the main site.

RDP Stuck at Securing remote connection.

At first, because the client had SBS at their main site I assumed this was the problem, but sadly it was not.

Solution

The following process goes through the steps taken to identify and rectify the problem.

1. Firstly, I’m assuming you can ‘ping’ the target server both by name and IP address, if you can’t do this, then read no further, you have a communication problem, fix that first!

2. Check that RDP (TCP Port 3389) is open by attempting to Telnet to that port on the destination server.

You may receive the following error;

Windows – ‘Telnet’ is not recognized as an internal or external command

If you simply see a ‘cursor’ then the port is open, if not it will give you an error. (If that is the case then you need to look at comms to make sure TCP Port 3389 is not being blocked, either by a hardware firewall/router, or a software firewall on either of the machines.)

3. Check no third party security software is blocking RDP, by issuing the following command;

[box]
fltmc[/box]

This indicates the machine I’m on is running, “Trend Micro’.

4. Try disabling the security software to see if that rectifies the problem,

After much hand wringing, and a few days of rebuilding firewall VPNs, patching servers, and installing hot-fixes, I admitted defeat and got Microsoft on the phone.

5. The fist thing they found, was if they attempted to open a UNC path to the destinations server IP address it worked.

6. BUT If they did the same to the server name it failed.

Error: The specified network name is no longer valid

7. Normally this is an indication that the secure channel between this machine, and the target machine is broken. Normally this can be fixed with the following commands;

[box]

net stop KDC

klist purge

netdom resetpwd /server:{IP address of domain controller}/userd:{your-domain-name}administrator /passwordd:*

Then supply the domain administrators password

net start KDC

[/box]

However this did not fix our problem, but indicated that it was not just RDP that was failing. Both the machine we were using, and the destination machine were domain controllers, so domain replication was checked and the following was found;

Event ID 1865

Log Name: Directory Service
Source: Microsoft-Windows-ActiveDirectory_DomainService
Event ID: 1311
Task Category: Knowledge Consistency Checker
Level: Error
Keywords: Classic
User: ANONYMOUS LOGON
Computer: your-server-your-domain.com
Description:
The Knowledge Consistency Checker (KCC) has detected problems with the following 
directory partition. 

Directory partition: CN=Configuration,DC=your-domain,DC=com There is insufficient site connectivity information for the KCC to create a spanning tree replication topology. Or, one or more directory servers with this directory partition are unable to replicate the directory partition information. This is probably due to inaccessible directory servers. User Action Perform one of the following actions: – Publish sufficient site connectivity information so that the KCC can determine a route by which this directory partition can reach this site. This is the preferred option. – Add a Connection object to a directory service that contains the directory partition in this site from a directory service that contains the same directory partition in another site.

Event ID 1311

Log Name: Directory Service
Source: Microsoft-Windows-ActiveDirectory_DomainService
Event ID: 1566
Task Category: Knowledge Consistency Checker
Level: Warning
Keywords: Classic
User: ANONYMOUS LOGON
Computer: your-server-your-domain.com
Description:
All directory servers in the following site that can replicate the directory partition 
over this transport are currently unavailable. 

Event ID 1566

Log Name: Directory Service
Source: Microsoft-Windows-ActiveDirectory_DomainService
Event ID: 1865
Task Category: Knowledge Consistency Checker
Level: Warning
Keywords: Classic
User: ANONYMOUS LOGON
Computer: your-server-your-domain.com
Description:
The Knowledge Consistency Checker (KCC) was unable to form a complete spanning tree network topology. As a result, the following list of sites cannot be reached from the local site. 
Sites:
CN=Your-OU,CN=Sites,CN=Configuration,DC=your-domain,DC=com 

8. So we DO have a communications problem, some things work others do not! Let’s make sure our traffic is not getting fragmented, you would expect a packet of 1500 bytes to be able to get though, ours did not, using trial and error Microsoft ascertained that 1320 was the highest we could get though without error.

[box]

ping -f -l {packet size}

[/box]

Note: To get the figure exactly right, you need to keep decreasing the packet size by 1, then when you have found the largest size permissible, you need to add 28 to it (for the overhead of the IP Header).

9. So the MTU was ‘locked’ at BOTH ENDS (source machine and destination server). To do so, Windows Key+R > regedit > Navigate to;

[box]

HKEY_LOCAL_MACHINE > System > CurrentControlSet > Services > Tcpip > Parameters > Interfaces

[/box]

Note: There may be many ‘keys’ here, check each one in turn, to find the one that equates to the IP address on your machine, (the one you are working on).

When you have located the correct key, create a new DWORD (32 bit) value (or edit one if it exists) set the DECIMAL value to the same size that you could get though without error in step 8.

10. Reboot the machines and try again.

Related Articles, References, Credits, or External Links

Special thanks and credit to Harprit Singh at Microsoft, for his hard work and outstanding support.

Cisco Simple GRE Tunnels (With IPSEC)

KB ID 0000951

Problem

I’ve spent years setting up VPN tunnels between firewalls. The only time I’ve ever dealt with GRE is for letting VPN client software though firewalls. GRE’s job is to ‘encapsulate’ other protocols and transport those protocols inside a virtual point to point link. Below is the topology, I’m going to use.

The tunnel will run form Router R1 to Router R3, once complete I should be able to ping Host2 from Host1.

Solution

Configure Router R1 for GRE

1. Create and configure a tunnel interface on the R1 Router. It will need an IP address, (here I’m using 10.0.0.1/30). Then you need to specify the source and destination of the GRE tunnel. Finally I’ve changed some MTU settings because typically MTU’s are set to 1500 and GRE adds an overhead, I’m dropping the MTU to 1400 and setting the maximum segment size to 1360.

[box]

R1#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)#interface Tunnel0
*Mar 1 00:01:27.747: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to down
R1(config-if)#ip address 10.0.0.1 255.255.255.252
R1(config-if)#ip mtu 1400
R1(config-if)#ip tcp adjust-mss 1360
R1(config-if)#tunnel source 1.1.1.1
R1(config-if)#tunnel destination 2.2.2.1
R1(config-if)#exit

[/box]

2. Then we need to add a static route to the router’s routing table so it knows to use that tunnel for traffic destined for the 192.168.2.0/24 network.

[box]

R1(config)#ip route 192.168.2.0 255.255.255.0 Tunnel0

[/box]

Configure Router R3 for GRE

3. This is simply a mirror image, of the configuration you carried our on router R1.

[box]

R3#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R3(config)#interface Tunnel0
*Mar 1 00:01:30.747: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to down
R3(config-if)#ip address 10.0.0.1 255.255.255.252
R3(config-if)#ip mtu 1400
R3(config-if)#ip tcp adjust-mss 1360
R3(config-if)#tunnel source 2.2.2.1
R3(config-if)#tunnel destination 1.1.1.1
R3(config-if)#exit
R3(config)#ip route 192.168.1.0 255.255.255.0 Tunnel0
[/box]

Verify GRE Tunnel

 

4. Use the following command to check the status of the GRE tunnel.

[box]

R1# show interface tunnel 0
Tunnel0 is up, line protocol is up
  Hardware is Tunnel
  Internet address is 10.0.0.1/30
  MTU 1514 bytes, BW 9 Kbit/sec, DLY 500000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation TUNNEL, loopback not set
  Keepalive not set
  Tunnel source 1.1.1.1, destination 2.2.2.1
  Tunnel protocol/transport GRE/IP
    Key disabled, sequencing disabled
    Checksumming of packets disabled
  Tunnel TTL 255
  Fast tunneling enabled
  Tunnel transmit bandwidth 8000 (kbps)
  Tunnel receive bandwidth 8000 (kbps)
  Last input never, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/0 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     0 packets output, 0 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out

[/box]

5. Then make sure that traffic passes over the tunnel.

[box]

R1#ping 192.168.2.10

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.2.10, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 24/64/88 ms
R1#

[/box]

Securing the Tunnel with IPsec

6. Our traffic is now going where we want it to, and it’s encapsulated, but it’s still being ‘sent in clear’ if traffic is intercepted ‘in flight’ it can be read. So we need to secure that traffic by encrypting it. First Job is to create an ISAKMP policy that will establish ‘phase-1’ of our secure tunnel. I’m using AES, with Diffie Hellman group 2, and SHA hashing. Ive specified that I will be using a pre-shared-key so that’s been created with the last command, and is assigned to the IP of the ‘other end’ of the VPN tunnel.

[box]

R1#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)#crypto isakmp policy 10
R1(config-isakmp)#authentication pre-share
R1(config-isakmp)#encryption aes
R1(config-isakmp)#group 2
R1(config-isakmp)#hash sha
R1(config-isakmp)#exit
R1(config)#crypto isakmp key 0 Sh@reds3cret address 2.2.2.1

[/box]

7. Phase 2 of our tunnel (IPsec) is encrypted and hashed with a ‘transform set’ again I’m using AES and SHA, then I create a profile that uses my transform set.

[box]

R3(config)#crypto ipsec transform-set TFS-PNL esp-aes esp-sha-hmac
R3(cfg-crypto-trans)#exit
R1(config)#crypto ipsec profile PF-PNL
R1(ipsec-profile)#set transform-set TFS-PNL
R1(ipsec-profile)#exit

[/box]

8. The last job is to apply the profile I created above, to our GRE tunnel interface.

[box]

R3(config)#interface tun0
R3(config-if)#tunnel mode ipsec ipv4
R3(config-if)#tunnel protection ipsec profile PF-PNL
*Mar 1 00:20:32.271: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to down
R3(config-if)#
*Mar 1 00:20:33.175: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
R3(config-if)#
*Mar 1 00:20:33.699: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up
R3(config-if)#exit
R3(config)#
[/box]

9. Again configure router 3 as a mirror image.

[box]

R3#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R3(config)#crypto isakmp policy 10
R3(config-isakmp)#authentication pre-share
R3(config-isakmp)#encryption aes
R3(config-isakmp)#group 2
R3(config-isakmp)#hash sha
R3(config-isakmp)#exit
R3(config)#crypto isakmp key 0 Sh@reds3cret address 1.1.1.1
R3(config)#crypto ipsec transform-set TFS-PNL esp-aes esp-sha-hmac
R3(cfg-crypto-trans)#exit
R3(config)#crypto ipsec profile PF-PNL
R3(ipsec-profile)#set transform-set TFS-PNL
R3(ipsec-profile)#exit
R3(config)#interface tun0
R3(config-if)#tunnel mode ipsec ipv4
R3(config-if)#tunnel protection ipsec profile PF-PNL
R3(config-if)#
*Mar 1 00:25:32.271: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to down
R3(config-if)#
*Mar 1 00:25:33.175: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
R3(config-if)#
*Mar 1 00:25:33.699: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up
R3(config-if)#exit
R3(config)#
[/box]

Cisco IOS Verify IPsec VPN Tunnel Is Up

 

Note: To bring up the tunnel simply send some traffic over it by pinging something on the other side of the tunnel. If you get a reply then the tunnel is up! But to check it status firstly make sure phase 1 has established.

[box]

R3#show crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
1.1.1.1         2.2.2.1         QM_IDLE           1001    0 ACTIVE

IPv6 Crypto ISAKMP SA

R3#

[/box]

QM_IDLE means that phase 1 has established (in Quick Mode), and is in an idle state (this is what you want to see, if you see any other state message you may need to start debugging things).

Once you know phase 1 is established you need to check phase 2.

[box]

R3#show crypto ipsec sa
interface: Tunnel0
    Crypto map tag: Tunnel0-head-0, local addr 2.2.2.1

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
   remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
   current_peer 1.1.1.1 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 0

     local crypto endpt.: 2.2.2.1, remote crypto endpt.: 1.1.1.1
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/1
     current outbound spi: 0x3AA3F6B0(983824048)

     inbound esp sas:
      spi: 0x5C5C5EF1(1549557489)
        transform: esp-aes esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 1, flow_id: SW:1, crypto map: Tunnel0-head-0
        sa timing: remaining key lifetime (k/sec): (4559832/3506)
        IV size: 16 bytes
        replay detection support: Y
        Status: ACTIVE

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:
      spi: 0x3AA3F6B0(983824048)
        transform: esp-aes esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 2, flow_id: SW:2, crypto map: Tunnel0-head-0
        sa timing: remaining key lifetime (k/sec): (4559832/3506)
        IV size: 16 bytes
        replay detection support: Y
        Status: ACTIVE

     outbound ah sas:

     outbound pcp sas:

[/box]

 

Related Articles, References, Credits, or External Links

NA

Cisco ASA 5500 – Configuring PPPoE

KB ID 0000831 

Problem

Until very recently I’d never had to configure PPPoE. Most of my clients in that sort of connection speed range have ADSL with a router provided by their ISP. A Router that connects via PPPoA usually.

Here in the UK the main ISP’s (BT and Virgin) are busy rolling out FTTC connections that terminate with a ‘modem’ that presents an RJ45 socket.

So without the need for a router, you can get the ASA to perform the authentication (supply the username and password via PPPoE) and either use a static IP address, or obtain its IP via DHCP.

Solution

1. Before you attempt to configure the connection, you will need the following from your ISP;

  • Username
  • Password
  • IP Details (If you have bought static IP’s you will need the range of IP addresses and the IP address to use as the firewalls default route (default gateway). Some ISP’s will expect you to configure DHCP and will make sure you always get the same IP.
  • Authentication method: CHAP, MSCHAP, or PAP (If your ISP acts confused when you ask, it’s probably CHAP).

2. Connect to the ASA > Go to enable mode > Go to configuration mode > Create a ‘vpdn’ group, (here I’ve called it PNL-DIALER-GROUP) > Set the authentication method, (here I’m using CHAP).

[box]

User Access Verification

Password:
Type help or '?' for a list of available commands.
PetesASA> enable
Password: ********
PetesASA# configure terminal
PetesASA(config)# vpdn group PNL-DIALER-GROUP request dialout pppoe
PetesASA(config)# vpdn group PNL-DIALER-GROUP ppp authentication chap

[/box]

3. Supply your username and password. (The store-local command puts the details in a protected area of flash memory).

[box]

PetesASA(config)# vpdn group PNL-DIALER-GROUP localname username1234-fttc@ispname.net
PetesASA(config)# vpdn username username1234-fttc@ispname.net password Password12345 store-local

[/box]

4. Apply your vpdn group to the interface (or VLAN if your using an ASA5505), then configure the static IP address or DHCP;

[box]

Command for an ASA 5505 (By Default VLAN 2 will be the outside VLAN).

PetesASA(config)# Interface vlan2
PetesASA(config-if)#

Command for an ASA 5510 (or greater) (By Default Ethernet0/0 will be the outside interface).

PetesASA(config)# Ethernet0/0
PetesASA(config-if)#

The rest of the configuration is the same for all models

PetesASA(config-if)# pppoe client vpdn group PNL-DIALER-GROUP

If you have been given a static IP address and default route

PetesASA(config-if)# ip address 123.123.123.123 255.255.255.248 pppoe
PetesASA(config-if)# route outside 0.0.0.0 0.0.0.0 123.123.123.124

If you are getting you IP address via DHCP

PetesASA(config-if)# ip address pppoe setroute

[/box]

5. Save your changes.

[box]

PetesASA(config-if)# exit
PetesASA(config)# exit
PetesASA# write mem
Building configuration…
Cryptochecksum: 79745c0a 509726e5 b2c66028 021fdc7d

7424 bytes copied in 1.710 secs (7424 bytes/sec)
[OK]
PetesASA#

[/box]

Cisco ASA Debugging PPPoE

You can use the following command/syntax to troubleshoot.

[box]

PetesASA# show ip address outside pppoe
PetesASA# [no] debug pppoe {event | error | packet}
PetesASA# show vpdn session [l2tp | pppoe] [id sess_id | packets | state | window]
PetesASA# show vpdn

[/box]

Configure PPPoE via ASDM

1. Connect to the ASDM > Configuration > Device Setup > Interfaces > Ethernet0/0 > Edit > General Tab > Use PPPoE.

2. Enter VPDN Group name username and password.

3. Set the PPP Authentication method PAP / CHAP or MSCHAP.

4. Tick store username and password in flash > IP Address and Route Settings
Apply > OK.

5. File > Save Running Configuration to flash.

Related Articles, References, Credits, or External Links

NA