VMware: Change IOPS Limit From 1000 to 1

KB ID 0001532

Problem

I got asked to do this by a client this week, HP has requested that this be set for connections to their Storevirtual VSA that had been having some problems.

Solution

I followed the instructions and was at first confused because I could not see the settings that needed changing? That’s because this only applies if you have MULTIPATHING enabled and set to ‘Round Robin’. So if your storage does NOT look like below, (All paths Active I/O). then this procedure is not applicable.

So assuming you are using round robin multipathing, and, <ahem!> the storage vendor hasn’t just pulled a solution from a list of things that might work, rather than actually diagnosing the problem. Then you can see the current setting with the following command;

[box]

esxcli storage nmp device list

[/box]

Take note of the iSCSI storage names, below you can see they all start with naa.6000, you can also see the IOPS value is set to 1000.

To change the value use the following command (change the value in red to match yours);

[box]

for i in `esxcfg-scsidevs -c |awk '{print $1}' | grep naa.6000`; do esxcli storage nmp psp roundrobin deviceconfig set --type=iops --iops=1 --device=$i; done

[/box]

Then recheck, the new value should be ‘1’.

Related Articles, References, Credits, or External Links

Disable ATS Heartbeat

VMware ‘Disable DelayedAck’ Does Not Work?

 

Cisco ASA VPN to Cisco Router “MM_WAIT_MSG3”

KB ID 0001531

Problem

While migrating a VPN tunnel from an ASA 5520 firewall to a new 5516-X I got this problem. The other end was a Cisco router (2900). As soon as I swapped it over, it was stuck at MM_WAIT_MSG3, and phase 1 would not establish;

[box]

NUFC-ASA5516x(config-tunnel-ipsec)# show crypto isa

IKEv1 SAs:

   Active SA: 6
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 6

1   IKE Peer: 1.1.1.1
    Type    : user            Role    : responder
    Rekey   : no              State   : MM_WAIT_MSG3

[/box]

Debugs didn’t help much either;

[box]

Mar 25 2019 18:50:49: %ASA-7-713236: IP = 1.1.1.1, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 164
Mar 25 2019 18:50:49: %ASA-7-715047: IP = 1.1.1.1, processing SA payload
Mar 25 2019 18:50:49: %ASA-7-713906: IP = 1.1.1.1, Oakley proposal is acceptable
Mar 25 2019 18:50:49: %ASA-7-715047: IP = 1.1.1.1, processing VID payload
Mar 25 2019 18:50:49: %ASA-7-715049: IP = 1.1.1.1, Received NAT-Traversal RFC VID
Mar 25 2019 18:50:49: %ASA-7-715047: IP = 1.1.1.1, processing VID payload
Mar 25 2019 18:50:49: %ASA-7-715047: IP = 1.1.1.1, processing VID payload
Mar 25 2019 18:50:49: %ASA-7-715049: IP = 1.1.1.1, Received NAT-Traversal ver 03 VID
Mar 25 2019 18:50:49: %ASA-7-715047: IP = 1.1.1.1, processing VID payload
Mar 25 2019 18:50:49: %ASA-7-715049: IP = 1.1.1.1, Received NAT-Traversal ver 02 VID
Mar 25 2019 18:50:49: %ASA-7-715047: IP = 1.1.1.1, processing IKE SA payload
Mar 25 2019 18:50:49: %ASA-7-715028: IP = 1.1.1.1, IKE SA Proposal # 1, Transform # 1 acceptable  Matches global IKE entry # 8
Mar 25 2019 18:50:49: %ASA-7-715046: IP = 1.1.1.1, constructing ISAKMP SA payload
Mar 25 2019 18:50:49: %ASA-7-715046: IP = 1.1.1.1, constructing NAT-Traversal VID ver RFC payload
Mar 25 2019 18:50:49: %ASA-7-715046: IP = 1.1.1.1, constructing Fragmentation VID + extended capabilities payload
Mar 25 2019 18:50:49: %ASA-7-713236: IP = 1.1.1.1, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 128
Mar 25 2019 18:50:57: %ASA-7-713236: IP = 1.1.1.1, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 128
Mar 25 2019 18:50:59: %ASA-7-713906: IKE Receiver: Packet received on 2.2.2.2:500 from 1.1.1.1:500
Mar 25 2019 18:50:59: %ASA-5-713202: IP = 1.1.1.1, Duplicate first packet detected.  Ignoring packet.
Mar 25 2019 18:51:05: %ASA-7-713236: IP = 1.1.1.1, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 128
Mar 25 2019 18:51:21: %ASA-7-715065: IP = 1.1.1.1, IKE MM Responder FSM error history (struct &0x00007f4d2d293690)  , :  MM_DONE, EV_ERROR-->MM_WAIT_MSG3, EV_TIMEOUT-->MM_WAIT_MSG3, NullEvent-->MM_SND_MSG2, EV_SND_MSG-->MM_SND_MSG2, EV_START_TMR-->MM_SND_MSG2, EV_RESEND_MSG-->MM_WAIT_MSG3, EV_TIMEOUT-->MM_WAIT_MSG3, NullEvent
Mar 25 2019 18:51:21: %ASA-7-713906: IP = 1.1.1.1, IKE SA MM:5f3d6a94 terminating:  flags 0x01000002, refcnt 0, tuncnt 0
Mar 25 2019 18:51:21: %ASA-7-713906: IP = 1.1.1.1, sending delete/delete with reason message
Mar 25 2019 18:51:21: %ASA-6-713905: IP = 1.1.1.1, Warning: Ignoring IKE SA (src) without VM bit set

[/box]

Solution

Well, as you can tell from my Troubleshooting Phase 1 Cisco Site to Site (L2L) VPN Tunnels article MM_WAIT_MSG3 usually happens if something is blocking ISAKMP (UDP 500) in-between the peers. Or there’s a ‘bug’ that needs some newer or older code. The problem was not spotted by me, I was chatting to an ex colleague about it and he knew what it was straight away (annoyingly). The Router had an ACL on its outside interface that was NOT allowing the peer in to establish a VPN.

On the router take a look a the outside interface, and look for an access-list;

[box]

Remote-Router#show run interface GigabitEthernet 0/0
Building configuration...

Current configuration : 214 bytes
!
interface GigabitEthernet0/0
 bandwidth 200000
 ip address 1.1.1.1 255.255.255.248
 ip access-group outside-in in
 ip nat outside
 ip virtual-reassembly in
 duplex auto
 speed auto
 crypto map CMAP-1
end

Remote-Router#

[/box]

So it’s called outside-in what’s it doing?

[box]

Remote-Router#show access-list outside-in
Extended IP access list outside-in
    10 permit udp any eq domain any
    20 permit icmp any any echo-reply (956 matches)
    30 permit ip 123.123.123.64 0.0.0.15 any (127341882 matches)
    40 permit ip 123.123.123.128 0.0.0.127 any (572 matches)
    50 permit ip 222.222.222.96 0.0.0.15 any (4 matches)
    60 permit ip host 123.123.123.68 any
    70 permit udp host 222.222.222.76 eq ntp any
    80 permit udp host 222.222.222.204 eq ntp any
    90 permit udp host 222.222.222.232 eq ntp any
    100 permit icmp any any (1320 matches)
    110 permit ip 223.223.233.0 0.0.0.255 any

[/box]

Add in our peer IP address;

[box]

Remote-Router#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
Remote-Router(config)#ip access-list extended outside-in
Remote-Router(config-ext-nacl)# permit esp host 2.2.2.2 any
Remote-Router(config-ext-nacl)# permit udp host 2.2.2.2 any eq isakmp
Remote-Router(config-ext-nacl)# permit udp host 2.2.2.2 any eq non500-isakmp
Remote-Router(config-ext-nacl)#exit
Remote-Router(config)# 

[/box]

Let’s just make sure it’s there;

[box]

Remote-Router#show access-list outside-in
Extended IP access list outside-in
    10 permit udp any eq domain any
    20 permit icmp any any echo-reply (956 matches)
    30 permit ip 123.123.123.64 0.0.0.15 any (127341882 matches)
    40 permit ip 123.123.123.128 0.0.0.127 any (572 matches)
    50 permit ip 222.222.222.96 0.0.0.15 any (4 matches)
    60 permit ip host 123.123.123.68 any
    70 permit udp host 222.222.222.76 eq ntp any
    80 permit udp host 222.222.222.204 eq ntp any
    90 permit udp host 222.222.222.232 eq ntp any
    100 permit icmp any any (1320 matches)
    110 permit ip 223.223.233.0 0.0.0.255 any
    120 permit esp host 2.2.2.2 any
    130 permit udp host 2.2.2.2 any eq isakmp
    140 permit udp host 2.2.2.2 any eq non500-isakmp

[/box]

Don’t forget to save the changes with a ‘write mem’ command!

Related Articles, References, Credits, or External Links

Once again, thanks to SteveH for spotting, (in less than sixty seconds) what was wrong.

Cisco ASA: “Wrong Serial Number?”

KB ID 0001530

Problem

Cisco have done this for a while, the first time I saw it was years ago on a 5585, but all the NGFW models now have a ‘Serial Number” and a “Chassis Serial Number”. Normally you don’t care unless you need to log a TAC call online. So you issue a show version command, take a note of the serial number, and then it says, there’s no record of that serial number?

Solution

Just to be clear

SmartNets are registered to the Chassis Serial Number, this is NOT the serial number shown with a ‘show version‘ command.

Software (e.g. AnyConnect) is licensed to the Serial Number that IS shown with a ‘show version‘ command.

As a general rule, Cisco ASA chassis serial numbers start with JMX, and the serial numbers start with JAD.

How to Locate the Cisco ASA ‘Chassis Serial Number’

Well it’s printed on the chassis of course, but if it’s in a rack or a thousand miles away, that’s not much help! To get it remotely you use the ‘show inventory’ command;

[box]

Petes-ASA# show inventory
Name: "Chassis", DESCR: "ASA 5516-X with FirePOWER services, 8GE, AC, DES"
PID: ASA5516           , VID: V05     , SN: JMX1234ABCD

Name: "Storage Device 1", DESCR: "ASA 5516-X SSD"
PID: ASA5516-SSD       , VID: N/A     , SN: MSA21470XXX

Petes-ASA#

[/box]

How to Locate the Cisco ASA ‘Serial Number’

Same as with the old 5500 series firewalls, (and the PIX) use a show version command.

[box]

Petes-ASA# show version

Cisco Adaptive Security Appliance Software Version 9.8(2)24
Firepower Extensible Operating System Version 2.2(2.75)
Device Manager Version 7.8(2)151

Compiled on Thu 01-Mar-18 20:21 PST by builders
System image file is "disk0:/asa982-24-lfbff-k8.SPA"
Config file at boot was "startup-config"

Petes-ASA up 146 days 1 hour
failover cluster up 146 days 1 hour

Hardware:   ASA5516, 8192 MB RAM, CPU Atom C2000 series 2416 MHz, 1 CPU (8 cores)
Internal ATA Compact Flash, 8000MB
BIOS Flash M25P64 @ 0xfed01000, 16384KB

Encryption hardware device : Cisco ASA Crypto on-board accelerator (revision 0x1)
                             Number of accelerators: 1

 1: Ext: GigabitEthernet1/1  : address is 00a7.42e1.6ed6, irq 255
 2: Ext: GigabitEthernet1/2  : address is 00a7.42e1.6ed7, irq 255
 3: Ext: GigabitEthernet1/3  : address is 00a7.42e1.6ed8, irq 255
 4: Ext: GigabitEthernet1/4  : address is 00a7.42e1.6ed9, irq 255
 5: Ext: GigabitEthernet1/5  : address is 00a7.42e1.6eda, irq 255
 6: Ext: GigabitEthernet1/6  : address is 00a7.42e1.6edb, irq 255
 7: Ext: GigabitEthernet1/7  : address is 00a7.42e1.6edc, irq 255
 8: Ext: GigabitEthernet1/8  : address is 00a7.42e1.6edd, irq 255
 9: Int: Internal-Data1/1    : address is 00a7.42e1.6ed5, irq 255
10: Int: Internal-Data1/2    : address is 0000.0001.0002, irq 0
11: Int: Internal-Control1/1 : address is 0000.0001.0001, irq 0
12: Int: Internal-Data1/3    : address is 0000.0001.0003, irq 0
13: Ext: Management1/1       : address is 00a7.42e1.6ed5, irq 0
14: Int: Internal-Data1/4    : address is 0000.0100.0001, irq 0

Licensed features for this platform:
Maximum Physical Interfaces       : Unlimited      perpetual
Maximum VLANs                     : 150            perpetual
Inside Hosts                      : Unlimited      perpetual
Failover                          : Active/Active  perpetual
Encryption-DES                    : Enabled        perpetual
Encryption-3DES-AES               : Enabled        perpetual
Security Contexts                 : 2              perpetual
Carrier                           : Disabled       perpetual
AnyConnect Premium Peers          : 4              perpetual
AnyConnect Essentials             : Disabled       perpetual
Other VPN Peers                   : 300            perpetual
Total VPN Peers                   : 300            perpetual
AnyConnect for Mobile             : Disabled       perpetual
AnyConnect for Cisco VPN Phone    : Disabled       perpetual
Advanced Endpoint Assessment      : Disabled       perpetual
Shared License                    : Disabled       perpetual
Total TLS Proxy Sessions          : 1000           perpetual
Botnet Traffic Filter             : Disabled       perpetual
Cluster                           : Enabled        perpetual
Cluster Members                   : 2              perpetual
VPN Load Balancing                : Enabled        perpetual


Failover cluster licensed features for this platform:
Maximum Physical Interfaces       : Unlimited      perpetual
Maximum VLANs                     : 150            perpetual
Inside Hosts                      : Unlimited      perpetual
Failover                          : Active/Active  perpetual
Encryption-DES                    : Enabled        perpetual
Encryption-3DES-AES               : Enabled        perpetual
Security Contexts                 : 4              perpetual
Carrier                           : Disabled       perpetual
AnyConnect Premium Peers          : 8              perpetual
AnyConnect Essentials             : Disabled       perpetual
Other VPN Peers                   : 300            perpetual
Total VPN Peers                   : 300            perpetual
AnyConnect for Mobile             : Disabled       perpetual
AnyConnect for Cisco VPN Phone    : Disabled       perpetual
Advanced Endpoint Assessment      : Disabled       perpetual
Shared License                    : Disabled       perpetual
Total TLS Proxy Sessions          : 1000           perpetual
Botnet Traffic Filter             : Disabled       perpetual
Cluster                           : Enabled        perpetual
VPN Load Balancing                : Enabled        perpetual

The Running Activation Key feature: 2000 TLS Proxy sessions exceed the limit on the platform, reduced to 1000 TLS Proxy sessions.

Serial Number: JAD1234ABCD
Running Permanent Activation Key: 0x0037exxx 0x482ffyyy 0x04718yyy 0xaad48xxx 0x49343xxx
Configuration register is 0x1
Image type                : Release
Key Version               : A
Configuration last modified by PeteLong at 13:50:02.750 GMT Tue Mar 26 2019

Petes-ASA#

[/box]

Related Articles, References, Credits, or External Links

NA

Disable ATS Heartbeat

KB ID 0001529

Problem

After ESX 5.5 Update 2, VMware added ATS Heartbeat.Some vendors, (like HPE SureStore and VSA) recommend that this is disabled. I can’t find any info about whether it’s safe to do this in production, so to be on the safe side I placed the hosts in maintenance mode first.

Enter Maintenance Mode

Use the following command;

[box]

vim-cmd /hostsvc/maintenance_mode_enter

[/box]

Disable ATS Heartbeat

Use the following command to disable;

[box]

esxcli system settings advanced set -i 0 -o /VMFS3/UseATSForHBOnVMFS5

[/box]

Then confirm it worked with following command;

[box]

esxcli system settings advanced list -o /VMFS3/UseATSForHBOnVMFS5

[/box]

Confirm that INT Value is set to 0 (zero).

Exit Maintenance Mode

Use the following command;

[box]

vim-cmd /hostsvc/maintenance_mode_exit

[/box]

 

Related Articles, References, Credits, or External Links

NA

Exchange: Create a PFX Certificate and Import a Private Key

KB ID 0001528

Problem

On my test network I have an Exchange server (Exchange 2016). As it’s publicly available and connected to Office 365 it needs a digital certificate. Because its my test network, I don’t want to spend a fortune on a certificates, so I buy the cheapest one I can find. I replaced it last week with a 2 year certificate for about $5.00.

When the certificate came I had certificate-name.crt, and a CA bundle, but I could NOT import it into Exchange, (because the private key was missing). I had the private key, I downloaded it when I made the certificate request.

So I had the certificate and the private key, I needed to import the private key into my Exchange server, or create a PFX file that had the certificate and the private key in it, that I could import into Exchange.

Create a PFX File with OpenSSL

The simplest way to create a PFX, (if you are feeling lazy,) is to go here and let them do it for you. But I know I could do this with OpenSSL, being a mac user I already have OpenSSL, if you are a Windows user you can install OpenSSL for Windows and do the same thing.

Firstly place your private key file and certificate in a folder you can get at like so;

Then execute the following command;

[box]

openssl pkcs12 -export -out output-cert-name.pfx -inkey key-file-name.key -in input-cert-name.crt

[/box]

When prompted to do so, enter a password, (you will need to enter this on your Exchange server, so make a note of it!) You will then have your .PFX file.

Import PFX File into Exchange

I prefer to use PowerShell, (if you want to use the Exchange Admin Centre scroll down.) 

[box]

Import-ExchangeCertificate -Server Server-Name -FileName "\\Server-Name\Share-Name\Certificate-Name.pfx" -Password (ConvertTo-SecureString -String 'Your-Password' -AsPlainText -Force)

[/box]

Copy the certificate thumbprint (as shown) to the clipboard.

Then you need to associate the Exchange ‘Services’ with this new certificate;

[box]

Get-ExchangeCertificate -Thumbprint YOUR-CERT-THUMBPRINT | Enable-ExchangeCertificate -Services IIS,SMTP,IMAP,POP

[/box]

When prompted to do so enter ‘A’, (for replace all.)

You should see the correct certificate used now, like so;

Note: If you only see a blank page for OWA and Exchange Admin Center (ECP) after changing certificates, then see the following post;

Exchange – OWA and ECP Websites Blank After Logon

Import a PFX From Exchange Admin Center

Servers > Certificates > Select the appropriate Server > Ellipses > Import Exchange Certificate > Add the path to the PFX file, and its password > Next.

Add the server > Finish.

Now to enable the certificate for the appropriate Exchanges Services, select the cert > Edit > Services > Tick SMTP, IMAP, POP, and IIS > Save > OK.

 

Related Articles, References, Credits, or External Links

Exchange Certificate Import Error (reason: PrivateKeyMissing).

Exchange – Certificate Invalid ‘Revocation Check Failed’

Exchange: Blank Certificate Name

Exchange – OWA and ECP Websites Blank After Logon

Exchange: Can’t Delete a Database

KB ID 0001414

Problem

Every iteration of Exchange comes up with some new system/hidden mailbox type that stops me deleting mailbox databases!

[box]

This mailbox database contains one or more mailboxes, mailbox plans, archive mailboxes, public folder mailboxes or arbitration mailboxes, Audit mailboxes. To get a list of all mailboxes in this database, run the command Get-Mailbox -Database <Database ID>. To get a list of all mailbox plans in this database, run the command Get-MailboxPlan. To get a list of archive mailboxes in this database, run the command Get-Mailbox -Database <Database ID> -Archive. To get a list of all public folder mailboxes in this database, run the command Get-Mailbox -Database <Database ID> -PublicFolder. To get a list of all arbitration mailboxes in this database, run the command Get-Mailbox -Database <Database ID> -Arbitration. To get a list of all Audit mailboxes in this database, run the command Get-Mailbox -Database <Database ID> -AuditLog. To disable a non-arbitration mailbox so that you can delete the mailbox database, run the command Disable-Mailbox <Mailbox ID>. To disable an archive mailbox so you can delete the mailbox database, run the command Disable-Mailbox <Mailbox ID> -Archive. To disable a public folder mailbox so that you can delete the mailbox database, run the command Disable-Mailbox <Mailbox ID> -PublicFolder. To disable a Audit mailbox so that you can delete the mailbox database, run the command Get-Mailbox -AuditLog | Disable-Mailbox. Arbitration mailboxes should be moved to another server; to do this, run the command New-MoveRequest <parameters>. If this is the last server in the organization, run the command Disable-Mailbox <Mailbox ID> -Arbitration -DisableLastArbitrationMailboxAllowed to disable the arbitration mailbox. Mailbox plans should be moved to another server; to do this, run the command Set-MailboxPlan <MailboxPlan ID> -Database <Database ID>.

[/box]

 

Solution

OK, I’m assuming you don’t actually have any mailboxes in the database? The following will tell you;

[box]Get-Mailbox -Database “Database-Name“[/box]

If you are running Exchange 2016 you might have an AudiLog account;

[box]Get-Mailbox -Auditlog -Database “Database-Name“[/box]

Or a Monitoring Mailbox

[box]Get-Mailbox -Monitoring -Server “Server-Name“[/box]

For 2013 (and older) the likely culprits are Arbitration, Archive, or Discovery Search mailboxes, (the latter you need an extra command to see).

[box]Get-Mailbox -Auditlog -Database “Database-Name” -Arbitration

Get-Mailbox -Auditlog -Database “Database-Name” -Archive

Set-AdServerSettings -ViewEntireForest $true

Get-Mailbox -Database “Database-Name“[/box]

To move a Discovery Search Malbox

[box]Get-Mailbox DiscoverySearchMailbox* | New-MoveRequest -TargetDatabase “Target-Database“[/box]

Also Exchange 2013 or Newer may have one or more Public folder mailboxes;

[box]Get-Mailbox -PublicFolder | New-MoveRequest -TargetDatabase “Target-Database“[/box]

I Can’t Find Anything and it still Wont Let Me Delete the Datastore?

Well, there’s two things you can do;

1. On a Domain Controller, 0pen ADSIEdit.msc and Connect to ‘Configuration’. Navigate to Configuration > Services > Microsoft Exchange > {Organisation name} > Administrative Groups > {Administrative-Group-Name} > Databases  >Delete the database from here (BE CAREFUL CHECK TWICE, DELETE ONCE!). Then have a coffee refresh you datastore view and the offender will disappear.

2. With the database dismounted, move its .edb file to another folder, then mount the store, it will complain and ask if you want to mount and empty store > select ‘yes’ > You can then delete it.

 

Related Articles, References, Credits, or External Links

NA

Virtual SAN: Suppress ‘Datastore Usage on Disk’ Alarm

KB ID 0001527

Problem

While deploying a HPE StoreVirtual VSA this week. We noticed all the local Datastore were showing an ‘Alarm’. If you are unfamiliar with Virtual SANs, then you give all the LOCAL Storage to them, which then gets ‘aggregated’ and provided back to the host(s) as fault tolerant clustered storage, like so;

Which is great, but the VMware hosts just see that their LOCAL Datastore(s) are full, and they alarm;

Solution

Whilst in ‘Datastore View’: Create a ‘Local Datastore Folder‘ and add all you local datastore(s) to that, then create a ‘Virtual Datastore Folder’ and add all your virtual/clustered datastore(s) to that. Select the vCenter > Monitor > Issues > Alarm Definitions > Locate the ‘Datastore usage on disk’ alarm and Edit it. 

Untick: Enable this alarm.

You will recreate this alarm again below, so take a note all ALL its settings;

Now select your Virtual Datastore folder > Monitor > Issues > Alarm Definitions > Add > Recreate the alarm again, Note: I’ve called it ‘Virtual-Datastore-Usage-On-Disk‘.

Have a coffee, refresh the page, the alarms should have gone.

Related Articles, References, Credits, or External Links

NA

Deploying a ‘Nano’ Webserver (IIS)

KB ID 0001526

Problem

We’ve had Server Core for a while now, and I’ve never really seen it deployed in anger. Now we have Nano Server. You don’t install this like normal Windows Server distributions, i.e. it’s not an option when you run the install DVD. Originally you have to create the image with Powershell, but now you can use ‘Nano Server Image Builder‘.

Nano Server is a tiny distribution, and you simply add modules to it, or remove them as required. The thinking is, it’s a smaller attack platform, it’s simpler to patch, and can be managed remotely. If only Linus Torvalds had thought of this years ago? 🙂 

I needed a lightweight web server recently, so I thought I’d have a play with setting up a Nano server, and running IIS.

Solution

Download and install Nano Server Image Builder, (accept all the defaults). When you attempt to run it for the first time it will tell you, that it requires the Windows Assessment and Deployment Kit (ADK), which it will download and install for you, let it do so.

You only need;

  • Deployment tools
  • Windows Preinstallation Environment (Windows PE)

Have a copy of the Windows Server DVD Mounted, (Note: Here I’ve got Server 2016 Datacenter mounted).

Run the image builder > Create a new Nano Server Image > Next > Select the mounted Windows media drive > Next.

Accept the EULA > I’m exporting a VM image > Select the output directory > Choose a size for the virtual hard drive > You can specify another folder for the conversion files, if you are tight for space, I just used the same directory > Next > Next.

Add in the server ‘roles’ required, I only want Web Server (IIS) > Next > Add any additional drivers you want (If you are deploying on VMware add the VMware Tools drivers, see link at bottom of page)   > Next > Set the hostname, password and time zone of the Nano server > Next.

(Optionally) Join a domain > Next > Enter the networking details > Next > Create a basic Nano Server Image.

Next > (Oooh look, there’s the PowerShell! Might be worth copying that out for next time!) > Close.

Now you can import the virtual machine onto your favourite Hyper-Visor, (it’s obviously in Microsoft .VHD format), so you will need to convert it for VMware or open stack or whatever. I just uploaded a simple web page to the inetpub\wwwroot directory to test;

Related Articles, References, Credits, or External Links

Manually Extracting VMware Tools Drivers

VMware ‘Disable DelayedAck’ Does Not Work?

KB ID 0001525

Problem

I’ve got a client that’s been having some performance issues with their VMs. Their storage vendor, (EMC) said that as a result of finding this in the logs;

[box]

B       02/28/19 09:50:53.953 scsitarg          117000e [INFO] System: iSCSI Logout Initiator Data: IP=192.168.200.161 Name=...-ec-21 Target Data: Port=2 Flags=0x00002002 Info=0x01200801
B       02/28/19 09:50:53.969 scsitarg          117000e [INFO] System: iSCSI Logout Initiator Data: IP=192.168.201.161 Name=...-ec-21 Target Data: Port=3 Flags=0x00002002 Info=0x01200801
B       02/28/19 09:51:16.413 Health              608fe [WARN] User: Host ESXi-01.petenetlive.com does not have any initiators logged into the storage system.
A       02/28/19 10:04:25.968 scsitarg          117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.200.161 Name=...-ec-21 Target Data: Port=2 Flags=0x00002002 Info=0x00000000 [Target]
B       02/28/19 10:04:26.034 scsitarg          117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.200.161 Name=...-ec-21 Target Data: Port=2 Flags=0x00002002 Info=0x00000000
A       02/28/19 10:04:31.996 scsitarg          117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.201.161 Name=...-ec-21 Target Data: Port=3 Flags=0x00002002 Info=0x00000000 [Target]
B       02/28/19 10:04:32.055 scsitarg          117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.201.161 Name=...-ec-21 Target Data: Port=3 Flags=0x00002002 Info=0x00000000
B       02/28/19 10:04:57.438 Health              608fc [INFO] User: Host ESXi-01.petenetlive.com is operating normally.
Host Host ESXi-01.petenetlive.com is accessing lun Datastore_3 as HLU 3, After the initiators for this host start logging in/logging,  unit attention update events will be logged as the paths to the luns have changed this is expected
2019/02/28-09:50:41.607527 ~~~~     7F3C92369703      std:TCD:   Unit Attention update from 0000001A to 0001030D for LUN 0x3.
2019/02/28-10:02:55.860669 ~~~~     7FE476E61702      std:TCD:   Unit Attention update from 00010149 to 00010157 for LUN 0x3.

[/box]

We should disable DelayedAck and they kindly gave me the VMware KB that outlined the procedure.

Solution

The procedure outlined (for VMware 6.x) is to put the host in maintenance mode, then edit the properties of the iSCSI controller(s), untick the DelayedAck options, reboot the Host, and everything will be peachy. However, even though (post reboot) everything looks good in the the vSphere Web console. If you look on the host you may find something like this;

[box]

vmkiscsid --dump-db | grep Delayed

[/box]

DelayedAck = ‘1’ means ENABLED, DelayedAck = ‘0’ means DISABLED

So half my iSCSI entries in the iSCSI database still have DelayedAck ENABLED?

Some Internet searching told me this was quite common, and that the best way to ‘fix‘ it was to, disable the iSCSI initiator, remove the iSCSI database, reboot and then setup iSCSI again;

[box]

cd /etc/vmware/vmkiscsid
esxcfg-swiscsi -d
rm -f vmkiscsid.db
reboot

[/box]

Which is fine IF YOU ARE USING A SOFTWARE iSCSI INITIATOR, I however was not, I had 2x dedicated hardware iSCSI HBAs on each host!

After many hours of messing about and trial and error, it became clear, I had to do things in a certain order, or DelayedAck would simply just be enabled whether I liked it or not. 🙁

Disable DelayedAck With Hardware iSCSI NICs / HBAs

MAKE SURE THE HOST IS IN MAINTENANCE MODE FIRST

Then take a note of your iSCSI setup, Port Groups, VMKernel Ports, and Physical NICs, you are going to delete the iSCSI database in a minute, and you will need to ‘rebind’ the VMKernel Ports and add the iSCSI targets back in again.

Manually remove your iSCSI target(s) for ALL the iSCSI NIC/HBA’s

Below if you re-run the command, vmkiscsid –dump-db | grep Delayed you will see there’s still some entries in the database with DelayedAck enabled! So unlike above (see example for software iSCSI) we are going to remove the iSCSI database, only here we don’t need to disable the software iSCSI initiator (because we are not using one!) Finally reboot the host.

[box]

cd /etc/vmware/vmkiscsid
rm -f vmkiscsid.db
reboot

[/box]

When the host is back online ADD in the Network Port Binding for the appropriate VMkernel adaptor.

Like so;

DON’T RESCAN THE CONTROLLER AS PROMPTED TO DO SO!

On the Advanced Settings of EACH hardware iSCSI NIC/HBA > Edit > UNTICK ‘DelayedAck’.

Double check they are both still unticked (I’ve seen them re-tick themselves for no discernible reason!) Then rescan the controller(s).

Target > Add.

Re-add the iSCSI target back in, (that you took note of above).

Select the Target > Advanced > Untick the DelayedAck option (Note: This time it’s not inherited). Repeat for any additional iSCSI targets.

When they are all added, rescan the storage controllers again.

Finally recheck all the database entries are set to DISABLED.

[box]

vmkiscsid --dump-db | grep Delayed

[/box]

Related Articles, References, Credits, or External Links

Thanks to Russell and Iain for their patience while I worked all that out!

Windows: Deploy and Configure DNSSEC

KB ID 0001524

Problem

Like many protocols and solutions, DNS was designed in a time when we all trusted each other, and a ‘hacker’ was some sort of farm equipment. To ensure when you get a response form a DNS server it is coming from where you expect it to, and you can trust it. We use DNSSEC. Basically it signs the DNS record with a digital cypher. 

The client can then validate the DNS record when it receives it and it knows its not been ‘tampered’ with.

Solution

DNSSEC – Digitally Sign a DNS Zone

From DN Manager, locate the one you wish to secure with DNSSEC > Right Click > DNSSEC > Sign the Zone.

Next > Accept the default ‘Customise zone signing parameters’ > Next. >The DNS Server {Server-Name} is the Key Master.

Note: You will only have one Key Master, as this is the first DNS DNSSEC server you will configure, it selects the local host by default.

Next > Add > Accept tech defaults > OK.

Note: KSK (Key Signing Key) is the MASTER public/private key pair that Weill be used to sign the ZSK (Zone Signing Key).

Next > Next > Add > Again accept the defaults > OK.

Note: ZSK (Zone Signing key) is the public/private key pair that will be used to sign and produce RRSIG (Resource Record Signatures), These will be used to sign the each of the records in an RRSET (Resource Record Set).

Next > Use NSEC3 > Next > Select ‘Enable the distribution of trust anchors for this zone’ > Next

Note: NSEC3 (and NSEC which is older and less secure), is a method of digitally signing a response for a record that does NOT EXIST in a zone.

Note: Trust Anchor This is basically a DNS record that says ‘Trust me, and everything above me”.

Next > Next > Finish.

Thats the DNS Zone signed. Now you need to tell your clients they need to ‘Validate’ their DNS requests this is simply done with a Group Policy.

Deploy a DNSSEC Name Resolution Policy

Create a GPO object (or edit an existing one), and link it to the location where your ‘Computers’ are (or simply the root of the domain).

Navigate to;

[box]

Computer Configuration > Policies > Window Settings > Name Resolution Policy

[/box]

After ‘Suffix’ enter your domain name > Tick ‘Enable DNSSEC in this rule’ > Tick ‘Require  DNS clients to check…’ > Create > Scroll Down.

Ensure it’s been added to the Name Resolution Policy Table > Apply.

Then wait or Force Group Policy.

Testing & Troubleshooting DNSSEC

Ensure clients have the correct Name Resolution Policy Table, with the following PowerShell Command;

[box]

Get-DnsClientNrptPolicy

[/box]

To check the ‘Trust Anchors’ and ‘Trust Points”.

[box]

Get-DnsServerTrustAnchor -Name Test.net

Get-DnsServerTrustPoint

[/box]

And to ensure that records are actually getting signed, use the following syntax;

[box]

Resolve-DnsName -Name lan-host -Type A -Server lan-dc-2016 -DnsSecOK

[/box]

Related Articles, References, Credits, or External Links

NA