Tag Archives: software iSCSI

VMware ‘Disable DelayedAck’ Does Not Work?

Posted on by
Reply

KB ID 0001525

Problem

I’ve got a client that’s been having some performance issues with their VMs. Their storage vendor, (EMC) said that as a result of finding this in the logs;

[box]

B       02/28/19 09:50:53.953 scsitarg          117000e [INFO] System: iSCSI Logout Initiator Data: IP=192.168.200.161 Name=...-ec-21 Target Data: Port=2 Flags=0x00002002 Info=0x01200801
B       02/28/19 09:50:53.969 scsitarg          117000e [INFO] System: iSCSI Logout Initiator Data: IP=192.168.201.161 Name=...-ec-21 Target Data: Port=3 Flags=0x00002002 Info=0x01200801
B       02/28/19 09:51:16.413 Health              608fe [WARN] User: Host ESXi-01.petenetlive.com does not have any initiators logged into the storage system.
A       02/28/19 10:04:25.968 scsitarg          117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.200.161 Name=...-ec-21 Target Data: Port=2 Flags=0x00002002 Info=0x00000000 [Target]
B       02/28/19 10:04:26.034 scsitarg          117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.200.161 Name=...-ec-21 Target Data: Port=2 Flags=0x00002002 Info=0x00000000
A       02/28/19 10:04:31.996 scsitarg          117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.201.161 Name=...-ec-21 Target Data: Port=3 Flags=0x00002002 Info=0x00000000 [Target]
B       02/28/19 10:04:32.055 scsitarg          117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.201.161 Name=...-ec-21 Target Data: Port=3 Flags=0x00002002 Info=0x00000000
B       02/28/19 10:04:57.438 Health              608fc [INFO] User: Host ESXi-01.petenetlive.com is operating normally.
Host Host ESXi-01.petenetlive.com is accessing lun Datastore_3 as HLU 3, After the initiators for this host start logging in/logging,  unit attention update events will be logged as the paths to the luns have changed this is expected
2019/02/28-09:50:41.607527 ~~~~     7F3C92369703      std:TCD:   Unit Attention update from 0000001A to 0001030D for LUN 0x3.
2019/02/28-10:02:55.860669 ~~~~     7FE476E61702      std:TCD:   Unit Attention update from 00010149 to 00010157 for LUN 0x3.

[/box]

We should disable DelayedAck and they kindly gave me the VMware KB that outlined the procedure.

Solution

The procedure outlined (for VMware 6.x) is to put the host in maintenance mode, then edit the properties of the iSCSI controller(s), untick the DelayedAck options, reboot the Host, and everything will be peachy. However, even though (post reboot) everything looks good in the the vSphere Web console. If you look on the host you may find something like this;

[box]

vmkiscsid --dump-db | grep Delayed

[/box]

DelayedAck = ‘1’ means ENABLED, DelayedAck = ‘0’ means DISABLED

So half my iSCSI entries in the iSCSI database still have DelayedAck ENABLED?

Some Internet searching told me this was quite common, and that the best way to ‘fix‘ it was to, disable the iSCSI initiator, remove the iSCSI database, reboot and then setup iSCSI again;

[box]

cd /etc/vmware/vmkiscsid
esxcfg-swiscsi -d
rm -f vmkiscsid.db
reboot

[/box]

Which is fine IF YOU ARE USING A SOFTWARE iSCSI INITIATOR, I however was not, I had 2x dedicated hardware iSCSI HBAs on each host!

After many hours of messing about and trial and error, it became clear, I had to do things in a certain order, or DelayedAck would simply just be enabled whether I liked it or not. 🙁

Disable DelayedAck With Hardware iSCSI NICs / HBAs

MAKE SURE THE HOST IS IN MAINTENANCE MODE FIRST

Then take a note of your iSCSI setup, Port Groups, VMKernel Ports, and Physical NICs, you are going to delete the iSCSI database in a minute, and you will need to ‘rebind’ the VMKernel Ports and add the iSCSI targets back in again.

Manually remove your iSCSI target(s) for ALL the iSCSI NIC/HBA’s

Below if you re-run the command, vmkiscsid –dump-db | grep Delayed you will see there’s still some entries in the database with DelayedAck enabled! So unlike above (see example for software iSCSI) we are going to remove the iSCSI database, only here we don’t need to disable the software iSCSI initiator (because we are not using one!) Finally reboot the host.

[box]

cd /etc/vmware/vmkiscsid
rm -f vmkiscsid.db
reboot

[/box]

When the host is back online ADD in the Network Port Binding for the appropriate VMkernel adaptor.

Like so;

DON’T RESCAN THE CONTROLLER AS PROMPTED TO DO SO!

On the Advanced Settings of EACH hardware iSCSI NIC/HBA > Edit > UNTICK ‘DelayedAck’.

Double check they are both still unticked (I’ve seen them re-tick themselves for no discernible reason!) Then rescan the controller(s).

Target > Add.

Re-add the iSCSI target back in, (that you took note of above).

Select the Target > Advanced > Untick the DelayedAck option (Note: This time it’s not inherited). Repeat for any additional iSCSI targets.

When they are all added, rescan the storage controllers again.

Finally recheck all the database entries are set to DISABLED.

[box]

vmkiscsid --dump-db | grep Delayed

[/box]

Related Articles, References, Credits, or External Links

Thanks to Russell and Iain for their patience while I worked all that out!

vSphere ESX – Configure Buffalo Terastation 5000 as an iSCSI Target

Posted on by

KB ID 0000899 

Problem

This little NAS box is a cheap way of adding a large amount of storage. Below I’m going to configure it as an iSCSI target, then connect my ESX5 host to it.

I’m not setting up any CHAP authentication, but I’ll show you where it’s configured, if you want to deploy yours a little more securely.

Also it’s considered good practice to separate your storage network traffic, from your actual network traffic (either physically or via VLANs). Here I’m also NOT doing that.

Solution

Initial Configuration of the TS5000

If you connect either of the NAS box’s NICs to your network they will pick up an IP address via DHCP (You will see it in your DHCP leases).

1. Connect to the NAS with a web browser, the default password is ‘password’.

2. To change the default password: Enter Easy Admin mode > Reset Password > Follow the instructions.

3. Team / Trunk The NICs: You can have each NIC with its own IP address, but I prefer to aggregate them > Network > Port Trunking > Configure port trunking.

4. Select ‘Link 1’.

5. Select All > Assign.

6. It may take a few seconds.

Configure iSCSI

7. First enable iSCSI > Drives > iSCSI > Click the switch to enable.

8. Configure iSCSI

9. Create Volume.

10. Give the volume a name, description, and specify the volume size > OK.

Note: If you wanted to configure authentication select enabled, and set accordingly.

11. Enter the numbers as requested > OK.

12. This can take a couple of minutes also.

Configure ESX For iSCSI

13. Connect to either your ESX host or vCenter > Select a host > Configuration > Networking > Add Networking > Create a new vSwitch > Add a VMkernel port group (called iSCSI or something sensible) > Assign a free NIC, and give it an IP address on the same range as the NAS box.

14. Storage Adaptors > If you do not see any, select ‘Add’ and add in a software iSCSI Adapter.

15. Right click your iSCSI Adapter > Network configuration > Bind it to the port group you created in step 13.

16. Then on either the Dynamic or the Static discovery tab, enter the IP address of the NAS box.

17. Storage > Add Storage > Disk/LUN > Select the iSCSI storage > Follow the instructions.

18. Repeat the process on your remaining ESX hosts. (Note: You will only need to create the VMFS volume(s) for the first one).

Related Articles, References, Credits, or External Links

Cisco Small Business (SG500) Link Aggregation (LAG) With LACP