KB ID 0001292 

Problem

For the first time in ages I’ve been doing a VMware upgrade this week, a client had an MSA P2000 G3 and two G8 DL380 servers running vSphere 5.5. I put in a new 6.5  VCSA, built some new G9 DL380 servers,  I noticed that the SAN was presenting five storage LUNs but the new ESX 6.5 servers could only see three of them?

Strangely when I selected the SAS storage controllers they could see all 5 storage LUNs, but the datastores refused to appear.

Solution

I checked that the SAN was not masking the LUN’s (it wasn’t, the default was read/write for everything). I connected to the  console and proved the storage could be seen.

[box]

[root@ESX1:~] esxcli storage core path list
------output removed for the sake of brevity------

sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014dfce99cd2d5401000000
   UID: sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014dfce99cd2d5401000000
   Runtime Name: vmhba3:C1:T1:L4
   Device: naa.600c0ff00014dfce99cd2d5401000000
   Device Display Name: HP Serial Attached SCSI Disk (naa.600c0ff00014dfce99cd2d5401000000)
   Adapter: vmhba3
   Channel: 1
   Target: 1
   LUN: 4 <-- First missing LUN
   Plugin: NMP
   State: active
   Transport: sas
   Adapter Identifier: sas.50014380388d8480
   Target Identifier: sas.d0b8d32406430000
   Adapter Transport Details: 50014380388d8480
   Target Transport Details: d0b8d32406430000
   Maximum IO Size: 4194304

sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014ddb44c57ac5401000000
   UID: sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014ddb44c57ac5401000000
   Runtime Name: vmhba3:C1:T1:L5
   Device: naa.600c0ff00014ddb44c57ac5401000000
   Device Display Name: HP Serial Attached SCSI Disk (naa.600c0ff00014ddb44c57ac5401000000)
   Adapter: vmhba3
   Channel: 1
   Target: 1
   LUN: 5 <--Second Missing LUN
   Plugin: NMP
   State: active
   Transport: sas
   Adapter Identifier: sas.50014380388d8480
   Target Identifier: sas.d0b8d32406430000
   Adapter Transport Details: 50014380388d8480
   Target Transport Details: d0b8d32406430000
   Maximum IO Size: 4194304

------output removed for the sake of brevity------

[/box]

At this point I opened a support call with VMware and started doing other work while I waited for them to ring back. By the following morning I was still waiting, but I had found this article, I had built the new servers with HP Build versions of ESX, but perhaps I just needed to install the HP VAAI Plugin? I was fiddling with this when a nice chap called Supreet rang from VMware. I explained what I was trying to do, and got him WebEx’d on (I try not to waste a ton of time saying I’ve done X,Y, and Z, people do that to me all the time, and it just slows the process down, if anything I’d done was correct, it would have been fixed already!)

He confirmed the hosts were definitely seeing the storage;

[box]

[root@ESX1:/var/log] esxcli storage vmfs extent list
Volume Name     VMFS UUID                            Extent Number  Device Name                           Partition
--------------  -----------------------------------  -------------  ------------------------------------  ---------
P2000_SAS1      502cba95-9e8cab7c-749d-ac162d6f719d              0  naa.600c0ff00014ddb4d3d82a5001000000          1
P2000_SAS2      502cbb3d-c7b6c728-f088-ac162d6f719d              0  naa.600c0ff00014dfce05d92a5001000000          1
P2000_SAS3_R10  50753c5d-384acc4c-c4b6-ac162d6f719c              0  naa.600c0ff00014ddb4514e745001000000          1
P2000_SAS4      542eb3f8-da4ea518-553e-ac162d6f719c              0  naa.600c0ff00014dfce99cd2d5401000000          1
P2000_SAS5_R1   54ad3e22-b39316bd-3e65-ac162d6f719c              0  naa.600c0ff00014ddb44c57ac5401000000          1

[/box]

Note: The bottom two are the missing ones. Using that information he had a look in the logs.

[box]

[root@ESX1:/var/log] grep -i "542eb3f8-da4ea518-553e-ac162d6f719c" vmkernel.log | less
[root@ESX1:/var/log] grep -i "54ad3e22-b39316bd-3e65-ac162d6f719c" vmkernel.log | less

[/box]

That showed up the following;

[box]

2017-03-20T16:23:16.754Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 542eb3f8-da4ea518-553e-ac162d6f719c: Not supported
2017-03-20T16:23:16.754Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 0 gen 0 :Not supported
2017-03-20T16:23:16.754Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 4 gen 1 :Not supported
2017-03-20T16:23:16.896Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 542eb3f8-da4ea518-553e-ac162d6f719c: Not supported
2017-03-20T16:23:16.896Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 0 gen 0 :Not supported
2017-03-20T16:23:16.896Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 4 gen 1 :Not supported
 
2017-03-20T16:23:16.675Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 54ad3e22-b39316bd-3e65-ac162d6f719c: Not supported
2017-03-20T16:23:16.675Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 0 gen 0 :Not supported
2017-03-20T16:23:16.675Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 4 gen 1 :Not supported
2017-03-20T16:23:16.910Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 54ad3e22-b39316bd-3e65-ac162d6f719c: Not supported
2017-03-20T16:23:16.910Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 0 gen 0 :Not supported
2017-03-20T16:23:16.910Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 4 gen 1 :Not supported

[/box]

That pointed him towards the VAAI, (perhaps the stuff I’d been reading, had me on the right track?)

[box]

[root@ESX1:/var/log] esxcli storage core device vaai status get
naa.600c0ff00014ddb44c57ac5401000000
   VAAI Plugin Name:
   ATS Status: unsupported
   Clone Status: unsupported
   Zero Status: supported
   Delete Status: unsupported

naa.600c0ff00014dfce99cd2d5401000000
   VAAI Plugin Name:
   ATS Status: unsupported
   Clone Status: unsupported
   Zero Status: supported
   Delete Status: unsupported

------output removed for the sake of brevity------

[/box]

Note the difference, the following is on the older servers that were working fine;

[box]

VMware ESXi 5.5.0 Update 3
~ # esxcli storage core device vaai status get
naa.600c0ff00014ddb44c57ac5401000000
   VAAI Plugin Name: hp_vaaip_p2000
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: unsupported

naa.600c0ff00014dfce99cd2d5401000000
   VAAI Plugin Name: hp_vaaip_p2000
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: unsupported

[/box]

At this point, I piped up about the VAAI stuff I’d been reading, and told them that I’d download the VIB, and it was already on the offending server.

So they installed it and rebooted the server, (there were no running VMs on the new box).

I waited with bated breath, and it didn’t fix it 🙁 It was at this point that they gave me the bad news, HP P2000 G3 is not supported on ESX 6.5, (In fact its not supported on 6.0 either!)

Well I suppose that’s an answer, but not the one I wanted! I downgraded the hosts to 5.5U3a Same Problem! So I downgraded them to 5.0.0, then they wouldn’t boot, (error indicate unsupported hardware). So I set about upgrading them to 5.5U2, (to be on the safe side).

Also while this was going on, I updated the firmware on the SAN controllers;

Thankfully this time the servers booted up fine, and saw the storage and mounted all the datastores.

 

Related Articles, References, Credits, or External Links

NA