KB ID 0001292
Problem
For the first time in ages I’ve been doing a VMware upgrade this week, a client had an MSA P2000 G3 and two G8 DL380 servers running vSphere 5.5. I put in a new 6.5 VCSA, built some new G9 DL380 servers, I noticed that the SAN was presenting five storage LUNs but the new ESX 6.5 servers could only see three of them?
Strangely when I selected the SAS storage controllers they could see all 5 storage LUNs, but the datastores refused to appear.
Solution
I checked that the SAN was not masking the LUN’s (it wasn’t, the default was read/write for everything). I connected to the console and proved the storage could be seen.
[box]
[root@ESX1:~] esxcli storage core path list ------output removed for the sake of brevity------ sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014dfce99cd2d5401000000 UID: sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014dfce99cd2d5401000000 Runtime Name: vmhba3:C1:T1:L4 Device: naa.600c0ff00014dfce99cd2d5401000000 Device Display Name: HP Serial Attached SCSI Disk (naa.600c0ff00014dfce99cd2d5401000000) Adapter: vmhba3 Channel: 1 Target: 1 LUN: 4 <-- First missing LUN Plugin: NMP State: active Transport: sas Adapter Identifier: sas.50014380388d8480 Target Identifier: sas.d0b8d32406430000 Adapter Transport Details: 50014380388d8480 Target Transport Details: d0b8d32406430000 Maximum IO Size: 4194304 sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014ddb44c57ac5401000000 UID: sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014ddb44c57ac5401000000 Runtime Name: vmhba3:C1:T1:L5 Device: naa.600c0ff00014ddb44c57ac5401000000 Device Display Name: HP Serial Attached SCSI Disk (naa.600c0ff00014ddb44c57ac5401000000) Adapter: vmhba3 Channel: 1 Target: 1 LUN: 5 <--Second Missing LUN Plugin: NMP State: active Transport: sas Adapter Identifier: sas.50014380388d8480 Target Identifier: sas.d0b8d32406430000 Adapter Transport Details: 50014380388d8480 Target Transport Details: d0b8d32406430000 Maximum IO Size: 4194304
------output removed for the sake of brevity------
[/box]
At this point I opened a support call with VMware and started doing other work while I waited for them to ring back. By the following morning I was still waiting, but I had found this article, I had built the new servers with HP Build versions of ESX, but perhaps I just needed to install the HP VAAI Plugin? I was fiddling with this when a nice chap called Supreet rang from VMware. I explained what I was trying to do, and got him WebEx’d on (I try not to waste a ton of time saying I’ve done X,Y, and Z, people do that to me all the time, and it just slows the process down, if anything I’d done was correct, it would have been fixed already!)
He confirmed the hosts were definitely seeing the storage;
[box]
[root@ESX1:/var/log] esxcli storage vmfs extent list
Volume Name VMFS UUID Extent Number Device Name Partition
-------------- ----------------------------------- ------------- ------------------------------------ ---------
P2000_SAS1 502cba95-9e8cab7c-749d-ac162d6f719d 0 naa.600c0ff00014ddb4d3d82a5001000000 1
P2000_SAS2 502cbb3d-c7b6c728-f088-ac162d6f719d 0 naa.600c0ff00014dfce05d92a5001000000 1
P2000_SAS3_R10 50753c5d-384acc4c-c4b6-ac162d6f719c 0 naa.600c0ff00014ddb4514e745001000000 1
P2000_SAS4 542eb3f8-da4ea518-553e-ac162d6f719c 0 naa.600c0ff00014dfce99cd2d5401000000 1
P2000_SAS5_R1 54ad3e22-b39316bd-3e65-ac162d6f719c 0 naa.600c0ff00014ddb44c57ac5401000000 1
[/box]
Note: The bottom two are the missing ones. Using that information he had a look in the logs.
[box]
[root@ESX1:/var/log] grep -i "542eb3f8-da4ea518-553e-ac162d6f719c" vmkernel.log | less [root@ESX1:/var/log] grep -i "54ad3e22-b39316bd-3e65-ac162d6f719c" vmkernel.log | less
[/box]
That showed up the following;
[box]
2017-03-20T16:23:16.754Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 542eb3f8-da4ea518-553e-ac162d6f719c: Not supported 2017-03-20T16:23:16.754Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 0 gen 0 :Not supported 2017-03-20T16:23:16.754Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 4 gen 1 :Not supported 2017-03-20T16:23:16.896Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 542eb3f8-da4ea518-553e-ac162d6f719c: Not supported 2017-03-20T16:23:16.896Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 0 gen 0 :Not supported 2017-03-20T16:23:16.896Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 4 gen 1 :Not supported 2017-03-20T16:23:16.675Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 54ad3e22-b39316bd-3e65-ac162d6f719c: Not supported 2017-03-20T16:23:16.675Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 0 gen 0 :Not supported 2017-03-20T16:23:16.675Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 4 gen 1 :Not supported 2017-03-20T16:23:16.910Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 54ad3e22-b39316bd-3e65-ac162d6f719c: Not supported 2017-03-20T16:23:16.910Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 0 gen 0 :Not supported 2017-03-20T16:23:16.910Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 4 gen 1 :Not supported
[/box]
That pointed him towards the VAAI, (perhaps the stuff I’d been reading, had me on the right track?)
[box]
[root@ESX1:/var/log] esxcli storage core device vaai status get naa.600c0ff00014ddb44c57ac5401000000 VAAI Plugin Name: ATS Status: unsupported Clone Status: unsupported Zero Status: supported Delete Status: unsupported naa.600c0ff00014dfce99cd2d5401000000 VAAI Plugin Name: ATS Status: unsupported Clone Status: unsupported Zero Status: supported Delete Status: unsupported
------output removed for the sake of brevity------
[/box]
Note the difference, the following is on the older servers that were working fine;
[box]
VMware ESXi 5.5.0 Update 3 ~ # esxcli storage core device vaai status get naa.600c0ff00014ddb44c57ac5401000000 VAAI Plugin Name: hp_vaaip_p2000 ATS Status: supported Clone Status: supported Zero Status: supported Delete Status: unsupported naa.600c0ff00014dfce99cd2d5401000000 VAAI Plugin Name: hp_vaaip_p2000 ATS Status: supported Clone Status: supported Zero Status: supported Delete Status: unsupported
[/box]
At this point, I piped up about the VAAI stuff I’d been reading, and told them that I’d download the VIB, and it was already on the offending server.
So they installed it and rebooted the server, (there were no running VMs on the new box).
I waited with bated breath, and it didn’t fix it 🙁 It was at this point that they gave me the bad news, HP P2000 G3 is not supported on ESX 6.5, (In fact its not supported on 6.0 either!)
Well I suppose that’s an answer, but not the one I wanted! I downgraded the hosts to 5.5U3a Same Problem! So I downgraded them to 5.0.0, then they wouldn’t boot, (error indicate unsupported hardware). So I set about upgrading them to 5.5U2, (to be on the safe side).
Also while this was going on, I updated the firmware on the SAN controllers;
Thankfully this time the servers booted up fine, and saw the storage and mounted all the datastores.
Related Articles, References, Credits, or External Links
NA