ESX Cannot Mount MSA P2000 Datastores

KB ID 0001292 

Problem

For the first time in ages I’ve been doing a VMware upgrade this week, a client had an MSA P2000 G3 and two G8 DL380 servers running vSphere 5.5. I put in a new 6.5  VCSA, built some new G9 DL380 servers,  I noticed that the SAN was presenting five storage LUNs but the new ESX 6.5 servers could only see three of them?

Strangely when I selected the SAS storage controllers they could see all 5 storage LUNs, but the datastores refused to appear.

Solution

I checked that the SAN was not masking the LUN’s (it wasn’t, the default was read/write for everything). I connected to the  console and proved the storage could be seen.

[root@ESX1:~] esxcli storage core path list
------output removed for the sake of brevity------

sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014dfce99cd2d5401000000
   UID: sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014dfce99cd2d5401000000
   Runtime Name: vmhba3:C1:T1:L4
   Device: naa.600c0ff00014dfce99cd2d5401000000
   Device Display Name: HP Serial Attached SCSI Disk (naa.600c0ff00014dfce99cd2d5401000000)
   Adapter: vmhba3
   Channel: 1
   Target: 1
   LUN: 4 <-- First missing LUN
   Plugin: NMP
   State: active
   Transport: sas
   Adapter Identifier: sas.50014380388d8480
   Target Identifier: sas.d0b8d32406430000
   Adapter Transport Details: 50014380388d8480
   Target Transport Details: d0b8d32406430000
   Maximum IO Size: 4194304

sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014ddb44c57ac5401000000
   UID: sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014ddb44c57ac5401000000
   Runtime Name: vmhba3:C1:T1:L5
   Device: naa.600c0ff00014ddb44c57ac5401000000
   Device Display Name: HP Serial Attached SCSI Disk (naa.600c0ff00014ddb44c57ac5401000000)
   Adapter: vmhba3
   Channel: 1
   Target: 1
   LUN: 5 <--Second Missing LUN
   Plugin: NMP
   State: active
   Transport: sas
   Adapter Identifier: sas.50014380388d8480
   Target Identifier: sas.d0b8d32406430000
   Adapter Transport Details: 50014380388d8480
   Target Transport Details: d0b8d32406430000
   Maximum IO Size: 4194304

------output removed for the sake of brevity------

At this point I opened a support call with VMware and started doing other work while I waited for them to ring back. By the following morning I was still waiting, but I had found this article, I had built the new servers with HP Build versions of ESX, but perhaps I just needed to install the HP VAAI Plugin? I was fiddling with this when a nice chap called Supreet rang from VMware. I explained what I was trying to do, and got him WebEx’d on (I try not to waste a ton of time saying I’ve done X,Y, and Z, people do that to me all the time, and it just slows the process down, if anything I’d done was correct, it would have been fixed already!)

He confirmed the hosts were definitely seeing the storage;

[root@ESX1:/var/log] esxcli storage vmfs extent list
Volume Name     VMFS UUID                            Extent Number  Device Name                           Partition
--------------  -----------------------------------  -------------  ------------------------------------  ---------
P2000_SAS1      502cba95-9e8cab7c-749d-ac162d6f719d              0  naa.600c0ff00014ddb4d3d82a5001000000          1
P2000_SAS2      502cbb3d-c7b6c728-f088-ac162d6f719d              0  naa.600c0ff00014dfce05d92a5001000000          1
P2000_SAS3_R10  50753c5d-384acc4c-c4b6-ac162d6f719c              0  naa.600c0ff00014ddb4514e745001000000          1
P2000_SAS4      542eb3f8-da4ea518-553e-ac162d6f719c              0  naa.600c0ff00014dfce99cd2d5401000000          1
P2000_SAS5_R1   54ad3e22-b39316bd-3e65-ac162d6f719c              0  naa.600c0ff00014ddb44c57ac5401000000          1

Note: The bottom two are the missing ones. Using that information he had a look in the logs.

[root@ESX1:/var/log] grep -i "542eb3f8-da4ea518-553e-ac162d6f719c" vmkernel.log | less
[root@ESX1:/var/log] grep -i "54ad3e22-b39316bd-3e65-ac162d6f719c" vmkernel.log | less

That showed up the following;

2017-03-20T16:23:16.754Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 542eb3f8-da4ea518-553e-ac162d6f719c: Not supported
2017-03-20T16:23:16.754Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 0 gen 0 :Not supported
2017-03-20T16:23:16.754Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 4 gen 1 :Not supported
2017-03-20T16:23:16.896Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 542eb3f8-da4ea518-553e-ac162d6f719c: Not supported
2017-03-20T16:23:16.896Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 0 gen 0 :Not supported
2017-03-20T16:23:16.896Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 4 gen 1 :Not supported
 
2017-03-20T16:23:16.675Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 54ad3e22-b39316bd-3e65-ac162d6f719c: Not supported
2017-03-20T16:23:16.675Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 0 gen 0 :Not supported
2017-03-20T16:23:16.675Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 4 gen 1 :Not supported
2017-03-20T16:23:16.910Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 54ad3e22-b39316bd-3e65-ac162d6f719c: Not supported
2017-03-20T16:23:16.910Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 0 gen 0 :Not supported
2017-03-20T16:23:16.910Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 4 gen 1 :Not supported

That pointed him towards the VAAI, (perhaps the stuff I’d been reading, had me on the right track?)

[root@ESX1:/var/log] esxcli storage core device vaai status get
naa.600c0ff00014ddb44c57ac5401000000
   VAAI Plugin Name:
   ATS Status: unsupported
   Clone Status: unsupported
   Zero Status: supported
   Delete Status: unsupported

naa.600c0ff00014dfce99cd2d5401000000
   VAAI Plugin Name:
   ATS Status: unsupported
   Clone Status: unsupported
   Zero Status: supported
   Delete Status: unsupported
------output removed for the sake of brevity------

Note the difference, the following is on the older servers that were working fine;

VMware ESXi 5.5.0 Update 3
~ # esxcli storage core device vaai status get
naa.600c0ff00014ddb44c57ac5401000000
   VAAI Plugin Name: hp_vaaip_p2000
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: unsupported

naa.600c0ff00014dfce99cd2d5401000000
   VAAI Plugin Name: hp_vaaip_p2000
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: unsupported

At this point, I piped up about the VAAI stuff I’d been reading, and told them that I’d download the VIB, and it was already on the offending server.

HP VAAI Plugin ESX

So they installed it and rebooted the server, (there were no running VMs on the new box).

How to Install a VIB in ESX

I waited with bated breath, and it didn’t fix it 🙁 It was at this point that they gave me the bad news, HP P2000 G3 is not supported on ESX 6.5, (In fact its not supported on 6.0 either!)

P2000 G3 Not Supported ESX 6

Well I suppose that’s an answer, but not the one I wanted! I downgraded the hosts to 5.5U3a Same Problem! So I downgraded them to 5.0.0, then they wouldn’t boot, (error indicate unsupported hardware). So I set about upgrading them to 5.5U2, (to be on the safe side).

Also while this was going on, I updated the firmware on the SAN controllers;

MSA 200 P3 Firmware update

Thankfully this time the servers booted up fine, and saw the storage and mounted all the datastores.

 

Related Articles, References, Credits, or External Links

NA

Author: PeteLong

Share This Post On

14 Comments

  1. Hello.
    Do you think it will work on 6.5 with the new update on the SAN Controllers?

    Post a Reply
  2. Hi Kristoffer & PeteLong,

    We are in the same situation here, we have two DL380 G8 that are still compatible with vSphere 6.5 but with on old P2000 G3 storage.
    The current firmware on our P2000 still required the VAAI plug-in, we are trying to upgrade the controllers to the last version which does not need it (TS252P005), but we struggle with the HPE website to download it (if by any case you have it somewhere, I am more than interested).

    Back to my tests, the servers were on 6.0 (whatever version), I have upgraded them to 6.5 Update 1 and install the VAAI plug-in > no datastores.
    I have then downgraded them to 6.0 Update 3 and install the ViB > this combination is working for us.

    I will let you know if we manage the upgrade of the controllers.

    Best,

    Post a Reply
    • Thanks man! Your comment gave me some light to fix my setup: ESXi6.0u3 (with VAAI plugin installed) connected to P2000g3(TS240)

      Post a Reply
      • Hi Markus,

        Good Call.

        Pete

        Post a Reply
      • Thanks Markus, you’re a life saver! (No HP contract here)

        Post a Reply
    • For what it’s worth- the P2000 G3 works just fine with vpshere 6.5 (U1) and the latest Firmware.
      You just need to be careful when upgrading the Controllers, because the old VAAI plugin is not compatible with the new Firmware and might break your Datastores or cause an APD/PDL. You have to uninstall it and remove all configuration for it before upgrading the Firmware – it’s not needed afterwards anyway.

      Post a Reply
  3. Thanks Pete and posters. We’ll downgrade this customer to V6.0 U3, as V6.7 does not seem to want to show the datastores at all, and there is a HP image for Gen10 servers.
    This supposedly seems to work with old datastores.

    Post a Reply
  4. HP bl 460c g7 + vmware 6.5 u2 Standart edition (not hpe optimized) + two 6gb SAS BL switches + p2000 g3 MSA – works fine.

    Post a Reply
  5. Hey guys thanks a lot for this helpful article and comments !

    P2000 G3 upgraded to latest Seagate firmware. Then lost controllers completely ! No more drives detected. cli with “restart sc a/b” didnt work since no connection could be established.

    Finally found this resolution to my issue on HPE forums :

    – Power down array.

    – Pull out top controller

    – Power up array.

    – I was able to access SMU and see my drives and LUNs now

    – Insert top controller

    Don’t really know what happened here but now P2000 is fully good again. Seems the top controller crashed during the firmware upgrade.

    Servers are DL360p Gen8 (P71 FW) we just got for test lab running under ESXi7.0.1d.

    I was able to upgrade from ESXi5.5 to 6.7 without any issue.

    Then from here every new update broke my NTG3 drivers for Broadcom 5719 card. I tried everything, even vib install new drivers, uninstall or disable old drivers, but nothing worked.

    I tried a fresh usb stick with Rufus and ESXi7.0.1c Standard iso and booted server on it. Installation worked on first try and all network drivers are good.

    No error in VCenter 7, all sensors good ! System is healthy.

    Hope this is useful

    I guess it wouldn’t have been possible without this article and comments !

    Thanks again.

    Post a Reply
  6. Hello Everyone i had the same issue, and run the following command on hpe p2000 and Storage attached successfully “this solved ESXi 6.x and Microsoft”

    show unwritable-cache

    If percent > 0 then run:

    clear cache

    It helps for me…

    Post a Reply

Leave a Reply to PeteLong Cancel reply

Your email address will not be published. Required fields are marked *