ESX Cannot Mount MSA P2000 Datastores

Posted on 21/03/2017 by PeteLong

KB ID 0001292

Problem

For the first time in ages I’ve been doing a VMware upgrade this week, a client had an MSA P2000 G3 and two G8 DL380 servers running vSphere 5.5. I put in a new 6.5 VCSA, built some new G9 DL380 servers, I noticed that the SAN was presenting five storage LUNs but the new ESX 6.5 servers could only see three of them?

Strangely when I selected the SAS storage controllers they could see all 5 storage LUNs, but the datastores refused to appear.

Solution

I checked that the SAN was not masking the LUN’s (it wasn’t, the default was read/write for everything). I connected to the console and proved the storage could be seen.

[box]

[root@ESX1:~] esxcli storage core path list
------output removed for the sake of brevity------

sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014dfce99cd2d5401000000
   UID: sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014dfce99cd2d5401000000
   Runtime Name: vmhba3:C1:T1:L4
   Device: naa.600c0ff00014dfce99cd2d5401000000
   Device Display Name: HP Serial Attached SCSI Disk (naa.600c0ff00014dfce99cd2d5401000000)
   Adapter: vmhba3
   Channel: 1
   Target: 1
   LUN: 4 <-- First missing LUN
   Plugin: NMP
   State: active
   Transport: sas
   Adapter Identifier: sas.50014380388d8480
   Target Identifier: sas.d0b8d32406430000
   Adapter Transport Details: 50014380388d8480
   Target Transport Details: d0b8d32406430000
   Maximum IO Size: 4194304

sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014ddb44c57ac5401000000
   UID: sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014ddb44c57ac5401000000
   Runtime Name: vmhba3:C1:T1:L5
   Device: naa.600c0ff00014ddb44c57ac5401000000
   Device Display Name: HP Serial Attached SCSI Disk (naa.600c0ff00014ddb44c57ac5401000000)
   Adapter: vmhba3
   Channel: 1
   Target: 1
   LUN: 5 <--Second Missing LUN
   Plugin: NMP
   State: active
   Transport: sas
   Adapter Identifier: sas.50014380388d8480
   Target Identifier: sas.d0b8d32406430000
   Adapter Transport Details: 50014380388d8480
   Target Transport Details: d0b8d32406430000
   Maximum IO Size: 4194304

------output removed for the sake of brevity------

[/box]

At this point I opened a support call with VMware and started doing other work while I waited for them to ring back. By the following morning I was still waiting, but I had found this article, I had built the new servers with HP Build versions of ESX, but perhaps I just needed to install the HP VAAI Plugin? I was fiddling with this when a nice chap called Supreet rang from VMware. I explained what I was trying to do, and got him WebEx’d on (I try not to waste a ton of time saying I’ve done X,Y, and Z, people do that to me all the time, and it just slows the process down, if anything I’d done was correct, it would have been fixed already!)

He confirmed the hosts were definitely seeing the storage;

[box]

[root@ESX1:/var/log] esxcli storage vmfs extent list
Volume Name     VMFS UUID                            Extent Number  Device Name                           Partition
--------------  -----------------------------------  -------------  ------------------------------------  ---------
P2000_SAS1      502cba95-9e8cab7c-749d-ac162d6f719d              0  naa.600c0ff00014ddb4d3d82a5001000000          1
P2000_SAS2      502cbb3d-c7b6c728-f088-ac162d6f719d              0  naa.600c0ff00014dfce05d92a5001000000          1
P2000_SAS3_R10  50753c5d-384acc4c-c4b6-ac162d6f719c              0  naa.600c0ff00014ddb4514e745001000000          1
P2000_SAS4      542eb3f8-da4ea518-553e-ac162d6f719c              0  naa.600c0ff00014dfce99cd2d5401000000          1
P2000_SAS5_R1   54ad3e22-b39316bd-3e65-ac162d6f719c              0  naa.600c0ff00014ddb44c57ac5401000000          1

[/box]

Note: The bottom two are the missing ones. Using that information he had a look in the logs.

[box]

[root@ESX1:/var/log] grep -i "542eb3f8-da4ea518-553e-ac162d6f719c" vmkernel.log | less
[root@ESX1:/var/log] grep -i "54ad3e22-b39316bd-3e65-ac162d6f719c" vmkernel.log | less

[/box]

That showed up the following;

[box]

2017-03-20T16:23:16.754Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 542eb3f8-da4ea518-553e-ac162d6f719c: Not supported
2017-03-20T16:23:16.754Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 0 gen 0 :Not supported
2017-03-20T16:23:16.754Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 4 gen 1 :Not supported
2017-03-20T16:23:16.896Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 542eb3f8-da4ea518-553e-ac162d6f719c: Not supported
2017-03-20T16:23:16.896Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 0 gen 0 :Not supported
2017-03-20T16:23:16.896Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 4 gen 1 :Not supported
 
2017-03-20T16:23:16.675Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 54ad3e22-b39316bd-3e65-ac162d6f719c: Not supported
2017-03-20T16:23:16.675Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 0 gen 0 :Not supported
2017-03-20T16:23:16.675Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 4 gen 1 :Not supported
2017-03-20T16:23:16.910Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 54ad3e22-b39316bd-3e65-ac162d6f719c: Not supported
2017-03-20T16:23:16.910Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 0 gen 0 :Not supported
2017-03-20T16:23:16.910Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 4 gen 1 :Not supported

[/box]

That pointed him towards the VAAI, (perhaps the stuff I’d been reading, had me on the right track?)

[box]

[root@ESX1:/var/log] esxcli storage core device vaai status get
naa.600c0ff00014ddb44c57ac5401000000
   VAAI Plugin Name:
   ATS Status: unsupported
   Clone Status: unsupported
   Zero Status: supported
   Delete Status: unsupported

naa.600c0ff00014dfce99cd2d5401000000
   VAAI Plugin Name:
   ATS Status: unsupported
   Clone Status: unsupported
   Zero Status: supported
   Delete Status: unsupported

------output removed for the sake of brevity------

[/box]

Note the difference, the following is on the older servers that were working fine;

[box]

VMware ESXi 5.5.0 Update 3
~ # esxcli storage core device vaai status get
naa.600c0ff00014ddb44c57ac5401000000
   VAAI Plugin Name: hp_vaaip_p2000
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: unsupported

naa.600c0ff00014dfce99cd2d5401000000
   VAAI Plugin Name: hp_vaaip_p2000
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: unsupported

[/box]

At this point, I piped up about the VAAI stuff I’d been reading, and told them that I’d download the VIB, and it was already on the offending server.

So they installed it and rebooted the server, (there were no running VMs on the new box).

I waited with bated breath, and it didn’t fix it 🙁 It was at this point that they gave me the bad news, HP P2000 G3 is not supported on ESX 6.5, (In fact its not supported on 6.0 either!)

Well I suppose that’s an answer, but not the one I wanted! I downgraded the hosts to 5.5U3a Same Problem! So I downgraded them to 5.0.0, then they wouldn’t boot, (error indicate unsupported hardware). So I set about upgrading them to 5.5U2, (to be on the safe side).

Also while this was going on, I updated the firmware on the SAN controllers;

Thankfully this time the servers booted up fine, and saw the storage and mounted all the datastores.

HP StorageWorks P2000 – Connecting to and Configuring

Posted on 17/11/2015 by Migrated

KB ID 0000569

Problem

Normally I simply connect a new MSA to a clients network, and it gets it’s address from DHCP. Then I can get the address for the DHCP Scope, and point my web browser at it.

Yesterday I was starting with new virtual infrastructure and had no DHCP. With the G1 and G2 models, you got a console/serial cable and could just terminal in. With the G3 they have replaced the serial socket with a mini USB socket. Each time I put in a new P2000, I think “I wonder how that USB CLI socket works?” Yesterday I had to find out.

Solution

The Quickest Solution – is to connect the MSA to the network, and if it cannot get a DHCP address it automatically gives itself 10.0.0.2/24 on controller A and 10.0.0.3/24 on controller B.

MSA Default username = manage
MSA Default password = !manage

The Next Quickest Solution

1. If you do have DHCP running, connect your MSA and run the MSA Device Discovery Tool, (On the CD that came with the device).

2. Once you know the IP address, you can connect with your web browser.

Connect to and Manage your MSA via the USB/CLI Cable

1. For your machine to see the MSA as a device, you need to install a driver, there is a copy of the drivers on the CD that came with the device.

Note: Windows 7 users, use the Windows 2008 Drivers or use this one.

2. Install the driver.

3. Connect the USB lead from the MSA controller to your machine, TAKE NOTE of the COM port number it’s using.

4. Now you can use whatever terminal emulation program you prefer to connect to that COM port. (I prefer HyperTerminal, or you can use Putty if you want something a bit lighter).

5. Set the following, Bits per second = 115200, Data bits = 8, Parity = None, Stop bits = 1, and Flow control = None.

6. You will need to press {enter} to connect, then login.

MSA Default username = manage
MSA Default password = !manage

7. I only need to set the IP address of the controller(s) like so;

[box] set network-parameters ip {x.x.x.x} netmask {y.y.y.y} gateway {z.z.z.z} controller a [/box]

8. Then (If you have dual controllers) you can set the IP address of controller B.

[box] set network-parameters ip {a.a.a.a} netmask {b.b.b.b} gateway {c.c.c.c} controller b [/box]

9. Give the new IP addresses a quick test.

10. Note: You Cannot Manage an G3 P2000 with Internet Explorer 9, If you try it will simply say “Unable to Authenticate, Try again”

11. Normally I use Chrome, but that won’t work either 🙁 Firefox works fine though!

HP MSA P2000 (G3) Error – ‘Unwritable write-back cache data exists for a volume’

Posted on 17/11/2015 by Migrated

KB ID 0000817

Problem

Seen on a G3 P2000 SAN, the client had also had an MSA70 shelf, which contained a failed array. I was removing the MSA and after the job, this error was getting logged.

Unwritable write-back cache data exists for a volume (vdisk: unknown name, volume: unknown name, SN {Serial Number} it comprises {number}% of cache.

Solution

Essentially, there was data in the cache that needed writing to the array/vdisk, when it failed. If the volume was going to get repaired and brought back online the data would have got written back. However this volume was never coming back.

1. Connect to a controller via Telnet.

2. Issue the following command;

[box] clear cache [/box]

Tether your iPhone or Using your iPhone as a Wireless Access Point (without jail breaking it)

Posted on 17/11/2015 by Migrated

KB ID 0000373

Problem

Before you start, your provider has to allow tethering of your iPhone, if the option is not there you need to speak to them, some carriers will want to charge you more for the privilege, others will not care, thats not Apples fault, nor is it mine 🙂

Secondly, before you all start mailing in to berate me, yes I know I’m not turning my phone into a wireless access point, what I’m actually doing is tethering via bluetooth, which is very different, but if you in a hotel, or airport lounge with your laptop, do you want to argue semantics or get online?

Update: iPhone Personal Wireless Hotspot feature coming in version 4.3

Solution

Unfortunately (if you don’t like it) you have to have iTunes installed on the PC in question. This will put on all the Apple mobile device drivers you require.

1. If you are going to tether by the USB cable or via Bluetooth, the setup is the same. On the phone go to Settings > General > Network.