Upgrading vSphere ESXi Hosts (Including HP and Dell)

KB ID 0001343

Problem

Before you think about upgrading your hosts, you should upgrade your vCenter,

Upgrade vSphere vCenter Appliance

Assuming that’s all done, your task now is to upgrade your hosts, if you have a vanilla VMware ESXi version installed. You COULD simply SSH into the host, and execute the following commands;

[box]

esxcli network firewall ruleset set -e true -r httpClient
esxcli software profile install -p ESXi-6.5.0-20170702001-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

[/box]

Well that’s great but if you are using a custom ESXi image, the process won’t even start, you simply get warned that this won’t work, as it will break all the vendor specific drivers/software, (which is a good thing I suppose, the warning I mean, not the breaking things!)

Still, if you have a vendor modified copy of ESX what do you do? Well the following procedures will work for any version of the software, either vanilla VMware or HP/Dell/IBM etc. AND IT RETAINS ALL THE HOST SETTINGS, i.e. licences, vSwitches (standard and distributed), certificates etc.

Solution 1: Use an ‘Offline Bundle’ update

Firstly, you need a copy of the appropriate ‘offline bundle’ update, below you can see this is the customised one for HPE servers.

Upload the offline bundle into a datastore, (that the host to be upgraded has access to!) While in here, shut down the guest VMs on this host, and put the host into ‘maintenance mode‘.

NOTE: If you are updating an HPE Server, there will be a gen-9 and a pre-gen-9 update bundle! Pick the correct one!

Or, you can upload the bundle via SCP into the appropriate datastore, if you prefer.

SSH into the host you are going to upgrade. Below you can see me navigating to the Datastore;

[box]

cd /vmfs/volumes
ls

[/box]

Enable SSH Access to VMware vSphere ESX

And there’s my offline bundle ready to be installed.

Remember: Even if you’re in the correct directory, you need to specify the ‘full path’ to the ‘offline bundle‘ (or it looks in the ‘/var/log’ directory and won’t work). Execute the following command;

[box]

esxcli software vib update -d /vmfs/volumes/{Datastore-Name}/VMWare-ESXi-6.5.0-Update2-9298722-HPE-preGen9-650.U2.9.6.8.3-Sept2018-depot.zip

[/box]

For a while it will look like nothing is happening, (don’t panic.) After a while a LOT of text will scroll past (quickly). Scroll back up to the TOP of all that text, and you are looking for, is ‘The update completed successfully‘.

Reboot the host.

Solution 2: Use VMware Update Manager

Warning: You CANT deploy an ESXi image, thats OLDER than the Update Manager you are running, i.e. if your vCenter is 6.5 you cant upload an ESX 6.0 image, (it will fail – this cost me two hours onsite!)

Warning 2: You need vCenter 6.0 Update1 or newer to perform this function.

I’ve got two hosts, one’s a Dell PowerEdge and the other a HP Proliant, I’ve already upgraded the HP server, you will see that in the screen shots below, now I’m going to upgrade the Dell.

Before Starting;

  • Download the install .ISO file that has the newer version of the ESXi software.
  • vMotion all the VM’s off the host being upgraded, (to save time).
  • If you have two hosts, (like me.) You might want to disable HA and DRS on the cluster as well temporarily!

Connect to vCenter > Home.

Update Manager.

Select the vCenter > Manage > ESXi Images > Import ESXi image.

Browse to the .iso file you downloaded and upload it to update manager.

After a short pause you should see the image appear, (Note: you can see the HP one I uploaded earlier) > Select the one to deploy > Create Baseline.

Give it a sensible name > OK.

Back in ‘Hosts and Clusters’ View > Select the Cluster > Update Manager > Attach Baseline.

Select your new one > OK.

Scan for updates > Select Upgrade Only > OK.

After a while, you will see your baseline saying ‘Non-Compliant‘ > Select it > Remediate.

Select the new baseline again > Next > Select the ‘Target Host’ to upgrade > Next > Accept the EULA > Next.

Next > Next.

Next > Finish.

Watch the task bar > Remediation will start > The host will go into Maintenance mode, get upgraded, reboot, and be taken out of maintenance mode.

Here you can see my host is now upgraded. (Mine’s an older server, it took about 45 minutes).

Note: If you disabled HA, DRS etc, you will want to re-enable that now.

 

Related Articles, References, Credits, or External Links

Upgrade vSphere vCenter Appliance to Version 6.5

VMware Upgrading the vSphere Virtual Center Appliance

VMware ESXi 5 – Applying Patches and Updates

Update VMware ESXi from 4.0 to 4.1

Upgrade ESX 3 to version 4.1.0

ESX Cannot Mount MSA P2000 Datastores

KB ID 0001292 

Problem

For the first time in ages I’ve been doing a VMware upgrade this week, a client had an MSA P2000 G3 and two G8 DL380 servers running vSphere 5.5. I put in a new 6.5  VCSA, built some new G9 DL380 servers,  I noticed that the SAN was presenting five storage LUNs but the new ESX 6.5 servers could only see three of them?

Strangely when I selected the SAS storage controllers they could see all 5 storage LUNs, but the datastores refused to appear.

Solution

I checked that the SAN was not masking the LUN’s (it wasn’t, the default was read/write for everything). I connected to the  console and proved the storage could be seen.

[box]

[root@ESX1:~] esxcli storage core path list
------output removed for the sake of brevity------

sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014dfce99cd2d5401000000
   UID: sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014dfce99cd2d5401000000
   Runtime Name: vmhba3:C1:T1:L4
   Device: naa.600c0ff00014dfce99cd2d5401000000
   Device Display Name: HP Serial Attached SCSI Disk (naa.600c0ff00014dfce99cd2d5401000000)
   Adapter: vmhba3
   Channel: 1
   Target: 1
   LUN: 4 <-- First missing LUN
   Plugin: NMP
   State: active
   Transport: sas
   Adapter Identifier: sas.50014380388d8480
   Target Identifier: sas.d0b8d32406430000
   Adapter Transport Details: 50014380388d8480
   Target Transport Details: d0b8d32406430000
   Maximum IO Size: 4194304

sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014ddb44c57ac5401000000
   UID: sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014ddb44c57ac5401000000
   Runtime Name: vmhba3:C1:T1:L5
   Device: naa.600c0ff00014ddb44c57ac5401000000
   Device Display Name: HP Serial Attached SCSI Disk (naa.600c0ff00014ddb44c57ac5401000000)
   Adapter: vmhba3
   Channel: 1
   Target: 1
   LUN: 5 <--Second Missing LUN
   Plugin: NMP
   State: active
   Transport: sas
   Adapter Identifier: sas.50014380388d8480
   Target Identifier: sas.d0b8d32406430000
   Adapter Transport Details: 50014380388d8480
   Target Transport Details: d0b8d32406430000
   Maximum IO Size: 4194304

------output removed for the sake of brevity------

[/box]

At this point I opened a support call with VMware and started doing other work while I waited for them to ring back. By the following morning I was still waiting, but I had found this article, I had built the new servers with HP Build versions of ESX, but perhaps I just needed to install the HP VAAI Plugin? I was fiddling with this when a nice chap called Supreet rang from VMware. I explained what I was trying to do, and got him WebEx’d on (I try not to waste a ton of time saying I’ve done X,Y, and Z, people do that to me all the time, and it just slows the process down, if anything I’d done was correct, it would have been fixed already!)

He confirmed the hosts were definitely seeing the storage;

[box]

[root@ESX1:/var/log] esxcli storage vmfs extent list
Volume Name     VMFS UUID                            Extent Number  Device Name                           Partition
--------------  -----------------------------------  -------------  ------------------------------------  ---------
P2000_SAS1      502cba95-9e8cab7c-749d-ac162d6f719d              0  naa.600c0ff00014ddb4d3d82a5001000000          1
P2000_SAS2      502cbb3d-c7b6c728-f088-ac162d6f719d              0  naa.600c0ff00014dfce05d92a5001000000          1
P2000_SAS3_R10  50753c5d-384acc4c-c4b6-ac162d6f719c              0  naa.600c0ff00014ddb4514e745001000000          1
P2000_SAS4      542eb3f8-da4ea518-553e-ac162d6f719c              0  naa.600c0ff00014dfce99cd2d5401000000          1
P2000_SAS5_R1   54ad3e22-b39316bd-3e65-ac162d6f719c              0  naa.600c0ff00014ddb44c57ac5401000000          1

[/box]

Note: The bottom two are the missing ones. Using that information he had a look in the logs.

[box]

[root@ESX1:/var/log] grep -i "542eb3f8-da4ea518-553e-ac162d6f719c" vmkernel.log | less
[root@ESX1:/var/log] grep -i "54ad3e22-b39316bd-3e65-ac162d6f719c" vmkernel.log | less

[/box]

That showed up the following;

[box]

2017-03-20T16:23:16.754Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 542eb3f8-da4ea518-553e-ac162d6f719c: Not supported
2017-03-20T16:23:16.754Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 0 gen 0 :Not supported
2017-03-20T16:23:16.754Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 4 gen 1 :Not supported
2017-03-20T16:23:16.896Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 542eb3f8-da4ea518-553e-ac162d6f719c: Not supported
2017-03-20T16:23:16.896Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 0 gen 0 :Not supported
2017-03-20T16:23:16.896Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 4 gen 1 :Not supported
 
2017-03-20T16:23:16.675Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 54ad3e22-b39316bd-3e65-ac162d6f719c: Not supported
2017-03-20T16:23:16.675Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 0 gen 0 :Not supported
2017-03-20T16:23:16.675Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 4 gen 1 :Not supported
2017-03-20T16:23:16.910Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 54ad3e22-b39316bd-3e65-ac162d6f719c: Not supported
2017-03-20T16:23:16.910Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 0 gen 0 :Not supported
2017-03-20T16:23:16.910Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 4 gen 1 :Not supported

[/box]

That pointed him towards the VAAI, (perhaps the stuff I’d been reading, had me on the right track?)

[box]

[root@ESX1:/var/log] esxcli storage core device vaai status get
naa.600c0ff00014ddb44c57ac5401000000
   VAAI Plugin Name:
   ATS Status: unsupported
   Clone Status: unsupported
   Zero Status: supported
   Delete Status: unsupported

naa.600c0ff00014dfce99cd2d5401000000
   VAAI Plugin Name:
   ATS Status: unsupported
   Clone Status: unsupported
   Zero Status: supported
   Delete Status: unsupported
------output removed for the sake of brevity------

[/box]

Note the difference, the following is on the older servers that were working fine;

[box]

VMware ESXi 5.5.0 Update 3
~ # esxcli storage core device vaai status get
naa.600c0ff00014ddb44c57ac5401000000
   VAAI Plugin Name: hp_vaaip_p2000
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: unsupported

naa.600c0ff00014dfce99cd2d5401000000
   VAAI Plugin Name: hp_vaaip_p2000
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: unsupported

[/box]

At this point, I piped up about the VAAI stuff I’d been reading, and told them that I’d download the VIB, and it was already on the offending server.

So they installed it and rebooted the server, (there were no running VMs on the new box).

I waited with bated breath, and it didn’t fix it 🙁 It was at this point that they gave me the bad news, HP P2000 G3 is not supported on ESX 6.5, (In fact its not supported on 6.0 either!)

Well I suppose that’s an answer, but not the one I wanted! I downgraded the hosts to 5.5U3a Same Problem! So I downgraded them to 5.0.0, then they wouldn’t boot, (error indicate unsupported hardware). So I set about upgrading them to 5.5U2, (to be on the safe side).

Also while this was going on, I updated the firmware on the SAN controllers;

Thankfully this time the servers booted up fine, and saw the storage and mounted all the datastores.

 

Related Articles, References, Credits, or External Links

NA

ESX 4 to ESX 5 Upgrade Warning ‘Custom VIB Warning’

KB ID 0000500

Problem

Seen when upgrading ESXi 4.1 U1 to ESX 5.

Error: <3RD_PARTY_SOFTWARE_WARNING: Host contains third-party VIB(s) that have no substitute on the ISO. [‘oem-vmware-esx-drivers-net-vxge’, ‘oem-vmware-esx-drivers-scsi-3w-9xxx’]>

If attempting to upgrade with VMware update manager you will see the following;

Solution

Luckily this was on a test box so I was somewhat more cavalier in my attitude than if I’d been on a clients production network, It goes without saying that you should not be proceeding with an upgrade without a good backup of your production VM’s anyway.

I simply chose the “Force Migrate” option, and the upgrade proceeded seamlessly. WARNING if your error message has different VIB’s listed, then see then DO NOT proceed without seeing the warning below.

What’s this all about?

VMware are wanting you to use the VMware PowerCLI utility to inject the custom VIB’s Into the ESX5 install media. At time of writing I could locate neither the VIB’s, or the depot install package I needed to do this.

Related Articles, References, Credits, or External Links

NA

VMware ESXi 5 – Applying Patches and Updates

(ESXi 5 Update 1 and Patches)

KB ID 0000623

Problem

When VMware released ESX 4.1, they took away the “Host update utility”, (which was a mistake!). For people without VMware Update Manager, you now have to either put in the CD/DVD and do an ‘in place upgrade’, or grow a ginger pony tail and put some socks/sandals on and do some Linux.

Below I’ve got a build of ESX with no updates on it, I’m going to apply the ‘Update 1″ then the most recent patch to bring the host up to date (at time of writing).

Solution

1. Whilst connected to your host with the VI client > Select the host > Configuration > Security Profile > Services Section > Properties > SSH > Options >Start > OK > OK > Exit the VI client.

2. Connect to the host via an SSH command window (PuTTy shown). Log on as the root user, to check your current ‘build’ version issue the following command;

[box]vmware -v[/box]

3. You should have a fair idea what piece of storage has the most free space already, this host only has one datastore (datastore1). That’s the one I’m going to download the updates into. To view the Datastores issue the following command;

[box]ls -l /vmfs/volumes/[/box]

4. Change directory, so that you are ‘in’ that datastore and create a directory called ‘UPDATE’.

[box]cd /vmfs/volumes/datastore1<br /> mkdir UPDATE[/box]

5. Then change into that directory;

[box]cd UPDATE[/box]

6. From a browser visit the VMware patch portal, locate the update you want to download and install, (here it’s VMware 5.0.0 Update1). Right click the download link and copy the URL.

Note: Patches after major updates are cumulative, I also downloaded the ‘latest’ patch.

7. You need to remove the ‘s’ from the URL, I just paste the link into notepad, edit it then copy it again.

8. Back at your command window download the update with the following command;

[box]wget http://hostupdate.vmware.com/software/VUM/OFFLINE/release-328-20120312-212851/update-from-esxi5.0-5.0_update01.zip[/box]

9. When completed, download any other required patches;

[box]wget http://hostupdate.vmware.com/software/VUM/OFFLINE/release-341-20120605-165537/ESXi500-201206001.zip[/box]

10. Before applying the update/patches, the host needs to be in maintenance mode;

[box]vim-cmd hostsvc/maintenance_mode_enter[/box]

11. Then apply the update with the ‘esxcli’ command as follows;

[box]esxcli software vib install -d /vmfs/volumes/datastore1/UPDATE/update-from-esxi5.0-5.0_update01.zip[/box]

12. When complete, you will need to ‘scroll up’ and make sure it says, “Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.”

13. After the update, I am applying the latest patch with the same esxcli syntax, like so;

[box]esxcli software vib install -d /vmfs/volumes/datastore1/UPDATE/ESXi500-201206001.zip[/box]

14. Once again you will need to ‘scroll up’ and locate, “Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.”

15. Now reboot the host;

[box]reboot[/box]

16. Once the host is back online, connect and take it out of maintenance mode.

17. Then locate the folder containing the update software and patches.

18. And delete it.

19. Finally power your guest machines back on again.

Related Articles, References, Credits, or External Links

Enable SSH Access to vSphere ESXi

Upgrade vSphere 4 Environment to vSphere 5

Update VMware ESXi from 4.0 to 4.1

Upgrade ESX 3 to version 4.1.0