A few weeks ago I needed to sort out some extended warranty for a customers servers. To do that I needed the serial numbers of those servers, (a mixture of IBM/Lenovo and Dell Servers).
As I didn’t fancy a drive to two different datacenters, I wanted to try and get them programatically.
Solution
After some searching I came across a post by one of my old EE buddies LucD with exactly what I needed. I’m assuming you have PowerCLI setup before beginning.
Connect to your Virtual infrastructure;
[box]
Connect-VIServer {vCenter-server-FQDN}
[/box]
Then, (assuming you have a folder called C:\Temp that you can write to).
If you have fiber channel switches, (regardless of the vendor,) scratch the surface and underneath it’s probably a Brocade. (Unless it’s a Cisco Nexus then you are in the wrong place my friend, move along!) e.g an HP StorageWorks 8/8 SAN Switch (Yeah it’s a Brocade 6505), or an IBM SAN24B-4 Express Fabric Switch (you guessed it, Brocade).
If you need to get the serial number for them, here’s how.
Solution
SSH into the switch, and issue the following command;
[box]chassisshow[/box]
I need the Brocades Vendor Serial Number!
For ‘re-badged’ Brocades, some vendors, (HP for example), have a ‘Suppler Serial Number‘ also, you need to GUI into the switch manager to get that, (that means using a browser and having Java installed!) Warning: You will need to enter the switches IP or FQDN into the the ‘Safe’ list in the Java settings in the Control Panel or this will fail. Typically you then browse to http://{IP-Address}/switchExplorer_installed.html to then get access.
Go here;
1: Is the Brocade Serial Number.
2. Is the Supplier (Vendor) Serial Number.
Related Articles, References, Credits, or External Links
I had to find all the HP iLOs on a client network today, I could have got an IP scanner and then sorted the results for HP devices, but I would have ended up with switches, printers and God knows what else!
If only there was a PowerShell command I could use?
Solution
There is! You will need to download it though, (HPiLOCmdlets-x64,) extract them to your C: drive.
Here I’m installing them from PowerShell, but you can simply run the HPiLOCmdlets-x64.msi file, (if you do install from PowerShell, you will need to restart it, and might need to change your signing policy).
Run the following commands;
[box]
Find-HPiLO
{Then supply the range to scan i.e. 192.168.1.1-254}
[/box]
Related Articles, References, Credits, or External Links
Life was simpler when we had DVD Drives and a wallet full of CD/DVDs! I was building an HP DL360 This morning and needed to install Windows. I created a bootable USB with Unetbootin and selected a Windows Server ISO, it wouldn’t boot. So I thought ‘Fine I’ll play the game” I pressed F10 for Intelligent Provisioning.
After selecting USB media – the system could not see my USB Drive?
After a couple of seconds head scrathing the penny dropped, it wants the iso not a bootable drive, (doofus!) So I used a FAT formatted USB and that didn’t work either?
Option 1: Use iLO
Before you all start emailing me, you can install an operating system from virtual media WITHOUT and advanced iLO licence! Annoyingly I was building the server on the bench, so I had to connect my laptop into the iLO with a crossover cable but, here’s me proving it works.
Option 2: Use ExFAT
Format your USB drive using ExFAT, luckily I use macOS and Disk Utility will format a drive using ExFAT for me.
Note: Windows will also format as ExFAT 🙂
Then simply put your install .iSO file(s) on the media.
Now you can see your install media.
Option 3: Use the HP Media Creator
I didn’t try this option, but feel free to download it and give it a try, comment below to let me know how you get on.
After ESX 5.5 Update 2, VMware added ATS Heartbeat.Some vendors, (like HPE SureStore and VSA) recommend that this is disabled. I can’t find any info about whether it’s safe to do this in production, so to be on the safe side I placed the hosts in maintenance mode first.
Enter Maintenance Mode
Use the following command;
[box]
vim-cmd /hostsvc/maintenance_mode_enter
[/box]
Disable ATS Heartbeat
Use the following command to disable;
[box]
esxcli system settings advanced set -i 0 -o /VMFS3/UseATSForHBOnVMFS5
[/box]
Then confirm it worked with following command;
[box]
esxcli system settings advanced list -o /VMFS3/UseATSForHBOnVMFS5
[/box]
Confirm that INT Value is set to 0 (zero).
Exit Maintenance Mode
Use the following command;
[box]
vim-cmd /hostsvc/maintenance_mode_exit
[/box]
Related Articles, References, Credits, or External Links
I’ve got a client that’s been having some performance issues with their VMs. Their storage vendor, (EMC) said that as a result of finding this in the logs;
[box]
B 02/28/19 09:50:53.953 scsitarg 117000e [INFO] System: iSCSI Logout Initiator Data: IP=192.168.200.161 Name=...-ec-21 Target Data: Port=2 Flags=0x00002002 Info=0x01200801
B 02/28/19 09:50:53.969 scsitarg 117000e [INFO] System: iSCSI Logout Initiator Data: IP=192.168.201.161 Name=...-ec-21 Target Data: Port=3 Flags=0x00002002 Info=0x01200801
B 02/28/19 09:51:16.413 Health 608fe [WARN] User: Host ESXi-01.petenetlive.com does not have any initiators logged into the storage system.
A 02/28/19 10:04:25.968 scsitarg 117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.200.161 Name=...-ec-21 Target Data: Port=2 Flags=0x00002002 Info=0x00000000 [Target]
B 02/28/19 10:04:26.034 scsitarg 117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.200.161 Name=...-ec-21 Target Data: Port=2 Flags=0x00002002 Info=0x00000000
A 02/28/19 10:04:31.996 scsitarg 117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.201.161 Name=...-ec-21 Target Data: Port=3 Flags=0x00002002 Info=0x00000000 [Target]
B 02/28/19 10:04:32.055 scsitarg 117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.201.161 Name=...-ec-21 Target Data: Port=3 Flags=0x00002002 Info=0x00000000
B 02/28/19 10:04:57.438 Health 608fc [INFO] User: Host ESXi-01.petenetlive.com is operating normally.
Host Host ESXi-01.petenetlive.com is accessing lun Datastore_3 as HLU 3, After the initiators for this host start logging in/logging, unit attention update events will be logged as the paths to the luns have changed this is expected
2019/02/28-09:50:41.607527 ~~~~ 7F3C92369703 std:TCD: Unit Attention update from 0000001A to 0001030D for LUN 0x3.
2019/02/28-10:02:55.860669 ~~~~ 7FE476E61702 std:TCD: Unit Attention update from 00010149 to 00010157 for LUN 0x3.
[/box]
We should disable DelayedAck and they kindly gave me the VMware KB that outlined the procedure.
Solution
The procedure outlined (for VMware 6.x) is to put the host in maintenance mode, then edit the properties of the iSCSI controller(s), untick the DelayedAck options, reboot the Host, and everything will be peachy. However, even though (post reboot) everything looks good in the the vSphere Web console. If you look on the host you may find something like this;
[box]
vmkiscsid --dump-db | grep Delayed
[/box]
DelayedAck = ‘1’ means ENABLED, DelayedAck = ‘0’ means DISABLED
So half my iSCSI entries in the iSCSI database still have DelayedAck ENABLED?
Some Internet searching told me this was quite common, and that the best way to ‘fix‘ it was to, disable the iSCSI initiator, remove the iSCSI database, reboot and then setup iSCSI again;
[box]
cd /etc/vmware/vmkiscsid
esxcfg-swiscsi -d
rm -f vmkiscsid.db
reboot
[/box]
Which is fine IF YOU ARE USING A SOFTWARE iSCSI INITIATOR, I however was not, I had 2x dedicated hardware iSCSI HBAs on each host!
After many hours of messing about and trial and error, it became clear, I had to do things in a certain order, or DelayedAck would simply just be enabled whether I liked it or not. 🙁
Disable DelayedAck With Hardware iSCSI NICs / HBAs
MAKE SURE THE HOST IS IN MAINTENANCE MODE FIRST
Then take a note of your iSCSI setup, Port Groups, VMKernel Ports, and Physical NICs, you are going to delete the iSCSI database in a minute, and you will need to ‘rebind’ the VMKernel Ports and add the iSCSI targets back in again.
Manually remove your iSCSI target(s) for ALL the iSCSI NIC/HBA’s
Below if you re-run the command, vmkiscsid –dump-db | grep Delayed you will see there’s still some entries in the database with DelayedAck enabled! So unlike above (see example for software iSCSI) we are going to remove the iSCSI database, only here we don’t need to disable the software iSCSI initiator (because we are not using one!) Finally reboot the host.
[box]
cd /etc/vmware/vmkiscsid
rm -f vmkiscsid.db
reboot
[/box]
When the host is back online ADD in the Network Port Binding for the appropriate VMkernel adaptor.
Like so;
DON’T RESCAN THE CONTROLLER AS PROMPTED TO DO SO!
On the Advanced Settings of EACH hardware iSCSI NIC/HBA > Edit > UNTICK ‘DelayedAck’.
Double check they are both still unticked (I’ve seen them re-tick themselves for no discernible reason!) Then rescan the controller(s).
Target > Add.
Re-add the iSCSI target back in, (that you took note of above).
Select the Target > Advanced > Untick the DelayedAck option (Note: This time it’s not inherited). Repeat for any additional iSCSI targets.
When they are all added, rescan the storage controllers again.
Finally recheck all the database entries are set to DISABLED.
[box]
vmkiscsid --dump-db | grep Delayed
[/box]
Related Articles, References, Credits, or External Links
Thanks to Russell and Iain for their patience while I worked all that out!
Assuming that’s all done, your task now is to upgrade your hosts, if you have a vanilla VMware ESXi version installed. You COULD simply SSH into the host, and execute the following commands;
Well that’s great but if you are using a custom ESXi image, the process won’t even start, you simply get warned that this won’t work, as it will break all the vendor specific drivers/software, (which is a good thing I suppose, the warning I mean, not the breaking things!)
Still, if you have a vendor modified copy of ESX what do you do? Well the following procedures will work for any version of the software, either vanilla VMware or HP/Dell/IBM etc. AND IT RETAINS ALL THE HOST SETTINGS, i.e. licences, vSwitches (standard and distributed), certificates etc.
Solution 1: Use an ‘Offline Bundle’ update
Firstly, you need a copy of the appropriate ‘offline bundle’ update, below you can see this is the customised one for HPE servers.
Upload the offline bundle into a datastore, (that the host to be upgraded has access to!) While in here, shut down the guest VMs on this host, and put the host into ‘maintenance mode‘.
NOTE: If you are updating an HPE Server, there will be a gen-9 and a pre-gen-9 update bundle! Pick the correct one!
Or, you can upload the bundle via SCP into the appropriate datastore, if you prefer.
SSH into the host you are going to upgrade. Below you can see me navigating to the Datastore;
And there’s my offline bundle ready to be installed.
Remember: Even if you’re in the correct directory, you need to specify the ‘full path’ to the ‘offline bundle‘ (or it looks in the ‘/var/log’ directory and won’t work). Execute the following command;
For a while it will look like nothing is happening, (don’t panic.) After a while a LOT of text will scroll past (quickly). Scroll back up to the TOP of all that text, and you are looking for, is ‘The update completed successfully‘.
Reboot the host.
Solution 2: Use VMware Update Manager
Warning: You CANT deploy an ESXi image, thats OLDER than the Update Manager you are running, i.e. if your vCenter is 6.5 you cant upload an ESX 6.0 image, (it will fail – this cost me two hours onsite!)
Warning 2: You need vCenter 6.0 Update1 or newer to perform this function.
I’ve got two hosts, one’s a Dell PowerEdge and the other a HP Proliant, I’ve already upgraded the HP server, you will see that in the screen shots below, now I’m going to upgrade the Dell.
Before Starting;
Download the install .ISO file that has the newer version of the ESXi software.
vMotion all the VM’s off the host being upgraded, (to save time).
If you have two hosts, (like me.) You might want to disable HA and DRS on the cluster as well temporarily!
While spinning up some new BL460c (Gen 10) blades for a client, I noticed there was no RAID option? They were going to be VMware ESX hosts, and had two SSD drives so I just accepted the default and my ESX server saw a LUN for each drive and I created two DataStores on each host.
The client wanted them changing to RAID1 (fair enough) so I revisited the drive setup.
Solution
WARNING: Before you proceed enabling Smart Array S100i SR is NOT CURRENTLY SUPPORTED in VMware, (you wont even see a storage controller if you carry out the following procedure.) You will need a HPE Smart Array P204i-b SR instead!
Boot the server and Press (F9) > BIOS/Platform Configuration (RBSU) Storage Options > SATA Controller Option > Change the settings to “Smart Array SW RAID Support”.
These are new blank drives right? If so, accept the warning.
Now you can boot into Intelligent Provisioning (F10) > Launch Smart Storage Administrator > And now you can configure RAID.
Note: If you have a P204i-b SR RAID card, then it will look like this, (and will work fine with VMware.)
Related Articles, References, Credits, or External Links
I was building some HP blade servers today via iLO, and while trying to install ESX, I reached this stage;
Pressing F11 on a mac just increases the volume, and with the other function keys (F1 to F10) you simply press ‘fn’ and the appropriate F key, and it works properly. But not on a mac, ‘Fn+F11’ just shows you your desktop.
Even changing function keys in the keyboard settings has no effect!
Solution
System Preferences > Mission Control.
Change ‘Show Desktop’ to another key (in this case F13).
Problem solved!
Related Articles, References, Credits, or External Links
For the first time in ages I’ve been doing a VMware upgrade this week, a client had an MSA P2000 G3 and two G8 DL380 servers running vSphere 5.5. I put in a new 6.5 VCSA, built some new G9 DL380 servers, I noticed that the SAN was presenting five storage LUNs but the new ESX 6.5 servers could only see three of them?
Strangely when I selected the SAS storage controllers they could see all 5 storage LUNs, but the datastores refused to appear.
Solution
I checked that the SAN was not masking the LUN’s (it wasn’t, the default was read/write for everything). I connected to the console and proved the storage could be seen.
[box]
[root@ESX1:~] esxcli storage core path list------output removed for the sake of brevity------
sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014dfce99cd2d5401000000
UID: sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014dfce99cd2d5401000000
Runtime Name: vmhba3:C1:T1:L4
Device: naa.600c0ff00014dfce99cd2d5401000000
Device Display Name: HP Serial Attached SCSI Disk (naa.600c0ff00014dfce99cd2d5401000000)
Adapter: vmhba3
Channel: 1
Target: 1
LUN: 4 <-- First missing LUN
Plugin: NMP
State: active
Transport: sas
Adapter Identifier: sas.50014380388d8480
Target Identifier: sas.d0b8d32406430000
Adapter Transport Details: 50014380388d8480
Target Transport Details: d0b8d32406430000
Maximum IO Size: 4194304
sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014ddb44c57ac5401000000
UID: sas.50014380388d8480-sas.d0b8d32406430000-naa.600c0ff00014ddb44c57ac5401000000
Runtime Name: vmhba3:C1:T1:L5
Device: naa.600c0ff00014ddb44c57ac5401000000
Device Display Name: HP Serial Attached SCSI Disk (naa.600c0ff00014ddb44c57ac5401000000)
Adapter: vmhba3
Channel: 1
Target: 1
LUN: 5 <--Second Missing LUN
Plugin: NMP
State: active
Transport: sas
Adapter Identifier: sas.50014380388d8480
Target Identifier: sas.d0b8d32406430000
Adapter Transport Details: 50014380388d8480
Target Transport Details: d0b8d32406430000
Maximum IO Size: 4194304
------output removed for the sake of brevity------
[/box]
At this point I opened a support call with VMware and started doing other work while I waited for them to ring back. By the following morning I was still waiting, but I had found this article, I had built the new servers with HP Build versions of ESX, but perhaps I just needed to install the HP VAAI Plugin? I was fiddling with this when a nice chap called Supreet rang from VMware. I explained what I was trying to do, and got him WebEx’d on (I try not to waste a ton of time saying I’ve done X,Y, and Z, people do that to me all the time, and it just slows the process down, if anything I’d done was correct, it would have been fixed already!)
He confirmed the hosts were definitely seeing the storage;
Note: The bottom two are the missing ones. Using that information he had a look in the logs.
[box]
[root@ESX1:/var/log] grep -i "542eb3f8-da4ea518-553e-ac162d6f719c" vmkernel.log | less
[root@ESX1:/var/log] grep -i "54ad3e22-b39316bd-3e65-ac162d6f719c" vmkernel.log | less
[/box]
That showed up the following;
[box]
2017-03-20T16:23:16.754Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 542eb3f8-da4ea518-553e-ac162d6f719c: Not supported
2017-03-20T16:23:16.754Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 0 gen 0 :Not supported
2017-03-20T16:23:16.754Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 4 gen 1 :Not supported
2017-03-20T16:23:16.896Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 542eb3f8-da4ea518-553e-ac162d6f719c: Not supported
2017-03-20T16:23:16.896Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 0 gen 0 :Not supported
2017-03-20T16:23:16.896Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 542eb3f8-da4ea518-553e-ac162d6f719c FD 4 gen 1 :Not supported
2017-03-20T16:23:16.675Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 54ad3e22-b39316bd-3e65-ac162d6f719c: Not supported
2017-03-20T16:23:16.675Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 0 gen 0 :Not supported
2017-03-20T16:23:16.675Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 4 gen 1 :Not supported
2017-03-20T16:23:16.910Z cpu15:68106)WARNING: HBX: 2354: Failed to initialize VMFS distributed locking on volume 54ad3e22-b39316bd-3e65-ac162d6f719c: Not supported
2017-03-20T16:23:16.910Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 1 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 0 gen 0 :Not supported
2017-03-20T16:23:16.910Z cpu15:68106)Vol3: 3090: Failed to get object 28 type 2 uuid 54ad3e22-b39316bd-3e65-ac162d6f719c FD 4 gen 1 :Not supported
[/box]
That pointed him towards the VAAI, (perhaps the stuff I’d been reading, had me on the right track?)
[box]
[root@ESX1:/var/log] esxcli storage core device vaai status get
naa.600c0ff00014ddb44c57ac5401000000
VAAI Plugin Name:
ATS Status: unsupported
Clone Status: unsupported
Zero Status: supported
Delete Status: unsupported
naa.600c0ff00014dfce99cd2d5401000000
VAAI Plugin Name:
ATS Status: unsupported
Clone Status: unsupported
Zero Status: supported
Delete Status: unsupported
------output removed for the sake of brevity------
[/box]
Note the difference, the following is on the older servers that were working fine;
At this point, I piped up about the VAAI stuff I’d been reading, and told them that I’d download the VIB, and it was already on the offending server.
So they installed it and rebooted the server, (there were no running VMs on the new box).
I waited with bated breath, and it didn’t fix it 🙁 It was at this point that they gave me the bad news, HP P2000 G3 is not supported on ESX 6.5, (In fact its not supported on 6.0 either!)
Well I suppose that’s an answer, but not the one I wanted! I downgraded the hosts to 5.5U3a Same Problem! So I downgraded them to 5.0.0, then they wouldn’t boot, (error indicate unsupported hardware). So I set about upgrading them to 5.5U2, (to be on the safe side).
Also while this was going on, I updated the firmware on the SAN controllers;
Thankfully this time the servers booted up fine, and saw the storage and mounted all the datastores.
Related Articles, References, Credits, or External Links
