A few weeks ago I needed to sort out some extended warranty for a customers servers. To do that I needed the serial numbers of those servers, (a mixture of IBM/Lenovo and Dell Servers).
As I didn’t fancy a drive to two different datacenters, I wanted to try and get them programatically.
Solution
After some searching I came across a post by one of my old EE buddies LucD with exactly what I needed. I’m assuming you have PowerCLI setup before beginning.
Connect to your Virtual infrastructure;
[box]
Connect-VIServer {vCenter-server-FQDN}
[/box]
Then, (assuming you have a folder called C:\Temp that you can write to).
I’ve got a client that’s been having some performance issues with their VMs. Their storage vendor, (EMC) said that as a result of finding this in the logs;
[box]
B 02/28/19 09:50:53.953 scsitarg 117000e [INFO] System: iSCSI Logout Initiator Data: IP=192.168.200.161 Name=...-ec-21 Target Data: Port=2 Flags=0x00002002 Info=0x01200801
B 02/28/19 09:50:53.969 scsitarg 117000e [INFO] System: iSCSI Logout Initiator Data: IP=192.168.201.161 Name=...-ec-21 Target Data: Port=3 Flags=0x00002002 Info=0x01200801
B 02/28/19 09:51:16.413 Health 608fe [WARN] User: Host ESXi-01.petenetlive.com does not have any initiators logged into the storage system.
A 02/28/19 10:04:25.968 scsitarg 117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.200.161 Name=...-ec-21 Target Data: Port=2 Flags=0x00002002 Info=0x00000000 [Target]
B 02/28/19 10:04:26.034 scsitarg 117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.200.161 Name=...-ec-21 Target Data: Port=2 Flags=0x00002002 Info=0x00000000
A 02/28/19 10:04:31.996 scsitarg 117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.201.161 Name=...-ec-21 Target Data: Port=3 Flags=0x00002002 Info=0x00000000 [Target]
B 02/28/19 10:04:32.055 scsitarg 117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.201.161 Name=...-ec-21 Target Data: Port=3 Flags=0x00002002 Info=0x00000000
B 02/28/19 10:04:57.438 Health 608fc [INFO] User: Host ESXi-01.petenetlive.com is operating normally.
Host Host ESXi-01.petenetlive.com is accessing lun Datastore_3 as HLU 3, After the initiators for this host start logging in/logging, unit attention update events will be logged as the paths to the luns have changed this is expected
2019/02/28-09:50:41.607527 ~~~~ 7F3C92369703 std:TCD: Unit Attention update from 0000001A to 0001030D for LUN 0x3.
2019/02/28-10:02:55.860669 ~~~~ 7FE476E61702 std:TCD: Unit Attention update from 00010149 to 00010157 for LUN 0x3.
[/box]
We should disable DelayedAck and they kindly gave me the VMware KB that outlined the procedure.
Solution
The procedure outlined (for VMware 6.x) is to put the host in maintenance mode, then edit the properties of the iSCSI controller(s), untick the DelayedAck options, reboot the Host, and everything will be peachy. However, even though (post reboot) everything looks good in the the vSphere Web console. If you look on the host you may find something like this;
[box]
vmkiscsid --dump-db | grep Delayed
[/box]
DelayedAck = ‘1’ means ENABLED, DelayedAck = ‘0’ means DISABLED
So half my iSCSI entries in the iSCSI database still have DelayedAck ENABLED?
Some Internet searching told me this was quite common, and that the best way to ‘fix‘ it was to, disable the iSCSI initiator, remove the iSCSI database, reboot and then setup iSCSI again;
[box]
cd /etc/vmware/vmkiscsid
esxcfg-swiscsi -d
rm -f vmkiscsid.db
reboot
[/box]
Which is fine IF YOU ARE USING A SOFTWARE iSCSI INITIATOR, I however was not, I had 2x dedicated hardware iSCSI HBAs on each host!
After many hours of messing about and trial and error, it became clear, I had to do things in a certain order, or DelayedAck would simply just be enabled whether I liked it or not. 🙁
Disable DelayedAck With Hardware iSCSI NICs / HBAs
MAKE SURE THE HOST IS IN MAINTENANCE MODE FIRST
Then take a note of your iSCSI setup, Port Groups, VMKernel Ports, and Physical NICs, you are going to delete the iSCSI database in a minute, and you will need to ‘rebind’ the VMKernel Ports and add the iSCSI targets back in again.
Manually remove your iSCSI target(s) for ALL the iSCSI NIC/HBA’s
Below if you re-run the command, vmkiscsid –dump-db | grep Delayed you will see there’s still some entries in the database with DelayedAck enabled! So unlike above (see example for software iSCSI) we are going to remove the iSCSI database, only here we don’t need to disable the software iSCSI initiator (because we are not using one!) Finally reboot the host.
[box]
cd /etc/vmware/vmkiscsid
rm -f vmkiscsid.db
reboot
[/box]
When the host is back online ADD in the Network Port Binding for the appropriate VMkernel adaptor.
Like so;
DON’T RESCAN THE CONTROLLER AS PROMPTED TO DO SO!
On the Advanced Settings of EACH hardware iSCSI NIC/HBA > Edit > UNTICK ‘DelayedAck’.
Double check they are both still unticked (I’ve seen them re-tick themselves for no discernible reason!) Then rescan the controller(s).
Target > Add.
Re-add the iSCSI target back in, (that you took note of above).
Select the Target > Advanced > Untick the DelayedAck option (Note: This time it’s not inherited). Repeat for any additional iSCSI targets.
When they are all added, rescan the storage controllers again.
Finally recheck all the database entries are set to DISABLED.
[box]
vmkiscsid --dump-db | grep Delayed
[/box]
Related Articles, References, Credits, or External Links
Thanks to Russell and Iain for their patience while I worked all that out!
Assuming that’s all done, your task now is to upgrade your hosts, if you have a vanilla VMware ESXi version installed. You COULD simply SSH into the host, and execute the following commands;
Well that’s great but if you are using a custom ESXi image, the process won’t even start, you simply get warned that this won’t work, as it will break all the vendor specific drivers/software, (which is a good thing I suppose, the warning I mean, not the breaking things!)
Still, if you have a vendor modified copy of ESX what do you do? Well the following procedures will work for any version of the software, either vanilla VMware or HP/Dell/IBM etc. AND IT RETAINS ALL THE HOST SETTINGS, i.e. licences, vSwitches (standard and distributed), certificates etc.
Solution 1: Use an ‘Offline Bundle’ update
Firstly, you need a copy of the appropriate ‘offline bundle’ update, below you can see this is the customised one for HPE servers.
Upload the offline bundle into a datastore, (that the host to be upgraded has access to!) While in here, shut down the guest VMs on this host, and put the host into ‘maintenance mode‘.
NOTE: If you are updating an HPE Server, there will be a gen-9 and a pre-gen-9 update bundle! Pick the correct one!
Or, you can upload the bundle via SCP into the appropriate datastore, if you prefer.
SSH into the host you are going to upgrade. Below you can see me navigating to the Datastore;
And there’s my offline bundle ready to be installed.
Remember: Even if you’re in the correct directory, you need to specify the ‘full path’ to the ‘offline bundle‘ (or it looks in the ‘/var/log’ directory and won’t work). Execute the following command;
For a while it will look like nothing is happening, (don’t panic.) After a while a LOT of text will scroll past (quickly). Scroll back up to the TOP of all that text, and you are looking for, is ‘The update completed successfully‘.
Reboot the host.
Solution 2: Use VMware Update Manager
Warning: You CANT deploy an ESXi image, thats OLDER than the Update Manager you are running, i.e. if your vCenter is 6.5 you cant upload an ESX 6.0 image, (it will fail – this cost me two hours onsite!)
Warning 2: You need vCenter 6.0 Update1 or newer to perform this function.
I’ve got two hosts, one’s a Dell PowerEdge and the other a HP Proliant, I’ve already upgraded the HP server, you will see that in the screen shots below, now I’m going to upgrade the Dell.
Before Starting;
Download the install .ISO file that has the newer version of the ESXi software.
vMotion all the VM’s off the host being upgraded, (to save time).
If you have two hosts, (like me.) You might want to disable HA and DRS on the cluster as well temporarily!
If you’re running Cisco IOS on all you devices then you can use CDP to see what’s directly connected, (unless you are on a Cisco firewall, but I did say IOS devices).
[box]
Petes-Switch#show cdp neighbors
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
D - Remote, C - CVTA, M - Two-port Mac Relay
Device ID Local Intrfce Holdtme Capability Platform Port ID
Petes-CPE.petenetlive.com
Gig 1/0/19 148 R S I CISCO2901 Gig 0/0
Petes-Switch#
[/box]
But just about every other vendor uses LLDP (Link Layer Discovery Protocol,) to do the same job. So how do you discover what ports you are plugged into on them, or what they are?
Solution
Well in most cases, (depending on your code) your devices also support LLDP, it’s just disabled by default. You simply turn it on with an ‘lldp run’ command.
[box]
Petes-Switch(config)#lldp run
Petes-Switch(config)#exit
Petes-Switch#show lldp ?
entry Information for specific neighbor entry
errors LLDP computational errors and overflows
interface LLDP interface status and configuration
neighbors LLDP neighbor entries
traffic LLDP statistics
| Output modifiers
[/box]
Now I see my neighbours, (yes that is how it’s spelt I’m English!) In my example below the neighbour is a Dell Switch.
I promised to take a quick look at one of my mum’s friends laptops the other evening, the brief was; they had a problem opening office attachments, and their wireless mouse was’nt working. I assumed they needed the document conversion pack and probably had a flat battery, so I downloaded the update, and slung some batteries in my laptop bag.
The attachment problem was sorted (by re-installing office and changing some file assocations). But the mouse (Which was bluetooth) was somewhat more “fun” to fix.
Basically the laptop had been rebuilt some time ago, and since then the bluetooth would not work, normally I’d just download the drivers but that would have been too simple. Before I could get the mouse to work I needed to enable bluetooth on the laptop.
Solution
1. Stop searchig the chassis there is no switch to enable blueetooth, it can only be enabled by software.
4. It will dump all its files in C:delldriversR140135, once the files have extracted navigate to the C:delldriversR1401353100_216 folder and run setup.exe.
5. At the end it will tell you to enable wireless by pressing fn&F2 – THIS WONT WORK, click cancel.
6. Back in the C:delldriversR1401353100_216 folder you need to open the Win32 folder if your on a 32 bit machine or the Win64 folder if you on a 64 bit machine. (I dont know).
7. Locate the Inst.exe file and run it.
8. At the end it will once again ask you to press Fn and F2 – Do nothing, but leave the install sat there waiting.
9. With the install still waiting, navigate to “C:Program FilesWIDCOMMBluetooth Software” and locate BTTray.exe
10. Double click BTTray.exe and after a few seconds the bluetooth icon should appear in the taskbar (bottom right). Right click it and turn it on.
11. At this point, the install will burst back into life and setup your bluetooth wireless.
Related Articles, References, Credits, or External Links
I bought some SAS disks from eBay to go in my Power Edge 2900 Server, and scratched my head for a while trying to get the PERCRAID controller to use them. I could only see them marked as ‘Foreign’. Probably because they did not have their original config ‘wiped’ off them before I bought them.
Solution
1. Boot the server, Press CTRL+R when prompted, then press ‘C’ to enter the Configuration Utility.
WARNING: DONT clear the config on the controller or you lose ALL your Virtual Disks / Arrays.
After I got a new laptop, I attempted to connect to my test server and was greeted with this error.
Application Error
Unable to launch the application
Name: DRAC5 Console Redirection Client
Publisher: Dell Inc
Location: {ip-address}:443
So I hit the details tab and got this,
Error: Missing required Permissions manifest attribute in main jar:
https://{ip-address}:443/plugins/vkvm/avctDRAC5Viewer.jar
Solution
Luckily I had seen a similar error a couple of weeks ago, while a colleague was trying to set up a Cisco ASDM. This is a ‘security feature’ of Java version 7 (Update 51). Work had pre-built my new laptop, so I simply downgraded to Java version 7 Update 45. And it burst into life.
DRAC 5 Java Workaround
Currently Dell consider DRAC5 EOL (End Of Life) and will not fix the certificate issue with updated firmware. Their official response is ‘Maintain an older version of Java’. So either downgrade or lower the security settings;
1. Windows Key + X > Control Panel.
2. Java > Security> Set to Medium.
Related Articles, References, Credits, or External Links
The problem is, Java have pretty much blocked communications with SSLv3 (in light of the recent Poodle attack scares). Which is fair enough, but it’s not helping me get on my server that’s 50 miles away! The following procedure allows you to re-enable SSLv3 in Java (Tested on version 8 update 40). Obviously there is some security risks in doing so, this is as SSLv3 is exploitable, so best practice would be to just do this temporarily to get access then disable it again when no longer needed.</p?
1. You need to locate your java.security file, I’ve found paths all over the Internet to its location, the one that matters I found in the following locations;
MAC OSX: /Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/lib/security/java.security
2. Once located, edit the file and locate the line that reads;
[box]jdk.tls.disabledAlgorithms=SSLv3[/box]
MAC OSX Java.Security
Windows Java.Security
3. Simply remove SSLv3 from the end of the line and save the file. NOTE: With MAC OSX you will need to SUDO, to have the permissions to edit this file (see below).
Related Articles, References, Credits, or External Links
