Over the past few weeks I’ve been doing some work for a client that involves me adding some new ESX hosts. These will be setup the same as their existing ESX hosts. That’s fine, but they use standard vSwitches and have A LOT of port groups and VLANs.
I could sit and create them all manually, and tag them onto the the correct VLANs, but something that’s boring and repetitive is better solved with a bit of scripting!
Pre Requisites: You will need VMware PowerCLI installing (Install-Module -Name VMware.PowerCLI)
Solution
Connect to your vCenter
[box]
Connect-VIServer server-name.domain-name.com
[/box]
Note: If it complains about certificates (see below) issue this command ‘Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false‘ then try to connect again
Connect-VIServer Error: Invalid server certificate. Use Set-PowerCLIConfiguration to set the value for the InvalidCertificateAction option to Prompt if you’d like to connect once or to add a permanent exception for this server.
Note2: If migrating between hosts on different vCenters, then Connect-VIServer to BOTH vCenters before proceeding.
Once connected use the following two commands to specify your existing SOURCE ESX server name, and your new TARGET ESX server names, change yours accordingly.
Now, to copy and create all the port groups and VLANs use the following command. Note: Change the vSwitch numbers accordingly to match the source and destination vSwitch you are copying from and to.
You can observe them getting created in the vSphere web client like so;
Note: If you are migrating ‘vanilla’ Port Groups, you may see a couple of errors fly past if it tries to create ‘Management Network‘ and ‘VM Network‘ as these exist by default, so it can’t recreate them.
Related Articles, References, Credits, or External Links
EVERY VMware Job in Veeam Backup and Recovery failed with the following error, (for every Guest VM);
Processing SERVER-NAME Error: Failed to open VDDK disk [[DATASTORE-NAME] SERVER-NAME/SERVER-NAME.vmdk] ( is read-only mode – [true] ) Logon attempt with parameters [VC/ESX: [ESXi-SERVER-NAME.DOMAIN-NAME.COM];Port: 443;Login: [administrator@.vsphere.local];VMX Spec: [moref=vm-231];Snapshot mor: [snapshot-288575];Transports: [san];Read Only: [true]] failed because of the following errors: @Failed to open disk for read. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.
Solution
The fact that I had upgraded the vCenter from 6.5 to 6.7 the same day is probably a clue! reconnecting and rescanning the vCenter did NOT resolve the problem.
Fix: Update Veeam Backup and Recovery, (In my case to 9.5 update 4b.)
Stop and disable the Veeam Services.
Reboot the server, (or go into task manager and kill and running Veeam processes)
Mount the install ISO and carry out the upgrade.
Reboot.
Problem solved!
Related Articles, References, Credits, or External Links
This weekend I had to upgrade some servers from VMware 5.5 to version 6U3, (that’s the latest supported version, on the hardware I was using: HP BL460c). As the VC was at version 6.5 I couldn’t use the built in update manager, so I had to use the ‘offline bundle’ method.
TWO THINGS then jumped up and bit me;
Problem 1
[DependencyError]
File path of ‘etc/init.d/hp-mst.init’ is claimed by multiple overlay VIBs: set([‘Mellanox_bootbank_net-mst_4.3.0.29-10EM.550.0.0.1331820’, MEL_bootbank_nmst_1.3.0.29-10EM.600.0.0.2768847′[)
Problem 2
[DependencyError]
VIB VMware_bootbank_esx-base_6.0.0-3.100.931334 requires vsan >= 6.0.0-3.101, but the requirement cannot be satisfied within the ImageProfile.
VIB VMware_bootbank_esx-base_6.0.0-3.100.931334 requires vsan << 6.0.0-3.101, but the requirement cannot be satisfied within the ImageProfile.
Please refer to the log file for more details.
Solution
Both these problems BOTH need to be dealt with separately;
Solution to Problem 1
Easy one, execute the following command (Note: Despite what it says, you don’t need to reboot the host!)
[box]
esxcli software vib remove -n net-mst
[/box]
Solution to Problem 2
I’d been bitten by this one in the past, so luckily I knew what the fix was, you see this a lot with custom, (vendor) VMware ESX images HP, and Dell for example. To proceed you need to specify which ImageProfile to use, and you get that from your update bundle, like so
[box]
esxcli software sources profile list -d /vmfs/volumes/{volume-name}/VMware-ESXi-6.0.0-Update3-9313334-HPE-preGen9-600.9.8.5.4-Sep2018-depot.zip
[/box]
It will display the ImageProfile name, copy that to the clipboard;
Note: In the example (above) the ImageProfile name is HPE-ESXi-6.0.0-Update3-preGen9-600.9.8.5.4
Then you re run the upgrade command, BUT you specify the ImageProfile with a -p switch, like so;
This is a strange one? I was deploying FirePOWER to a pair of ASA 5550-8-X firewalls in Active / Standby failover last week. After each SFR was updated (via ASDM.) I could no longer ‘ping it’, the SFR itself could ping everything on the same VLAN, APART from its own default gateway, (which was an SVI on the Cisco 3750 switch it was connected to).
This happened every time I updated the SFR, (or re-imaged it.) Then after an hour or so it was fine?
Solution
If I connected to the switch that the SFR, (and firewall) was connected to, I could NOT ping the SFR. The interface was up/up on the switch, and the firewalls Management interface was also up/up.
[box]
Petes-3750#ping 10.2.1.252
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.2.1.252, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
[/box]
I did notice it was in the ARP table though, (with the correct MAC address), So I manually removed it;
[box]
Petes-3750#clear ip arp 10.2.1.252
[/box]
Then it was fine?
[box]
Petes-3750#ping 10.2.1.252
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.2.1.252, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
If you have a lot of IIS servers, and want to load balance between them, then you can either buy a load balancer, or use Microsoft ARR (Application Request Routing). Note: ARR does a lot more than simply load balancing, e.g. it can perform caching, and complex web routing, and even SSL offloading. Here we are just looking at load balancing.
I’m going to deploy TWO ARR servers in my DMZ, here I’ve got two ‘back-end’ IIS web servers, (you may have many more.)
WHY ARR?: Rather than use WAP (Web Application Proxy,) or a connection broker, ARR is application aware, i.e. it WONT attempt to serve pages from a broken IIS server, e.g. if the host server is online, but the site to IIS is broken for instance.
WHY TWO?: Well we are talking about balancing and availability, I’m deploying two so the event one fails, the other one will still be online, you can have these running on different hypervisors, or even in different datacenters, for added resiliency.
Deploy Network Load Balancing (NLB)
Our first task is to deploy NLB this will create a ‘Virtual IP’ for both of the ARR servers to use.
NLB is a ‘Feature‘ to enable it, launch Server Manager > Manage > Add Roles and Features > Next > Next > Next > Next > Tick ‘Network Load Balancer‘ > Next > Next > Finish.
Launch ‘Network Load Balancing Manager’.
New Cluster.
Add in the first host > Connect > Next.
Check the IP > Next.
Add a ‘Cluster IP’, (this is the IP that you will connect to for services), and is the ‘Shared’ IP > OK > Next.
Next.
Next.
Repeat the procedure to add the additional IP(s).
You will need to make the NLB IP is ‘Publicly Available,’ and open HTTP/HTTPS as required. Also the ARR hosts will need HTTP/HTTPS (as required) open to the internal IIS servers. I usually test all that at this point.
Deploy ARR and ‘URLRewrite’ for Load Balancing
ARR and URL rewrite are both IIS components, but you don’t need to install IIS yourself. You can if you wish, and then install URL Rewrite THEN ARR (In that order!) But it’s much simpler to download and use the ‘IIS Web Platform Installer‘.
Launch the Web Platform Installer, and do a search for URL > Select URL Rewrite > Add > Repeat the process, searching for ARR, and add Application Request Routing version 3, (Not the 2.5 version at the top!) > Next > Follow the wizard and complete the install.
Launch IIS Manager > Now you will see you have a new option ‘Server Farm‘ > Create Server Farm.
Give your server farm a name > Next > Add in all the ‘Back-end’ IIS servers > Finish.
You will get a pop-up asking if you want to create a URL rewrite rule. In this case we want a simple rewrite rule as we are doing plain old load balancing and we have no special requirements, so Select YES. (Only click No if you have specific rewrite requirements and you want to set them up manually).
Now test externally. WARNING don’t expect the page to ‘flip over’ every time, remember ARR is caching these web requests, and your browser will also be performing web page cashing, use a couple of browsers and wait a minute or two between refreshes to make sure that all the web servers are being used!.
Related Articles, References, Credits, or External Links
If you require a ‘Virtual Lab’ for testing patches or config changes, on copies of your live servers, or simply want to test the ‘integrity‘ of your backups, then this is the post for you!
Licence Requirements: SureBackup and On Demand Sandbox require Enterprise Plus Veeam Licensing.
Host Licences: Hosts that are only used for SureBackup/ On Demand Sandbox DO NOT NEED Licences, (in Veeam,) only hosts that you back up FROM need licences.
SureBackup and Virtual labs are built on vPower, which allows you to power on your ‘backup files’ in a test/sandbox environment. It’s actually the same technology that Veeam use for U-AIR recovery.
Three components make up a virtual lab;
1. Application Group: This is a group of VMs, and the ‘Order’ they need to be powered on, e.g. for Exchange server you would also need a DC (global catalog server,) and maybe your mail filter appliance to be in the same group.
2. Virtual Lab: Requires a ‘Host’, and a DataStore, (for redo logs only), this only needs to be 10% of the size of the VMs that are being powered on in the lab.
3. SureBackup: This is the process that ‘Tests backups‘, it will bring your backed up machines online, and perform some tests on them, some are simple like ‘ping’ tests others are specific to particular server roles, like additional tests for Domain Controllers, Exchange servers etc.
Here’s how it all ‘hangs together’. We are backing up a Domain Controller, and an Exchange Server, and we are going to use those backup files to power on a copy of the servers in our ‘Test-Lab’.
Note: I’m using VMware ESX, you can also use Microsoft Hyper-V.
These are presented though a ‘Veeam Proxy Appliance’, which presents them to the VEEAM server with a changed ‘octet’ in their IP address. (So by default any other machine needs a static mapping, {see below}).
Create a Veeam SureBackup Application Group
As mentioned above, make sure you have ‘Enterprise Plus‘ licences.
It should go without saying, but you will also need a ‘good’ backup of your servers.
Backup Infrastructure > SureBackup > Application Group > Add App Group > VMware.
Give the app group a name > Next > Add VM > From Backup > Select the VMs for the Lab > Add Next.
Put the server(s) in the correct order, i.e. the domain controllers at the top.
If you are just going to use SureBackup to check backups, then ‘Edit’ the servers, and change their ‘role’ so the correct tests get performed on them. If you are just wanting a Virtual Lab, don’t bother as you will be interacting with them directly anyway. Here are the settings for a Domain Controller.
Give the lab a name > Next > Choose > Select the ‘Target’ ESX Server to use > OK > Next > Choose > Select a datastore for the ‘redo’ logs, remember this needs to be about 10% of the size of the restored VMs. > OK > Next.
Next > ‘Advanced Single Host’ > Next > Add > Browse to the ‘Port Group’ your production VMs are in > Add > OK > Next.
Note: If you need to have your lab network on its own VLAN, this is where you need to specify that traffic to be ‘tagged’ accordingly.
Add > Specify the IP for the ‘inside’ of your Veeam Proxy Appliance, this MUST BE the same as the default gateway on the live network. Then select a sensible masquerade network address > OK > Next.
Veeam: What’s a Masquerade Address?
The proxy server basically will perform NAT from the test lab to the live network, (their actual IP addresses never change, that’s why the proxy appliance had the same IP as the default gateway on the live network. The Masquerade addresses simply change one ‘octet’ of the IP address so the Veeam server can speak directly to each sand-boxed, (Test lab) VM.
If required, Add a ‘Static Mapping‘ i.e if you want to be able to ‘speak’ to a test lab VM from the live network.
How Do Veeam Virtual Lab ‘Static Mappings’ Work?
Using the example, I used above, here if someone on the live network speaks to 192.168.100.21, they are actually talking to 192.168.100.196 in the test lab.
Apply > Finish.
Create a Veeam SureBackup Job
There are two ways of doing this, if you want to create a SureBackup job that just checks your backups, then you would schedule the job, and connect it to your backups, or if you just wanted to do some lab testing, you would create a ‘one off’ SureBackup job and leave the VMs powered on (I’ll point this out below).
Home > SureBackup Job > VMware > Give the job a name > Next.
Select the lab you created above > Next > Select the App Group you created above. (NOTE: If you want to leave your machines ‘powered on’ after the job, i.e. for performing upgrades, patch tests etc, then TICK the option indicated).
Link this job to the backup job for the VMs in question > Add > Select the backup Job > OK.
Note: The option at the bottom, specifies how many VMs are tested at a time in a standard SureBackup Job.
Next > Next.
Schedule the job (if required) > Apply > If you didn’t schedule, then you can click ‘Run the job when I click Finish‘ for ‘one-off’ jobs > Finish.
If you selected the option to leave the machines powered on, then there will ‘always’ be a job running and the job will stop at 99%. (You will need to manually stop the job to remove the test VMs). If you do continuous backups this will be a familiar sight anyway!
There’s my test VMs powered on, that I can interact with, update, patch, and change configurations, without it affecting my live servers.
Related Articles, References, Credits, or External Links
