People are always ‘a bit twitchy‘ about upgrading their backup software. Mines on my test network so I’m a little more cavalier about diving in and clicking next. So when you want to Upgrade Veeam 10 to Veeam 11, peoples main concerns are;
Will I lose my backup / replication jobs?
Will I need to rescan backup repositories?
Remember Veeam keeps all its data in a database, so your configs should remain unaffected but just to prove it.
Can I Upgrade Veeam 10 to Veeam 11 – Yes
Obviously close any Veeam management windows (in all logged in profiles) before starting.
‘Backup the Free Version of ESX?’ This was asked on Experts Exchange this week, I responded with the usual answer of “No the VAPI is not exposed in the free version, so it cannot be backed up by a third party backup application.”
The poster responded with, “That’s what I thought, but I’ve got a Technical Consultant here who says it can be done with version 7“
This should be easy to test, I simply need to spin up an ESX7 VM put a free licence on it and see if I can back it up right?
Backup the Free Version of ESX ?
Off on a Tangent: Things always go wrong the most when you are doing someone a favour! First my test machines (In a corporate data centre, didn’t have CPUs compatible with ESX7. No problem I’ll spin one up on my macbook in VMWare Fusion, and attempt a backup from the DC (I have a site to site VPN to the DC). Then none of my fusion machines could get network access, then when I fixed that, the VPN was down and needed fixing. Then I was in the middle of a Veeam upgrade anyway, that took about 12 hours!
Back to the answer: Whoever started this rumour needs a percussive adjustment to the face, free ESX still cannot be backed up, here’s what happens when you try with Veeam for example;
Veeam Error:
Error: Current vSphere license or ESXi version prohibits execution of the required operations
So I Can’t Backup the Free Version of ESX?
Well not using a VMWare aware backup solution, but theres nothing to stop you installing backup agents into your VMs, and backing them up that way, (treat them as if they are Physical machines.)
Alternative Backup Solution: I’ve seen people suggesting using GhettoVCB which might fit your requirements, but you need to install it into the ESX server as a VIB, then you can create a folder in your storage. Finally you can then script the VIB to take snapshots of your VMs, and then copy the backups into that folder, but it requires some CLI skills and it leaves your backups on the same box as you production VMs. It you’re an enthusiastic tech type (with no budget) you might want to take a look at it.
Related Articles, References, Credits, or External Links
At PNL we have always championed Veeam, below are our links to download Veeam, Theres no surprise it’s the market leader in backup and recovery, it came onto the market when backups were a major IT headache. I can remember having to change nearly 30 different server tapes (a day) and from 09:00 to probably lunch time every day I was fixing backup issues. (Simply Download Veeam and give it a trial!)
Now we still have backup issues, (don’t get me wrong) but with the correct product and some planning, design and forethought Backups these days are less painful and that’s due in no small part to the Veeam product suite.
Disclaimer: This is an affiliate sponsored document. By downloading Veeam products from these links you will support the upkeep of this site.
B&R The one you are probably looking for!
This product is for all Virtualised and Physical environment based servers. It is the backup product that we use and stand behind.
Manage, protect, and recover workloads within Azure AWS.
Why Buy Veeam?
A myriad of reasons;
Supports vSphere and Hyper-V
Instant VM recovery (from backup media into production)
Replication is included (often overlooked but this gives you a faster recovery method in the event of hardware failure and can also be used to recover files from). The product is called ‘Backup and Replication’ after all!
Instant file level recovery.
Application Aware backups (Exchange, SQL).
Simple offsite backup.
Built in compression and deduplication.
Simple to deploy and monitor.
Hardware agnostic.
Related Articles, References, Credits, or External Links
I’ve always been a fan of Veeam, I’ve championed it for years, as a consultant and engineer I want solutions that are easy to deploy, administer, and upgrade, that cause no problems. Like all things that are easy to use, and gain a lot of popularity, Veeam is starting to get DESTROYED BY DEVELOPMENT. What do I mean? Well, things that were simple and easy to find now require you to look at knowledge base articles and pull a ‘frowny face’. Also the quality of support has gone dramatically downhill. We stand at the point where another firm can come in and do what Veeam did, (march in and steal all the backup & replication revenue worldwide, with a product that simply works and is easy to use).
I digress (sorry). So you want to backup to public cloud yes?
Well then, you log into Veeam look at your backup infrastructure, and simply add an External Repository and backup to that? NO! That would be common sense, (and the way Veeam used to to things). External Repositories are not for that, Veeam points this out when you try and add one;
So how do you backup to public cloud? (I know other vendors are available, but we are talking primarily about Azure and AWS). Well to do that you need to be more familiar with Scale Out Backup Repositories (SOBR).
With an SOBR you can add ‘cloud storage’ i.e. Azure Cold Blob storage or AWS S3, as ‘Capacity Tier‘ storage. How is the Capacity Storage Tier Used? Well theres two options, ‘Backup to Capacity after x Days’ or ‘Backup to Capacity Tier as soon as backup are created‘. like so;
Send your backup to a Scale Out Backup Repository.
The backup gets placed into the Performance Tier.
Option 1: Copy to Cloud after x Days, or Option 2: Copy to cloud immediately.
Note: This is configured on the SOBR configuration NOT on individual backup jobs/sets.
Adding Azure Cold Blob Storage
Well before you can add cloud storage to a SOBR you need to add it to Veeam, how’s that done? Well firstly you need to create an Azure Storage account.
Then generate an ‘Access Key‘.
Then create a ‘Container‘ in your storage account.
Then within Veeam > Options > Manage cloud credentials > Add > Add Azure Storage Account > Enter the Storage account and Access Key > OK.
Adding ‘Cloud Storage’ as ‘Capacity Tier’ to a Scale Out Backup Repository
Either create a new Scale Out Backup Repository, (Backup Infrastructure > Scale Out Backup Repository,) or edit an existing one. When you get to Capacity Tier > Tick the ‘Extend..’ option > Add > Microsoft Azure Blob Storage.
Azure Blob Storage > Give the storage a name > Next.
Select the storage account you created above > Select your Gateway Server (usually the Veeam B&R server but it does not have to be) > Next > Browse.
Select or create a new folder > Limit the amount of space to use (if required) > Next > Finish.
What about AWS? Well Microsoft kindly give me a certain amount of ‘free‘ Azure credits every month so it’s easy to showcase their product, (I use this for learning and PNL tutorials), so Microsoft pretty much get the benefit. I know AWS have a free tier and a trial tier, but honestly after spending 2 hours trying to find out what you actually get, and am I going to get stung on my credit card bill If I do ‘xyz‘ I lost all interest!
AWS, be like Veeam used to be, make it easy! AWS is like flying with Ryanair,
‘Oh so you want a seat? That will be and extra £x a month, and for every trip to the toilet will be an extra £x a month. Will you be wanting nuts? Because we charge by the nut, and no one knows how many nuts are in each bag, so it will be different every time, and speaking of time if you want to look at the clock that will be £x a month also!‘
People will email me and complain Azure is the same, and to an certain extent I will agree, but nothing will change until, public cloud providers start charging fixed prices for things, so IT departments can work out what the Opex is going to be e.g. like private cloud providers do! Of course working for a private cloud provider maybe I’m a little biased?
Related Articles, References, Credits, or External Links
While setting up some new backups, all the virtual machine failed like so;
Unable to allocate processing resources. Error: No backup proxy is able to process this VM due to proxy processing mode restrictions.
Solution
On the face of it this looks like a simple problem, either you backup proxy does not have the capacity, you have too many jobs running at one, or you need to deploy some more backup proxies.
However, I tested this by creating a new job with one ‘problem’ VM in it and made sure no other jobs were running. It failed with the same error. So it’s got nothing to do with lack of resources!
The Actual Problem: Was due to the fact that the backup proxy (a physical machine) had direct fiber access to the storage array, and its backup mode was set to ‘Direct Storage Access’. These new VMs were in a different Datastore/LUN that the Veeam server could not see! So you can either tick ‘Failover to network mode if primary mode fails or is unavailable’
Or a better option would be to present the correct storage LUNS to the Veeam Backup server.
Related Articles, References, Credits, or External Links
I got tasked with looking at a failover plan for a customer recently, but I noticed their replications were failing, two of the VMs in the replication job were repeatedly failing with;
Error: Storage not initialized
Solution
This took a while to get to the bottom of! Essentially the problem was the ‘Metadata’ for these two VMs was either corrupt or could not be written to. So to fix the problem, the VMs had to be removed and re-added, though the procedure for this has to be done in the following sequence;
From the Replicas Ready section remove the affected VMs from configuration.
Go to the ‘Replica VM‘ itself (within your hyper visor), and remove all the snapshots.
Edit the replication job, and add the affected VMs back in again.
Make Sure: You repoint the Replica Mappingback to your original replica VM again, (simply hitting ‘Detect‘ will do this, 99 times out of 100).
Related Articles, References, Credits, or External Links
When adding repositories to a Veeam Scale Out Backup Repository you may see this error;
Unable to add extent {Repository-Name} because it serves as the target for one or more job types which are not supported by a scale-out backup repository
Selecting ‘Show jobs‘ shows;
Backup-Configuration-Job
Solution
If you didn’t already know, the Backup-Configuration-Job, is a backup of the Veeam database and job settings, it’s what you would use if you wanted to reinstall or migrate Veeam to another server. It gets created automatically, and gets put into your first created repository. Note: It’s considered good practice NOT to have this backup stored actually on the Veeam server, but if your only repositories are on the Veeam server, you don’t have much choice!
To Fix: Simply create a new repository just for the Configuration-Backup-Job, it can be on the same Storage/Array/Disk/Partition as an existing repository (as you can see below). I typically name the repository Config-Backup-Repository to avoid confusion in the future.
Now simply change the job to use the new repository, this is NOT done where all the other jobs are configured! Select Options > Configuration Backup > Change the Repository > (I manually run it, by clicking Backup-Now) at this point, just to make sure all is well.
You should now be able to create your Scale Out Backup Repository without an error.
Related Articles, References, Credits, or External Links
EVERY VMware Job in Veeam Backup and Recovery failed with the following error, (for every Guest VM);
Processing SERVER-NAME Error: Failed to open VDDK disk [[DATASTORE-NAME] SERVER-NAME/SERVER-NAME.vmdk] ( is read-only mode – [true] ) Logon attempt with parameters [VC/ESX: [ESXi-SERVER-NAME.DOMAIN-NAME.COM];Port: 443;Login: [administrator@.vsphere.local];VMX Spec: [moref=vm-231];Snapshot mor: [snapshot-288575];Transports: [san];Read Only: [true]] failed because of the following errors: @Failed to open disk for read. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.
Solution
The fact that I had upgraded the vCenter from 6.5 to 6.7 the same day is probably a clue! reconnecting and rescanning the vCenter did NOT resolve the problem.
Fix: Update Veeam Backup and Recovery, (In my case to 9.5 update 4b.)
Stop and disable the Veeam Services.
Reboot the server, (or go into task manager and kill and running Veeam processes)
Mount the install ISO and carry out the upgrade.
Reboot.
Problem solved!
Related Articles, References, Credits, or External Links
If you require a ‘Virtual Lab’ for testing patches or config changes, on copies of your live servers, or simply want to test the ‘integrity‘ of your backups, then this is the post for you!
Licence Requirements: SureBackup and On Demand Sandbox require Enterprise Plus Veeam Licensing.
Host Licences: Hosts that are only used for SureBackup/ On Demand Sandbox DO NOT NEED Licences, (in Veeam,) only hosts that you back up FROM need licences.
SureBackup and Virtual labs are built on vPower, which allows you to power on your ‘backup files’ in a test/sandbox environment. It’s actually the same technology that Veeam use for U-AIR recovery.
Three components make up a virtual lab;
1. Application Group: This is a group of VMs, and the ‘Order’ they need to be powered on, e.g. for Exchange server you would also need a DC (global catalog server,) and maybe your mail filter appliance to be in the same group.
2. Virtual Lab: Requires a ‘Host’, and a DataStore, (for redo logs only), this only needs to be 10% of the size of the VMs that are being powered on in the lab.
3. SureBackup: This is the process that ‘Tests backups‘, it will bring your backed up machines online, and perform some tests on them, some are simple like ‘ping’ tests others are specific to particular server roles, like additional tests for Domain Controllers, Exchange servers etc.
Here’s how it all ‘hangs together’. We are backing up a Domain Controller, and an Exchange Server, and we are going to use those backup files to power on a copy of the servers in our ‘Test-Lab’.
Note: I’m using VMware ESX, you can also use Microsoft Hyper-V.
These are presented though a ‘Veeam Proxy Appliance’, which presents them to the VEEAM server with a changed ‘octet’ in their IP address. (So by default any other machine needs a static mapping, {see below}).
Create a Veeam SureBackup Application Group
As mentioned above, make sure you have ‘Enterprise Plus‘ licences.
It should go without saying, but you will also need a ‘good’ backup of your servers.
Backup Infrastructure > SureBackup > Application Group > Add App Group > VMware.
Give the app group a name > Next > Add VM > From Backup > Select the VMs for the Lab > Add Next.
Put the server(s) in the correct order, i.e. the domain controllers at the top.
If you are just going to use SureBackup to check backups, then ‘Edit’ the servers, and change their ‘role’ so the correct tests get performed on them. If you are just wanting a Virtual Lab, don’t bother as you will be interacting with them directly anyway. Here are the settings for a Domain Controller.
Give the lab a name > Next > Choose > Select the ‘Target’ ESX Server to use > OK > Next > Choose > Select a datastore for the ‘redo’ logs, remember this needs to be about 10% of the size of the restored VMs. > OK > Next.
Next > ‘Advanced Single Host’ > Next > Add > Browse to the ‘Port Group’ your production VMs are in > Add > OK > Next.
Note: If you need to have your lab network on its own VLAN, this is where you need to specify that traffic to be ‘tagged’ accordingly.
Add > Specify the IP for the ‘inside’ of your Veeam Proxy Appliance, this MUST BE the same as the default gateway on the live network. Then select a sensible masquerade network address > OK > Next.
Veeam: What’s a Masquerade Address?
The proxy server basically will perform NAT from the test lab to the live network, (their actual IP addresses never change, that’s why the proxy appliance had the same IP as the default gateway on the live network. The Masquerade addresses simply change one ‘octet’ of the IP address so the Veeam server can speak directly to each sand-boxed, (Test lab) VM.
If required, Add a ‘Static Mapping‘ i.e if you want to be able to ‘speak’ to a test lab VM from the live network.
How Do Veeam Virtual Lab ‘Static Mappings’ Work?
Using the example, I used above, here if someone on the live network speaks to 192.168.100.21, they are actually talking to 192.168.100.196 in the test lab.
Apply > Finish.
Create a Veeam SureBackup Job
There are two ways of doing this, if you want to create a SureBackup job that just checks your backups, then you would schedule the job, and connect it to your backups, or if you just wanted to do some lab testing, you would create a ‘one off’ SureBackup job and leave the VMs powered on (I’ll point this out below).
Home > SureBackup Job > VMware > Give the job a name > Next.
Select the lab you created above > Next > Select the App Group you created above. (NOTE: If you want to leave your machines ‘powered on’ after the job, i.e. for performing upgrades, patch tests etc, then TICK the option indicated).
Link this job to the backup job for the VMs in question > Add > Select the backup Job > OK.
Note: The option at the bottom, specifies how many VMs are tested at a time in a standard SureBackup Job.
Next > Next.
Schedule the job (if required) > Apply > If you didn’t schedule, then you can click ‘Run the job when I click Finish‘ for ‘one-off’ jobs > Finish.
If you selected the option to leave the machines powered on, then there will ‘always’ be a job running and the job will stop at 99%. (You will need to manually stop the job to remove the test VMs). If you do continuous backups this will be a familiar sight anyway!
There’s my test VMs powered on, that I can interact with, update, patch, and change configurations, without it affecting my live servers.
Related Articles, References, Credits, or External Links
Natively with Veeam Backup and Restore, the only way to do this, is to perform an Agent Based backup of your VMware VM. If you are unfamiliar with doing this, you create a Protection Group, and add the machines to that protection group. It’s really designed for backing up physical machines, but there’s no reason you can’t backup a VM in the same way.
Just so you know it’s not smoke and mirrors. here’s my VM in vSphere.
Perform an ‘Agent Based‘ backup off this VM.
Restore > Agent Restore > Instant Recovery.
Select the VM > Next > Set the Hyper-V server as the ‘Host’.
Finish the ‘Wizard’, and perform the restore.
You will need to complete the process by ‘Migrating to production’.
Wait for the process to complete.
And now it’s running in Hyper-V.
I’d run devmgmt.msc and make sure all the drivers are OK. That being said, anything after server 2008 should have the correct drivers for Hyper-V, and Veeam does not support agent-based backups on Server 2008 (or earlier) so all ‘should’ be fine.
Related Articles, References, Credits, or External Links
