This is arguably the most boring post I’ve ever written. No graphics, lots of references to my own unique environment. But this is the real world of backup and recovery and I thought I’d share my experience. When was your last multi-machine recovery test?
No matter how over-the-top you are about creating backups, if you can’t restore them, you’re toast. I decided today to test restoring my main Hyper-V server with my main guest computer.
The host runs Server 2016, not domain joined. Its Windows Server backup is written to a separate internal backup drive, which is then copied to an external backup drive. The guest VHDX files are excluded from the host backup.
The guests runs various operating systems—see below. Their backups are either written to a VHDX file, which is copied nightly to the external backup drive, or in the case of Veeam Windows backup, the backup files are copied directly to the external drive,
I’m restoring to the same model server as the original (Lenovo TS140). The target has less RAM, so I wouldn’t be able to run all of my VMs, but I should at least be able to get the primary VM up.
Host Restore
The host restore went quickly: boot from Server 2016 on a USB drive and use System Image Recovery. The small C drive and some batch files were restored.
Quick, unplug the target’s network cable so you don’t have two servers on the network with the same name.
Guest Restores
The guest restore got a little tricky.
First, some message about services not starting made me realize that I needed to enable all the VT-x etc. switches in the BIOS. (The target had never been used as a Hyper-V server before.)
After starting Hyper-V, I had a long list of machines in a Saved-Critical state with the status, “Cannot connect to virtual machine configuration storage.” Well yeah, the VM data drives don’t exist yet.
Guest 1 Restore
This guest is Server 2016 Essentials, a domain controller, backed up with Windows Server Backup.
Annoyingly, you can’t view or edit the VM’s settings in the Saved-Critical state, so I created a new Generation 1 VM with just a C drive. I also attached as a SCSI drive the VHDX containing the guest’s backup.
I fortunately had an ISO for Server 2016 Essentials on the external drive. I booted my new VM from that ISO, but the restore wouldn’t work because it wasn’t UEFI. Oh.
Delete the Version 1 VM and re-create it as Version 2. Under Security, confirm UEFI boot is enabled and turn on the (virtual) TPM.
Try the restore again. This time, it can’t boot “because the host’s isolated user mode is off.” Online instructions for turning it on are apparently out of date; see this thread. Sensing another rabbit hole forming, I finally got around this by just turning off the vTPM. I think I’ll still be able to use BitLocker in the guest if I need it, at least temporarily, by just providing the unlock key. (I only use BitLocker on one pass-through disk, the one I use for Client Computer Backups.) Once I did that, I was able to restore the OS disk of the guest and start the guest. And once I did THAT, I was able to start Windows Server Backup inside the guest and restore the data volume to a separate VHDX.
One wee problem after restoring the data drive is that all the shared folders were no longer shared. A server reboot brought them back.
So that’s got the 2016 Server Essentials (domain controller) computer running.
Guest 2 Restore
Okay how about a Server 2016 SQL Server? This one I back up with the free Veeam Windows Agent. I had no trouble booting form the Veeam Recovery ISO, but I did have trouble seeing the Veeam backup files on the external hard drive—I wasn’t able to fool the Recovery environment into seeing the host’s drives as UNC drives. (Maybe if I had already set up a virtual switch?) I finally got it to work by creating a new VHDX virtual drive, attaching it to the guest, and copying the Veeam backup files into the VHDX.
Alas, SQL Server would not start. I finally noticed that it was looking for SQL data on drive D, but that is the drive that the guest assigned to the backup drive containing the Veeam files. I changed that drive’s letter, then I had to put the real Drive D online. (Veeam had restored it but apparently left it offline due to the drive letter conflict.)
The other issue was networking. The restored machines were connected to the hosts physical adapter, but that was not online. I decided to set up an “Internal” virtual switch in Hyper-V (very simple to do) and to connect the machines to that. Then I assigned static IPs to both machines, and set DNS for both to point to the primary machine. Once that was configured, SQL server worked, and I was able to run an app on the first machine that used a SQL database on the second machine.
Guest 3 Restore
What about restoring a Server 2012R2 machine, also running SQL? Like Guest 1, this one was backed up with Windows Server Backup. I hit two snags:
- This was a Generation 1 machine and would not restore to a Generation 2 (UEFI) VM. I had to create a Generation 1 machine.
- At least during the restore, the OS VHDX must be attached as an IDE disk. I had initially attached it as a SCSI disk, but I kept getting the message, “No disk that can be used for recovering the system disk can be found.”
The next issue wasn’t really a snag, but it slowed things down. The SQL data or this machine, stored on drive D, is fairly large, so I only copy the backups to the external backup drive, without backing up drive D directly. It is recoverable, but it means re-creating D:, copying the backups from the external drive to the VHDX, then restoring the databases from the backups. And because not even master.dbf is there, you can’t use the SQL GUI (SQL itself won’t start); you have to do a full disaster recovery scenario. I think I’ll change this Windows Server Backup plan to include the D: drive with the databases, just excluding the (large-ish) backup files.
Guest 4 Restore – Essentials Client
The final test is whether I can restore a client computer using the Windows Server 2016 Essentials Client backup/recovery tools.
First step is to copy 800GB of Server Folders\Client Computer Backups folder from the external hard drive to a large internal drive in the Hyper-V server. (Turn on BitLocker first so it will be easy to wipe this test drive.)
Note If you reboot the server, be sure to unlock this drive again from inside the Essentials guest or the Windows Server Essentials Computer Backup Service will fail to start at restores will fail.
Now mount that large drive as a pass-through disk in Hyper-V settings for Guest 1 (the 2016 Essentials server).
The Hyper-V server has several clients that store their backups using the Essentials tool. Re-create a VM for one of these clients. With Guest 1 running, try starting the new client using the PXE network boot. That didn’t work—even though the Windows Deployment Services Server is running on the Essentials box (Guest 1), PXE boot still needs a DHCP server to connect. (I later tested setting up an old router as DCHP server, not connected to the Internet. That allowed PXE boot to work.)
Besides PXE, the other supported boot method is USB stick—if you figure out how to create one of those from a Hyper-V guest (see the August 10, 2017 update at the bottom of this post). However you won’t be able to boot the Hyper-V guest from a USB stick. I finally managed to create a bootable ISO using ImgBurn, the files from server’s PXE boot folder:
C:\ProgramData\Microsoft\Windows Server\Data\ClientRestore\
10.1.15063.0_10.1.15063.0_10.0.14393.0\0409
and an etfsboot.com file from another bootable USB, Then I booted that file and was able to restore the client. Unlike with PXE boot, a legacy network adapter is not required.
Notes and Lessons
The hodge-podge of backup strategies makes a full restore a bit time-consuming. If it had to be done in the cloud (if no physical, local machine was available), it would be even more difficult. Fortunately, I haven’t had to do this kind of a restore in 10 or 15 years (seems there was a failed disk once…). Some notes and options for next time:
- It might be simplest for recovery if all backups were stored on a NAS drive. But I’d still have to copy that data to an external drive for weekly swaps, or copy the NAS to the cloud. And what if I only have the external drive/cloud and no NAS to work with during a disaster recovery? I’d prefer to be able to recover directly from an external drive.
- It’s really annoying that Hyper-V, even when fully restored, won’t let you see the settings of VMs that are missing their hard drives. You can’t see if you need a Generation 1 or 2 machine, you can’t see how much memory is configured, and you can’t see how big the disks are. It would help to have all of that documented somewhere.
- If you’re in a test environment (host network disabled/unplugged, no DHCP server available), set up a virtual switch and set static IP addresses and DNS server settings as you go—including on the Hyper-V host. I made the host IP the gateway IP for all machines. Another alternative would be to use an old router as a DCHP server; just don’t connect it to the Internet.
- As mentioned under Guest 3 above, if possible, the backup of a SQL server should include the databases even if it excludes the larger backups. At a minimum, back up the databases master, model, and msdb. That’s an easy change in Windows Server Backup.
- For simplicity, it’s worth considering whether to (additionally) do a full host-level backup of all guests. The caveat with native Windows Server Backup has always been that that it does not support file-level recovery of the guest VMs—for that, you have to run backup inside the VM, which is what I’ve been doing. It’s worth considering a solution that does both, like Veeam Backup and Replication. But with a large enough external hard drive, maybe the simplest thing is to just run Windows Server Backup on the host without excluding VHDX files. That should allow a fast full-server restore, while the granular backups taken inside the VMs would still be available for file-level restores.