Restore a Hyper-V Server to an Optiplex 960

Recently I upgraded a small server to run Windows Server 2008 R2 as a Hyper-V host, with Server 2008 R2 as a guest as well. I’ve read that Windows Server Backup can restore to dissimilar hardware. That could be especially helpful in a small environment where no other servers are available. Can I restore the entire Server 2008 host and guest to an Optiplex 960 desktop? How long will it take?

Environment Details

The server is a Dell PowerEdge 1900 with 4GB of RAM and two 250GB hard drives configured as RAID 1 (mirrored). As mentioned, it runs Windows Server 2008 R2 with the Hyper-V role, and it hosts a single Server 2008 R2 guest. The guest runs as a domain controller, file server, and email server. Windows Server Backup runs daily on the host, taking an image of the C: drive (containing the host operating system, 16GB) and the D: drive (containing the guest virtual machine, about 48GB used on 80GB of fixed VHD space). Backups are stored on a Seagate Freeagent Go USB drive. Windows Server Backup also runs daily on the guest, saving its backups to a VHD on the same Freeagent USB drive.

The desktop is an Optiplex 960 with 4GB of RAM and one 300GB hard drive. Most important is the processor, an Intel Core 2 E8400 that does support Intel Virtualization Technology and Execute Disable, so it should run Hyper-V.

Recovery Options

There are at least three options for recovering this guest server (which is of course the critical machine):

1. Restore the host and guest from the host’s backup. This makes use of (and tests) Windows Server Backup’s ability to restore to dissimilar hardware, as reported here.

2. Install Server 2008 R2 and the Hyper-V role in a small partition and apply all operating system updates. Restore the guest VHD in a larger partition. Re-create the Hyper-V guest machine using the VHD. This avoids the dissimilar hardware restore at least for the host.

3. Restore the guest backup directly. This would be a virtual-to-physical restore, so also dissimilar hardware.

Option 1 should be the fastest and the most flexible, since it would also accommodate the scenario where multiple guests needed to be restored.

Recovery Experience

The recovery—and this post—became quite long. You may want to skip to the “Conclusions” section, then come back to the step-by-step logs for details as necessary

Time Log – Day 1

Here my log of the recovery process with notes for next time.

3:20pm  Insert Server 2008 R2 DVD into Optiplex 960 and attach USB drive containing backup data. During reboot, check BIOS to make sure Virtualization Technology and Execute-Disable are turned on. Boot from DVD.

3:22pm DVD boot is fast! Glad I didn’t waste time making a bootable thumb drive. Choose Repair your computer, then Restore your computer…. Can’t see external USB drive as backup source. Start googling.

3:35pm Oh yeah, the whole backup drive is Bitlocker-encrypted. Apparently Repair doesn’t detect this as there is no prompt to unlock. Press Shift-F10 to get a command prompt. Commands:
manage-bde –status to show drive (it’s mounted as G:)
manage-bde –unlock –h for parameter help
manage-bde –unlock G: –RecoveryPassword 111111-222222-…-8888888 to unlock using recovery key

3:56pm Drive unlocked. Close command prompt. Press Refresh in Repair wizard. Backup drive found. Choose most recent backup, accept defaults (do check Format and repartition disks, do not check Only restore system drives). Do not install drivers or change Advanced options. Start restore. “This might take from a few minutes to a few hours.”

4:06pm Drive C: restore complete.

4:28pm Drive D: restore about half done.

4:48pm Drive D: restore complete.

4:51pm Drive S: restore complete. “The restored drives are not encrypted.” Remove DVD, unplug USB hard drive, Restart Now.

4:53pm Stuck on boot. “Intel® Matrix Storage Manager” text on screen. I suspect boot drive number is off. Reboot from DVD.

4:58pm Choose Repair your computer, then Use recovery tools…. and Command Prompt. Start fiddling with diskpart:

diskpart
list disk
select disk 0
list volume – letters are all messed up
select volume 1
remove letter=C System Reserved should not have a letter
select volume 2
remove letter=F This is the boot volume
assign letter=C
active identify as the boot volume
select volume 3 this is already assigned drive D, so skip
select volume 4
remove letter=E
assign letter=S
list volumes confirm correct lettering
exit

5:16pm Can’t find a boot.ini file. A little googling confirms that has been replaced in Server 2008 and above with the Boot Manager Configuration file. Let’s just reboot and see what happens.

5:20pm This time we got far enough to see “BOOTMGR is missing. Press Ctrl+Alt+Del to restart”. Reboot from DVD.

5:24pm Operating system is not listed for repair, so follow Method 2 of accepted solution in this thread. From a command prompt, type
bootrec /rebuildbcd
and let it add C:\Windows to the boot list. Remove DVD and restart.

5:27pm Same message:  “BOOTMGR is missing.” Boot from DVD.

This time, operating system is listed, so follow Method 1 at the previous link. Select the operating system. Unfortunately there is no Startup Repair option (apparently that is only available on Windows Vista/7 media). Try a couple other bootrec options:
bootrec /fixmbr
bootrec /fixboot
and reboot.

5:40pm Same message. Giving up for today. Will contact Microsoft Partner Support for suggestions. In a real disaster recovery situation, I would be on the phone with support and/or moving to Recovery Option 2 above. In fact, since the guest’s VHD is already restored, I could probably just install a clean Server 2008 R2 with Hyper-V on top of the botched C drive and use that to load the guest. But I’d like to understand why Recovery Option 1 hasn’t worked and get it working if possible.

Time Log – Day 2

1:10pm MS Support suggests copying the bootmgr file from \Windows\Boot\PCAT\ on the DVD to the root of C:. That location does not exist on the DVD, but it does exist on the restored OS, so I copied that to the root of C:. Voila! Windows boots and immediately reboots, telling me it failed to start. Well at least it’s finding the BOOTMGR.

1:23pm Windows won’t start in Safe Mode either. Don’t even see a blue screen flicker by; just gets partway through the boot and restarts. (It finishes loading CLASSPNP.SYS but fails on whatever comes after that.) Start googling for how to disable Recovery Options > Automatically Restart when you can’t get into the OS.

1:55pm No joy finding out how to change boot behavior. Just for grins, try copying BOOTMGR from the root of the DVD drive to the root of C. Same problem:  won’t start in Safe Mode, just restarts without blue screen.

2:10pm Grasping at straws:  trying copying bootmgr.efi from the root of the DVD to the root of C. Same problem persists.

2:25pm Interesting thread here. Though not directly related, I see I didn’t try one of the recommend commands earlier. Try this while booted from the DVD:
bootrec /rebuildbcd
bootrec /fixmbr
bootrec /fixboot
G: switch to DVD drive
\boot\bootsect.exe /nt60 C: – this is new
After reboot, same issue.

2:40pm Follow an idea near the bottom of this thread and change Optiplex 960 BIOS to put SATA Operation in Legacy mode (was in RAID On mode). Still gets through CLASSPNP.SYS, but does not immediately reboot; it just hangs, won’t even accept Ctrl-Alt-Del. That’s different, so the SATA Operation mode may be the issue! Try the other two SATA modes.  RAID Autodetect / AHCI behaves like RAID On:  a reboot loop. RAID Autodetect / ATA … worked! It hung for a while after CLASSPNP.SYS, but then it gave me a Safe Mode logon screen.

2:53pm After logging on, I see popup telling me “You must restart to apply these changes.” First I went in to System properties > Advanced system settings > Startup and Recovery and unchecked “Automatically restart”. That should stop the reboot loop, should it recur.

2:57pm It rebooted quickly into full operating system and logged on.

Of course the NIC has changed. In Hyper-V manager, run Virtual Network Manager. Change the “Local Area Connection – Virtual Network” to use the External adapter in the Optiplex 960 (Intel 82567LM-3 Gigabit Network Connection).

3:15pm Errors in Optiplex’s Server event log confirm that this machine’s name conflicts with the live server. Temporarily shut down the live server.  Also disable port 25 on router to block incoming email (to keep email from going to the test machine when we start it.) Reboot the Optiplex.

3:18pm Start test guest server. “The virtual machine could not be started because the hypervisor is not running.” This corresponds to Hyper-V-Worker error 3112. The third suggestion may apply:  “If you have made changes to the Boot Configuration Data store, review these changes to ensure that the hypervisor is configured to launch automatically.” We have been mucking around with the boot manager. Hmm, I wonder if I had stayed with the version in the C:\Windows\Boot\PCAT folder instead of copying from the DVD… In fact, maybe I would not have had to rebuild the BCD if I’d just set the SATA mode first.

3:30pm Reboot to DVD. Delete C:\bootmgr, C:\bootmgr.efi, and C:\bootsqm.dat. Reboot. “BOOTMGR is missing.” Oh well. Copy bootmgr from C:\Windows\boot\PCAT to C:\. Windows boots fine. (I realized later that the bootmgr file was not the issue, it was the boot configuration, not stored in the bootmgr file.)

3:41pm I was thinking I would need to re-install the Hyper-V role to get the hypervisor running again, but then I came across this article that shows how to fix it with one command:
bcdedit /set hypervisorlaunchtype auto

3:52pm After rebooting, try starting the guest OS again. Same problem:  the hypervisor is not running. Now in the event log there is Hyper-V-Hypervisor error 41, “Hyper-V launch failed; Either VMX not present or not enabled in BIOS.” Huh? I know I enabled the virtualization stuff in the BIOS.

4:06pm This article suggests turning the BIOS options off and back on. And the first comment in this article says Trusted Execution should NOT be checked in the BIOS, which I agree is counter-intuitive. Followed both suggestions, with full power-off, then started Windows.

4:11pm No error 41 in the event log. Started guest. Failed to start because external drive is not attached to Optiplex. (The guest targets a VHD on that drive for backups.) Use Hyper-V manager to update the settings for the guest, removing that drive from the configuration.

4:13pm Start guest again. It tells me it had not shut down successfully (it wasn’t shut down at all; it was backed up from a VSS snapshot). Let it Start Windows Normally.

4:16pm Guest logon prompt. (Took a long time Applying Computer Settings.) After logon, “You must restart your computer to apply these changes.” Clicked Restart Now.

4:20pm Guest logon prompt. Logged on. Event log looks fairly normal (typical warnings from a startup). Since the guest server has a fixed IP, it automatically is online at the same address as the live server, ready to accept communications from client computers and the Internet. Tested a few components:

Web server works
Active directory works, including accepting logon from client, group policy, folder redirection
Shared drives are available and allow opening QuickBooks data on test server
Email server can retrieve POP mail and send mail

Theoretically at this point I should do a System State restore from the guest’s backup (see my article Careful with Image-Based Backup of Exchange and Active Directory). But I think we’ll call this good for now.

4:33pm Shut down test server, restart live server host and guest, and re-open port 25 on router to allow inbound email.

And a little bonus disaster recovery…

I was anticipating a quick restore of the Optiplex from the Windows Home Server backup. Here’s what happened.

4:53pm In Optiplex BIOS, set SATA Operation back to Raid On. Boot Optiplex from Windows Home Server PC Restore CD. Network driver was missing, but WHS provides an easy way to retrieve the required drivers (by logging on to the WHS console from another computer and copying a the “Windows Home Server Drivers for Restore” folder directly from the WHS backup). Unfortunately that’s the Windows 7 driver, and WHS restore needs the Vista driver.

5:43pm Vista NIC driver downloaded from Dell support site and supplied via USB stick. WHS restore started.

5:57pm Restore complete.

5:59pm Optiplex booted back to Windows 7.

Conclusions and Notes for Next Time

Obviously things did not go as smoothly as hoped. Two hours and 20 minutes on the first day plus three hours and 10 minutes on the second day to get a final logon prompt for the Hyper-V guest. Five and a half hours of solid futzing.

The good news is that it did work; in a real disaster, the client would be back in business.

The other good news is that this was a test, which re-affirms the importance of testing your disaster recovery plan. Next time, I’ll know to check a few key things during the restore process:

• In the Optiplex BIOS, set SATA Operation mode to to RAID Autodetect / ATA before starting the restore.
• In the Optiplex BIOS, set Virtualization, VT for Direct I/O Access, and Execute Disable to ON, but set Trusted Execution to OFF.
• Unlock Bitllocker-protected backup drives using manage-bde commands above.
• If just a blank screen appears after restore, use diskpart to assign correct drive letters and set the C: drive as Active.
• If BOOTMGR is missing, copy bootmgr from C:\Windows\boot\PCAT to C:\.
• Once the host starts, update the Hyper-V Virtual Network Manager to point to the new host NIC.
• If the guest references unavailable and unneeded hardware (e.g. an external backup drive), remove it from the Hyper-V configuration before starting the guest.
• If the guest won’t start because the hypervisor is not running, set it to auto-start using bcdedit command above.

Maybe I’ll try this again someday and see how fast it goes with all that in mind.