Windows Server Backup: VSS Failure Backing Up Guest VM

I’m setting up Windows Server Backup on a Hyper-V host (Windows Server 2008 R2). I want to do an online backup of the guest virtual machine (Windows Server 2003 R2). I’ve already registered the Hyper-V VSS writer per MSKB 958662. So why are my backups failing?

The Errors

Here’s the error in the Application event log on the Hyper-V host:

Log Name:      Application
Source:        Microsoft-Windows-Backup
Event ID:      521
Level:         Error
User:          SYSTEM
Description:  The backup operation that started at ‘‎2010‎-‎09‎-‎13T23:25:48.855000000Z’ has failed because the Volume Shadow Copy Service operation to create a shadow copy of the volumes being backed up failed with following error code ‘2155348129’. Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.

That corresponds to this error in the Hyper-V-VMMS event log on the host:

Log Name:      Microsoft-Windows-Hyper-V-VMMS-Admin
Source:        Microsoft-Windows-Hyper-V-VMMS
Event ID:      10137
Level:         Error
User:          SYSTEM
Description:  No snapshots to revert were found for virtual machine ‘<GuestOS>. (Virtual machine ID 7C88D4F5-71DF-4725-B4B2-E75181BE80EF)

The Wrong Answers

Googling that message takes you to MSKB 2004712, which tells you to enable automount, presumably on the host. However on my machine, automount is already enabled.

There’s a helpful troubleshooting walkthrough on TechNet relating to VSS issues and Windows Server Backup. Even though the example is a similar scenario, I don’t have any errors in the guest, so my issue must be different.

Trying to Work Out the Correct Configuration

Next I turned to the TechNet article Planning for Backup. In the section “Understanding online and offline backups,” I was mightily confused by this sentence: 

Volume Shadow Copy Service must be enabled on all volumes used by the virtual machine with a specific configuration.

This sentence is repeated in several places on the web, but the only clarification I found was at this German site. The author asks two important questions:

1. Does Volume Shadow Copy Service need to be enabled on the host or the guest OS?
2. What should the “specific configuration” look like?

Well, my Server 2003 R2 guest has two volumes, C: for its OS and D: for the data. Volume Shadow Copies are already enabled on D:. Alas, enabling Shadow Copies on C: as well as D: did not allow the host backup to complete, so I disabled them again on C:.

Okay, what about the host? My Hyper-V host also has a C: drive for the OS and a D: drive for data, which in this case are virtual hard drive configuration and disk files. Until now I had not enabled Shadow Copies on either drive. Enabling Shadow Copies on D: was not enough; I got the same error. Additionally enabling Shadow Copies on C: also did not resolve the issue.

For the sake of completeness, I re-enabled Shadow Copies on the guest’s volume C:. So now Shadow Copies are enabled on both C: and D: on both the host and guest. By now, it’s no surprise that the backup failed with the same error:  “No snapshots to revert were found for virtual machine ‘<GuestOS>’.”

Microsoft Support

Microsoft Partner Support clarified two things:

1. The Volume Shadow Copy Service must be enabled on the host, not the guest.

2. I need to check for duplicate disk signatures between the host and the guest.

Aha! I had in fact received warnings about disk signatures in the System event log:

Log Name:      System
Source:        partmgr
Event ID:      58
Level:         Warning
Description:  The disk signature of disk 4 is equal to the disk signature of disk 0.

However when I looked in diskmgmt.msc, there was no disk 4, so I figured it was some issue perhaps related to the virtual devices provided by Dell’s DRAC card. Apparently it was related to a disk attached for creating the backup…

Check and Fix the Disk Signature Conflict

After downloading Sector Inspector and running it on the host and guest machines, it was quickly apparent that since they both had Disk Signature 0x00000080, a disk signature collision could well be my problem.

The fix provided my Microsoft Support was quite simple:

1. Shut down the affected VMs. (I copied the guest VHD to another drive at this point for safety’s sake.)
2. Run the Disk Management MMC on the host.
3. Manually attach the all VHDs to the Hyper-V server via Disk Management MMC.
4. If any VHDs are attached in an “offline” state, a disk signature conflict was found. To fix the conflict, right click the disk and “online” the disk manually.
5. Once all VHD disks are attached in an “online” state, detach them one at a time.
6. Start the VMs.

When I ran Sector Inspector on the guest again, the Disk Signature had changed to 0x4a0d3609. This allowed the host Windows Server Backup to proceed.

Disk2Vhd Gotcha

In retrospect, I realized that the disk signature conflict was a “gotcha” from creating the guest VM using Sysinternals’ Disk2Vhd. Previously, the the Server 2003 R2 machine ran on bare metal. I used Sysinternals Disk2vhd to convert that to a VHD, then installed Server 2008 R2 on the same physical machine and created a VM guest with the Server 2003 R2. Disk2Vhd faithfully copied the disk signature of the source disk. When installed as a guest VM on the same physical disk, the collision occurred.