Fix – Windows 8 and Server 2012 virtual machines don’t ever reboot

I recently ran into a problem were virtual machines running Windows 8 and Server 2012 don’t ever come back from a reboot, but rather get stuck on the Windows boot splash screen. The only work around to get the machine to boot is to manually power it off and then power it on again.

Good news is, there is a solution to this crazy weirdness; and it is documented below…

What do we know?

Here is a summary of the scenario for the Windows 8 and Server 2012 virtual machines don’t ever reboot issue that I was faced with:

• The VMs are running Windows 8, Windows Server 2012 or Windows Server 2012 R2
• The VMs are running on an ESXi 5.5 hosts
• The VMs are configured with more than one vCPU
• The VMs are configured with virtual machine hardware version 10 (vmx-10)

What was the symptoms?

Here are the symptoms I experienced:

• After a reboot, the Windows boot splash screen would appear and would never disappear (the scrolling wheel would go round and round in circles endlessly)
• The only resolution would be to do a full power cycle of the virtual machine (i.e. power off and then power on again)
• If left for a few hours, the machine might eventually boot (although I could not get this to happen consistently during testing)

Just to make the issue 100% clear, here is a screenshot of the issue (i.e. stuck on the Windows 8 or Server 2012 boot splash screen):

Why is it happening?

I managed to find a VMware KB article (KB2092807) which documents why this issue is occurring. Rather than trying to explain it, here it is straight from the horse’s mouth:

Starting with Windows 8/Windows 2012 Server, during the boot process, the operating system resets the TSC (TimeStampCounter, increments by 1 for each passed cycle) on CPU0 when it detects it to be equal or larger than 0x40000000000000. It does not reset the TSC of the other vCPUs and the resulting discrepancy between two vCPUs TSC values may result in the issues described under the Symptoms section. This only applies to virtual machine hardware version 10 as Windows resets the TSC on all CPUs on virtual machines with older hardware versions (which do not support hypervisor.cpuid.v2).

For virtual machines with legacy BIOS:

Rebooting Windows on virtual machines up to hardware version 10 (vmx-10) is done via a soft reset during which INIT is asserted. Unlike after a hard reset (assert of CPURST, PCIRST# and RSTDRV), operating systems should not expect the TSC to be reset to 0x0 after a soft reset. The default behavior of not resetting the TSCs upon a soft reset can be changed with a configuration option.

For virtual machines with EFI:

This is a known issue with the VMware EFI Firmware in which it shows hard reset to be available, but only does a soft reset.

Resolution

Ok now the good part, how to fix the “Windows 8 and Server 2012 virtual machines don’t ever reboot” issue. There are actually two solutions, both documented below:

Option 1: Configure Virtual Machines

The most common and easiest way to resolve the stuck on reboot issue would be to add monitor_control.enable_softResetClearTSC = TRUE to the advanced configuration of each virtual machine that has the issue. To do this, complete these steps:

1. Log into the vSphere Client
2. Find the VM with the issue and power the VM off
3. Once powered off, right click the VM and select Edit Settings
4. From the Options tab, navigate to Advanced >> General
5. Click on Configuration Parameters
6. From the window that appears, click on the Add Row button
7. A new row will be created at the bottom of the list
8. In the new row enter the following:
   • Name: monitor_control.enable_softResetClearTSC
   • Value: TRUE
9. Click OK
10. Click OK to the Edit Settings window to save the changes
11. Power on the Virtual Machine

Option 2: Upgrade

The second option is to upgrade to vSphere 6.0 and then once completed, upgrade the problem VMs to virtual machine hardware 11 (vmx-11). Easier for me to say, but much harder to implement in reality.

Automate the Solution?

The solution above is great if you only have a handful of Windows 8 or Windows Server 2012 machines, but what happens if you have a whole bunch of VMs with the issue?

Well here is how to automate the resolution for both existing and new virtual machines:

Deploy to existing VMs

To deploy to all existing VMs with the issue, you can run the following PowerCLI script. Note: Each VM will still need to be either vMotioned to another host or power cycled for the change to take affect!

ForEach ($vm in (Get-VM)) {
  $vmv = Get-VM $vm | Get-View
  $name = $vmv.Name
  $guestid = $vmv.Summary.Config.GuestId
  $vmx = New-Object VMware.Vim.VirtualMachineConfigSpec
  $vmx.extraConfig += New-Object VMware.Vim.OptionValue
  $vmx.extraConfig[0].key = "monitor_control.enable_softResetClearTSC"
  $vmx.extraConfig[0].value = "TRUE"

  If ($guestid -like "windows8*Guest") {
    ($vmv).ReconfigVM_Task($vmx)
  }
}

Deploy new VMs

Easiest way to deploy this to all new VMs is to add it into your Windows 8 or Windows Server 2012 templates. I did this to my templates and it works great!

More Information?

Here is the link to the VMware KB article that saved my life >>> VMware KB2092807.

Any questions, let me know in the comments below.

Thanks
Luca