Skip to main content

Windows VM crash investigation: How to use vmware.log to determine crash origin

 Windows VM crash investigation: How to use vmware.log to determine crash origin

Issue/Introduction

  • A Windows virtual machine experiences an unexpected crash or blue screen (BSOD)
  • The Windows crash dump analysis shows bugcheck 0x80 (NMI_HARDWARE_FAILURE)
  • Microsoft or another OS vendor indicates the crash may be related to virtual hardware and recommends engaging VMware Support
  • In the VM's vmware.log, you see entries similar to:
    YYYY-MM-DDTHH:MM:SS.XXXZ vcpu-0 - WinBSOD: Synthetic MSR[0x40000100] 0x80
YYYY-MM-DDTHH:MM:SS.XXXZ vcpu-0 - WinBSOD: Synthetic MSR[0x40000101] 0x4f4454
YYYY-MM-DDTHH:MM:SS.XXXZ vcpu-0 - WinBSOD: Synthetic MSR[0x40000102] 0x0
YYYY-MM-DDTHH:MM:SS.XXXZ vcpu-0 - WinBSOD: Synthetic MSR[0x40000103] 0x0
YYYY-MM-DDTHH:MM:SS.XXXZ vcpu-0 - WinBSOD: Synthetic MSR[0x40000104] 0x0
  • In hostd.log, you see entries similar to:
    YYYY-MM-DDTHH:MM:SS.XXXZ Hostd[#####]: Guest operating system crash detected.

Environment

  • VMware vSphere ESXi 7.x
  • VMware vSphere ESXi 8.x
  • Windows guest operating system

Cause

When a Windows guest OS crashes, it writes crash parameters to the hypervisor using synthetic Model-Specific Registers (MSRs). These are special registers that allow the guest to communicate with the VMware hypervisor. The WinBSOD entries in vmware.log represent Windows notifying the hypervisor about a crash that has already occurred inside the guest.

The presence of WinBSOD entries does not indicate the hypervisor caused the crash. It indicates the hypervisor received notification of a crash from Windows.

Resolution

Use the following methodology to determine whether a Windows VM crash originated from the hypervisor or from within the guest OS.

Understanding WinBSOD synthetic MSR entries

The synthetic MSR values in vmware.log correspond directly to Windows bugcheck parameters:

MSR Addressvmware.log LabelWindows Bugcheck Parameter
0x40000100MSR[0x40000100]Bugcheck Code
0x40000101MSR[0x40000101]Arg1
0x40000102MSR[0x40000102]Arg2
0x40000103MSR[0x40000103]Arg3
0x40000104MSR[0x40000104]Arg4

For example, if vmware.log shows MSR[0x40000100] 0x80, this corresponds to Windows bugcheck code 0x80 (NMI_HARDWARE_FAILURE).

Step 1: Collect the VM's vmware.log

  1. Identify the datastore path for the affected VM.
  2. Retrieve the vmware.log file from the VM's directory. If the VM has been powered on since the crash, the relevant log may be archived as vmware-1.log, vmware-2.log, etc.

Step 2: Confirm the crash parameters match

  1. Compare the WinBSOD synthetic MSR values in vmware.log to the bugcheck parameters from the Windows crash dump analysis.
  2. If the values match, this confirms both logs are describing the same crash event.

Step 3: Check for hypervisor-initiated NMI

Review the following logs for evidence that the hypervisor sent an NMI to the guest:

  1. Search hostd.log for NMI-related operations:

    grep -i "Send_NMI_To_Guest\|HungVM" /var/log/hostd.log

    If results show /cgi-bin/vm-support.cgi?manifests=...HungVM:Send_NMI_To_Guest... near the crash time, the NMI was manually triggered by an administrator. See Manual Crash Triggered by NMI on Virtual machine hosted on vCenter/ESXi for more information.

  2. Search vmkernel.log for NMI injection events:

    grep -i "nmi" /var/log/vmkernel.log

  3. If no NMI injection events are found, the hypervisor did not send an NMI to the guest.

Step 4: Rule out other hypervisor-side causes

Review the following logs for the timeframe surrounding the crash:

Potential CauseLog to ReviewWhat to Search For
VM stun/snapshotvmware.log, vmkernel.logstun, snapshot, freeze, quiesce
Storage timeoutvmkernel.logSCSI, abort, timeout, APD, PDL
Host memory errorvmkernel.logMCE, MCA, ECC, machine check
Hardware eventIPMI/SEL logsEvents near the crash timestamp
Virtual device errorvmware.logDevice errors prior to the WinBSOD entries

 

Step 5: Interpret the findings

If no hypervisor-side events are found: The crash originated within the Windows guest OS. The WinBSOD entries confirm Windows reported its own crash to the hypervisor. Continue investigation with the OS vendor (Microsoft).

If hypervisor-side events are found: The crash may have been triggered or influenced by the hypervisor or underlying infrastructure. Collect an ESXi support bundle and engage VMware Support for further analysis.

Additional Information

Comments

Post a Comment

Popular posts from this blog

Step-by-Step Explanation of Ballooning, Compression & Swapping in VMware

 🔹 Step-by-Step Explanation of Ballooning, Compression & Swapping in VMware ⸻ 1️⃣ Memory Ballooning (vmmemctl) Ballooning is the first memory reclamation technique used when ESXi detects memory pressure. ➤ Step-by-Step: How Ballooning Works  1. VMware Tools installs the balloon driver (vmmemctl) inside the guest OS.  2. ESXi detects low free memory on the host.  3. ESXi inflates the balloon in selected VMs.  4. Balloon driver occupies guest memory, making the OS think RAM is full.  5. Guest OS frees idle / unused pages (because it believes memory is needed).  6. ESXi reclaims those freed pages and makes them available to other VMs. Why Ballooning Happens?  • Host free memory is very low.  • ESXi wants the VM to release unused pages before resorting to swapping. Example  • Host memory: 64 GB  • VMs used: 62 GB  • Free: 2 GB → ESXi triggers ballooning  • VM1 (8 GB RAM): Balloon inflates to 2 GB → OS frees 2 GB → ESXi re...
Post a Comment
Read more

ESXi Host Troubleshooting Checklist

  🛠️ ESXi Host Troubleshooting Checklist (With Complete Log Locations) ✅ 1. Host Status & Connectivity Check host state in vCenter (Connected / Not Responding / Disconnected) Review CPU, RAM, and datastore usage Validate HA/DRS recommendations ✅ 2. Hardware Health Monitor hardware sensors (CPU, DIMMs, fans, PSU, RAID) Check RAID controller logs via vendor tools View hardware status in: vCenter → Monitor → Hardware Health DCUI → Hardware Status ✅ 3. Network Validation Verify vSwitches, Port Groups, NIC teaming, VLAN tagging Check vmkernel ports (mgmt, vMotion, iSCSI, vSAN) Test reachability using: Shell vmkping < IP > Show more lines Validate physical NIC status and link speed ✅ 4. Storage & Datastore Checks Confirm datastore accessibility Rescan storage adapters (iSCSI/FC/NFS) Validate multipathing (Round Robin / Fixed / MRU) Check datastore latency with esxtop ✅ **5. Critical Logs & Their Locations (Full List) Here are key ESXi log files that every VMware adm...
Post a Comment
Read more
  vCenter Troubleshooting Tips, Common Issues & Log Locations 🚀 1. General Troubleshooting Approach A. Check service health first For vCenter Server Appliance (VCSA): Shell https : / / < vcenter - FQDN > : 5480 Show more lines Go to Services → Health to verify: vCenter Server ESXi hosts vSphere Client DNS, NTP, DB connections vpxd, vpxd-svcs, vsphere-ui B. DNS & NTP checks vCenter is highly dependent on correct DNS & time sync. Check: Shell nslookup < vcenter - fqdn > nslookup < esxi - host - fqdn > Show more lines Time drift > 5 minutes causes: Login failures vpxd crashes PSC authentication issues C. Check storage/database health Slow DB/storage = UI slow, tasks hung. For VCSA: Shell df - h du - sh / storage / log / * / var / log Show more lines Free up space if partitions hit >80%. D. Restart critical services safely On VCSA: Shell service - control - - status service - control - - stop - - all service - control - - start - - all ...
Post a Comment
Read more