Exchange 2013 WMI May Crash on Windows 2012

I found an interesting issue when troubleshooting a problem with Managed Availability. A server in my environment went to an unhealthy state for many processes, but the server failed to bug check. If you restarted the Health Manager service, the problem would still pursist until rebooted. It was discovered, that the WmiPrvSE service running under the Network Service account was crashing several times a day and generating an Event ID: 5612.

Source: WMI
Event ID: 5612
Description:
Windows Management Instrumentation has stopped WMIPRVSE.EXE because a quota reached a warning value. Quota: HandleCount Value: 4140 Maximum value: 4096 WMIPRVSE PID: 30744 Providers hosted in this process: %systemroot%\system32\wbem\cimwin32.dll, %systemroot%\system32\wbem\ntevt.dll, %systemroot%\system32\wbem\mqmtprovider.dll, %systemroot%\system32\wbem\tscrqwmi.dll

The error above refered to exceeding the handle count of this process and therefor causing the process to crash and restart. If you were to add the handles column to the detail view of task manager, you would see the number of handles for this WMI service incrementing slowly. For some of my mailbox servers, this was happening around 7 times a day and all 20 Mailbox and 8 CAS were affected.

We needed to determine if this a problem with a low handle quota or is there a memory leak in the WMI service. I ran the following steps to double the quote limit and validate if a memory leak is present.

  • Go to Start–> Run and type wbemtest.exe.
  • Click Connect.
  • In the namespace text box type “root” (without quotes).
  • Click Connect.
  • Click Enum Instances…
  • In the Class Info dialog box enter Superclass Name as “__ProviderHostQuotaConfiguration” (without quotes) and press OK.
    Note:the Superclass name includes a double underscore at the front.
  • In the Query Result window, double-click “__ProviderHostQuotaConfiguration=@”
  • In the Object Editor window, double-click HandlesPerHost.
  • In the Value dialog, type in 8192.
  • Click Save Property.
  • Click Save Object.
  • Close Wbemtest.
  • Restart the computer.

After a period of time running the server at the higher quota, we again saw the event id and process crash. Though this validated a memory leak was present we still needed confirmation by getting a dump of this process through WinDbg by Premier. Once identified, Microsoft located a hotfix that resolved the issue (KB2790831). The cause of the memory leak per the KB, is

This issue occurs because, when Performance Data Helper log files are opened, Pdh.dll creates a new thread by using the CreateThread() API to process the log files. The CreateThread() API then returns a handle to the newly created thread. The handle remains open after these log files are closed, causing a handle leak.

This hotfix is a year old, so I do wonder why this hadn’t made it to a list of must haves for Exchange 2013, but at least it resolved my particular issue. I will say that my staging environment, which is completely running on VMWare was not affected, but it also had no production traffic.

Handle leak in WmiPrvSE.exe process on a Windows 8-based or Windows Server 2012-based computer
http://support.microsoft.com/kb/2790831

Advertisements