Exchange 2013 WMI May Crash on Windows 2012

I found an interesting issue when troubleshooting a problem with Managed Availability. A server in my environment went to an unhealthy state for many processes, but the server failed to bug check. If you restarted the Health Manager service, the problem would still pursist until rebooted. It was discovered, that the WmiPrvSE service running under the Network Service account was crashing several times a day and generating an Event ID: 5612.

Source: WMI
Event ID: 5612
Description:
Windows Management Instrumentation has stopped WMIPRVSE.EXE because a quota reached a warning value. Quota: HandleCount Value: 4140 Maximum value: 4096 WMIPRVSE PID: 30744 Providers hosted in this process: %systemroot%\system32\wbem\cimwin32.dll, %systemroot%\system32\wbem\ntevt.dll, %systemroot%\system32\wbem\mqmtprovider.dll, %systemroot%\system32\wbem\tscrqwmi.dll

The error above refered to exceeding the handle count of this process and therefor causing the process to crash and restart. If you were to add the handles column to the detail view of task manager, you would see the number of handles for this WMI service incrementing slowly. For some of my mailbox servers, this was happening around 7 times a day and all 20 Mailbox and 8 CAS were affected.

We needed to determine if this a problem with a low handle quota or is there a memory leak in the WMI service. I ran the following steps to double the quote limit and validate if a memory leak is present.

  • Go to Start–> Run and type wbemtest.exe.
  • Click Connect.
  • In the namespace text box type “root” (without quotes).
  • Click Connect.
  • Click Enum Instances…
  • In the Class Info dialog box enter Superclass Name as “__ProviderHostQuotaConfiguration” (without quotes) and press OK.
    Note:the Superclass name includes a double underscore at the front.
  • In the Query Result window, double-click “__ProviderHostQuotaConfiguration=@”
  • In the Object Editor window, double-click HandlesPerHost.
  • In the Value dialog, type in 8192.
  • Click Save Property.
  • Click Save Object.
  • Close Wbemtest.
  • Restart the computer.

After a period of time running the server at the higher quota, we again saw the event id and process crash. Though this validated a memory leak was present we still needed confirmation by getting a dump of this process through WinDbg by Premier. Once identified, Microsoft located a hotfix that resolved the issue (KB2790831). The cause of the memory leak per the KB, is

This issue occurs because, when Performance Data Helper log files are opened, Pdh.dll creates a new thread by using the CreateThread() API to process the log files. The CreateThread() API then returns a handle to the newly created thread. The handle remains open after these log files are closed, causing a handle leak.

This hotfix is a year old, so I do wonder why this hadn’t made it to a list of must haves for Exchange 2013, but at least it resolved my particular issue. I will say that my staging environment, which is completely running on VMWare was not affected, but it also had no production traffic.

Handle leak in WmiPrvSE.exe process on a Windows 8-based or Windows Server 2012-based computer
http://support.microsoft.com/kb/2790831

Scripting the Removal of all Databases from a DAG

Most of you may never need to do this task, but if you like I, have a staging or lab environment you may find yourself doing this from time to time. In my particular case, I decided to rebuild all the databases in all 3 of my DAGs once I installed SP1. I only have 100 users on the platform and once we move to SP1, we will begin migration of the remaining 33,000 mailboxes, so this would be my only chance to do this. Some may think this is unnecessary, but I’ve put these servers through a lot of hard testing and I want to ensure my users have a nice clean database when we move them.

Mailbox Migrations

When thinking of the best way to remove the all the databases, there were many questions I asked myself and then validated in my lab. One hurdle was how do I move all the users from a DAG? This can be done relatively easy, but in production I have 120 databases and I don’t want to cherry pick the target database. Exchange 2013 has a workflow that will choose the databases and I wanted to leverage that process in the move. Before I can allow this process I needed to isolate the DAGs I was targeting for deletion to ensure mailboxes were migrated to only the DAG I was not. This little script did the job nicely

Get-MailboxDatabase | ?{$_.MasterServerOrAvailabilityGroup -eq "DAG-NAME"}` 
| Set-MailboxDatabase -IsExcludedFromProvisioning $true 

Once isolated, I ran a script to dump all mailboxes to a CSV and import it back into a New-MigrationBatch. I’m not as proficient as PowerShell as I would like to be, so if anyone has some thoughts on simplification, I welcome the advice.

$Databases = Get-MailboxDatabase | ?{$_.MasterServerOrAvailabilityGroup -eq "DAG-NAME"}
$path = "C:\DAG-MBXExport.csv"

ForEach ($Database in $Databases) {
       Set-ADServerSettings -ViewEntireForest $true
       Get-Mailbox -Database $database.Name | ForEach-Object {
           $users = $_.PrimarySMTPAddress
           New-Object -TypeName PSObject -Property @{EmailAddress = $users
       } | Select-Object EmailAddress
   } | Export-Csv $path -NoTypeInformation -append

       Get-Mailbox -Database $database.Name -Arbitration | ForEach-Object {
           $Arbitration = $_.PrimarySMTPAddress
           New-Object -TypeName PSObject -Property @{EmailAddress = $Arbitration
       } | Select-Object EmailAddress
   } | Export-Csv $path -NoTypeInformation -append
}
New-MigrationBatch -Name "DAG-Migration-Batch" -local -CSVData ([System.IO.File]::ReadAllBytes($path))

If you want to autostart the batch and autocomplete, just add the parameters -AutoStart or -AutoComplete at the end of the New-MigrationBatch line. You can also add other parameters referenced in this link.

Database Removal

Every mailbox should now be removed from your DAG, so now you can prepare for deleting all database copies. Now, I could have come up with a couple of scripts to do the job, but I like doing things the hard way, so I did it all in one. The following script will disable circular logging if enabled and deleted all the databases currently not mounted, then go back and remove the last database copy for that DB. There is a known issue when removing a database and getting an error when trying to delete the monitoring mailbox, apparently Exchange Trusted Subsystem doesn’t have rights to the Monitoring Mailboxes OU, so it is on my list of things to do to add some functionality to compensate. I’ve tested this script on a DAG with 120 databases across 10 DAG members each having 5 copies and it worked really well.

<#
.Author: Mike DiVergilio, Senior Systems Engineer, Cox Communications

.Date: 4/4/2014

.Synopsis
    Script to remove all database copies from an Exchange 2013 DAG


    Current Build: 1.0

.Description
    This script performs the process of disabling circular logging on each database, blocking 
    activation to ensure a switchover does not occur during this time and then remove all 
    passive and active copies
#>

Function remove_lastcopy(){

ForEach($Server in $Servers){

    $LastDatabases = Get-MailboxDatabase -Server $Server.Identity
    foreach($LastDatabase in $LastDatabases){
        Write-Host "Removing Last Database Copy" $LastDatabase.Name "on Server $Server" -ForegroundColor Red
        Remove-MailboxDatabase $LastDatabase.Name -Confirm:$false
        Start-Sleep 5
        }
    }
}
Function disable_circularlogging(){

$CircularLogDBs = Get-MailboxDatabase -Server $Server.Name

    If($CircularLogDBs.CircularLoggingEnabled -eq $true){
    ForEach($CircularLogDB in $CircularLogDBs){
         Write-Host "Note: Disabling Circular Logging on database $CircularLogDB" -ForegroundColor Yellow
         Set-MailboxDatabase -Identity $CircularLogDB.name -CircularLoggingEnabled $false
         }
    }
    Write-Host "Note: Circular Logging Disabled on all Databases for Server $Server" -ForegroundColor Yellow
}

$Servers = Get-MailboxServer| ?{$_.DatabaseAvailabilityGroup -eq "DAGName" }

ForEach($Server in $Servers){

disable_circularlogging
Set-MailboxServer $Server.Identity -DatabaseCopyAutoActivationPolicy Blocked
$DatabaseCopies = Get-MailboxDatabaseCopyStatus -Server $Server.Identity | select DatabaseName,Name,Status

ForEach($Database in $DatabaseCopies){

    if($database.status -ne "Mounted"){
          Write-Host "Removing Database Copy" $Database.DatabaseName "on Server $Server" -ForegroundColor Green
          Remove-MailboxDatabaseCopy $Database.Name -Confirm:$false
          Start-Sleep 5
    }Else{
    Write-Host "Database" $Database.DatabaseName "is Mounted, Skipping..."
    }
  }
}
Start-sleep 300
remove_lastcopy

Once completed, you will need to remove the EDB and Log files if you decide not to just blow it all away via the Storage Calculator Diskpart script. I may also look into adding this in a future update. I hope someone out there benefits from this process, I know I did when faced with the option of doing this manually or through.