Intermittent Mailbox Stalls Occuring when Migrating to Exchange Online

We have been actively migrating mailboxes to Exchange Online these last few month, over 5000 users this month alone. Some of those migrations have been self batched while others have leveraged the Microsoft FastTrack Center. This week we received a report that a number of mailboxes were stalled due to disk latency, an example of this information can be seen below.

Stalled_Migrated_Mailboxes (2)

This disk latency is occurring on the target database in one of the many Microsoft datacenters. The information I have received from FastTrack points to a problem that has been ongoing since last week and the Product Group is engaged and is slowly rolling a fix to all datacenters.

If you’ve opened a premier ticket regarding this issue be sure you work with your TAM to tag the BUG to the ticket so you are not charged for any support hours.

Microsoft Ignite 2017…I’m Baacckkk!!!!

Microsoft Ignite 2017
Well, it has been a couple of years since I last posted on this blog and there were a couple of reasons to why such a long absence. First, I was completing my Undergraduate degree in Business Administration and secondly I was promoted into a leadership position within my company. Both of these milestones have taken a lot of time away from being able to blog about technology.

Sometimes it takes a spark to reignite a passion and this year while attending the 2017 Ignite conference in Orlando I sat in as session about how to become an MVP and one of the speakers talked about writting for a blog. Now I don’t have any illusions that I have what it takes to become an MVP, but I do like to write and I feel I do have something unique to say. One of the most relevant points made in the session was from Tony Redmond, an Exchange MVP which I have great respect for. I was a little late to the session, but his first comment I heard was towards those bloggers who regurgitate information and never write something that is their own, this is not path that will make you stand out among other bloggers and would most likely make you the target if shame and ridicule. He also talked about finding your own voice and trying to discover a different take on the topic you are trying to present. If the MVP was a religious organization, then for Exchange admins Tony would be our Pope so it would be useful to heed is wise words.

I’m looking forward to learning as much as I can during this week and hopefully return to work with a new perspective and a new dedication to writing relevant information here. It may not be about detailed Exchange technical information, I feel that as a manager I don’t have the opportunity to get my hands dirty anymore. What I can provide you all is a perspective of enterprise leadership as we drive Office 365 adoption and training, develop business drivers for product acquisition and continue to invest in Microsoft services. I look forward to sharing with you all. ~Mike

Apply Forefront Endpoint Protection Exchange 2013 Exclusions via Powershell

With a large environment I wanted to find a way to copy an xml file that’s used by Forefront Endpoint Protection or FEP to apply Exchange 2013 exclusions for all my servers. While FEP and the newer version, System Center Endpoint Protection 2012 are managed through SCCM, you may find it necessary to bypass SCCM for the managing exclusions especially if you have a large number of servers or applications. This would also allow the system owner to manage the exclusions while the SCCM owner manages the deployment of definition files and remediation of infected systems.

Before deploying exclusions to your systems, you first need to build the policy template. I’m not going to spend time discussing the ins and outs of these templates, the easiest method is to take a template and reverse engineer it to work with Exchange. You can find some templates on TechNet here. http://gallery.technet.microsoft.com/System-Center-Endpoint-65917b04. For myself, I included one that I have been using in my lab. I have many custom paths primarily for log directories, I did this to prevent the thousands of logs per day from being generated on my OS drive. To get a basic understanding of the template layout I would refer you to TechNet once again. http://technet.microsoft.com/en-us/library/gg398037.aspx.

Policy XML:

<?xml version="1.0" encoding="US-ASCII"?>
<SecurityPolicy Name="Exchange 2013 Server Policy" Version="2" Description="" ProductVersion="1.0.0.0" IsBuiltIn="false" LastModifiedBy="" CreatedBy="" LastModificationTime="2014-09-04T17:23:18.2764252Z" CreationTime="2014-09-04T17:23:18.1345795Z" xmlns="http://forefront.microsoft.com/FEP/2010/01/PolicyData">
 <PolicySection Name="FEP.AmPolicy" Disabled="false">
  <LocalGroupPolicySettings>
   <IgnoreKey Name="SOFTWARE\Policies\Microsoft\Microsoft Antimalware"/>
    <AddKey Name="SOFTWARE\Policies\Microsoft\Microsoft Antimalware\Real-Time Protection">
     <AddValue Name="DisableRealtimeMonitoring" Type="REG_DWORD">0</AddValue>
     <AddValue Name="DisableOnAccessProtection" Type="REG_DWORD">0</AddValue>
     <AddValue Name="RealTimeScanDirection" Type="REG_DWORD">0</AddValue>
     <AddValue Name="LocalSettingOverrideDisableRealTimeMonitoring" Type="REG_DWORD">0</AddValue>
     <AddValue Name="LocalSettingOverrideDisableIntrusionPreventionSystem" Type="REG_DWORD">0</AddValue>
     <AddValue Name="LocalSettingOverrideDisableOnAccessProtection" Type="REG_DWORD">0</AddValue>
     <AddValue Name="LocalSettingOverrideDisableIOAVProtection" Type="REG_DWORD">0</AddValue>
     <AddValue Name="LocalSettingOverrideDisableBehaviorMonitoring" Type="REG_DWORD">0</AddValue>
     <AddValue Name="LocalSettingOverrideRealTimeScanDirection" Type="REG_DWORD">0</AddValue>
     <AddValue Name="DisableIntrusionPreventionSystem" Type="REG_DWORD">1</AddValue>
     <AddValue Name="DisableIOAVProtection" Type="REG_DWORD">0</AddValue>
     <AddValue Name="DisableBehaviorMonitoring" Type="REG_DWORD">0</AddValue>
     <AddValue Name="DisableScriptScanning" Type="REG_DWORD">1</AddValue>
     <AddValue Name="LocalSettingOverrideDisableScriptScanning" Type="REG_DWORD">0</AddValue>
    </AddKey>
    <AddKey Name="SOFTWARE\Policies\Microsoft\Microsoft Antimalware\Threats\ThreatSeverityDefaultAction">
     <AddValue Name="1" Type="REG_DWORD">6</AddValue>
     <AddValue Name="2" Type="REG_DWORD">2</AddValue>
     <AddValue Name="4" Type="REG_DWORD">2</AddValue>
     <AddValue Name="5" Type="REG_DWORD">2</AddValue>
    </AddKey>
    <AddKey Name="SOFTWARE\Policies\Microsoft\Microsoft Antimalware">
     <AddValue Name="DisableRoutinelyTakingAction" Type="REG_DWORD">0</AddValue>
     <AddValue Name="RandomizeScheduleTaskTimes" Type="REG_DWORD">1</AddValue>
     <AddValue Name="DisableLocalAdminMerge" Type="REG_DWORD">1</AddValue>
    </AddKey>
    <AddKey Name="SOFTWARE\Policies\Microsoft\Microsoft Antimalware\UX Configuration">
     <AddValue Name="CustomDefaultActionToastString" Disabled="true" Type="REG_SZ"/>
     <AddValue Name="Notification_Suppress" Type="REG_DWORD">0</AddValue>
    </AddKey>
    <AddKey Name="SOFTWARE\Policies\Microsoft\Microsoft Antimalware\Scan">
     <AddValue Name="ScheduleQuickScanTime" Type="REG_DWORD" PreviousValue="60">0</AddValue>
     <AddValue Name="ScanParameters" Type="REG_DWORD">1</AddValue>
     <AddValue Name="ScheduleTime" Type="REG_DWORD">60</AddValue>
     <AddValue Name="ScheduleDay" Type="REG_DWORD">0</AddValue>
     <AddValue Name="LocalSettingOverrideScheduleTime" Type="REG_DWORD">0</AddValue>
     <AddValue Name="LocalSettingOverrideScheduleQuickScanTime" Type="REG_DWORD">0</AddValue>
     <AddValue Name="LocalSettingOverrideScheduleDay" Type="REG_DWORD">0</AddValue>
     <AddValue Name="LocalSettingOverrideScanParameters" Type="REG_DWORD">0</AddValue>
     <AddValue Name="DisableCatchupQuickScan" Type="REG_DWORD">0</AddValue>
     <AddValue Name="DisableCatchupFullScan" Type="REG_DWORD">0</AddValue>
     <AddValue Name="CheckForSignaturesBeforeRunningScan" Type="REG_DWORD">1</AddValue>
     <AddValue Name="ScanOnlyIfIdle" Type="REG_DWORD">0</AddValue>
     <AddValue Name="LocalSettingOverrideAvgCPULoadFactor" Type="REG_DWORD">0</AddValue>
     <AddValue Name="AvgCPULoadFactor" Type="REG_DWORD">20</AddValue>
     <AddValue Name="DisableScanningNetworkFiles" Type="REG_DWORD">1</AddValue>
     <AddValue Name="DisableScanningMappedNetworkDrivesForFullScan" Type="REG_DWORD">1</AddValue>
     <AddValue Name="DisableArchiveScanning" Type="REG_DWORD">0</AddValue>
     <AddValue Name="DisableRemovableDriveScanning" Type="REG_DWORD">1</AddValue>
     <AddValue Name="DisableHeuristics" Type="REG_DWORD">0</AddValue>
     <AddValue Name="DisableRestorePoint" Type="REG_DWORD">1</AddValue>
    </AddKey>
    <AddKey Name="SOFTWARE\Policies\Microsoft\Microsoft Antimalware\Exclusions\Extensions" Disabled="false">
     <AddValue Name=".sdb" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name=".config" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name=".dia" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name=".wsb" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name=".chk" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name=".edb" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name=".jrs" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name=".jsl" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name=".log" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name=".que" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name=".lzx" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name=".ci" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name=".dir" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name=".wid" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name=".000" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name=".001" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name=".002" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name=".cfg" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name=".grxml" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name=".dsc" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name=".txt" Type="REG_DWORD" Disabled="false">0</AddValue>
    </AddKey>
    <AddKey Name="SOFTWARE\Policies\Microsoft\Microsoft Antimalware\Exclusions\Paths" Disabled="false">
     <AddValue Name="%windir%\SoftwareDistribution\Datastore\Datastore.edb" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%windir%\SoftwareDistribution\Datastore\Logs\edb.chk" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%windir%\SoftwareDistribution\Datastore\Logs\edb*.log" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%windir%\SoftwareDistribution\Datastore\Logs\Edbres00001.jrs" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%windir%\SoftwareDistribution\Datastore\Logs\Edbres00002.jrs" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%windir%\SoftwareDistribution\Datastore\Logs\Res1.log" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%windir%\SoftwareDistribution\Datastore\Logs\Res2.log" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%windir%\SoftwareDistribution\Datastore\Logs\tmp.edb" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ProgramData%\Microsoft\Search\Data\Applications\Windows" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%windir%\Security\database\*.chk" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%windir%\Security\database\*.edb" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%windir%\Security\database\*.jrs" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%windir%\Security\database\*.log" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%windir%\Security\database\*.sdb" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%allusersprofile%\NTUser.pol" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%SystemRoot%\System32\GroupPolicy\Machine\registry.pol" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%SystemRoot%\System32\GroupPolicy\User\registry.pol" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="C:\Users\Default\AppData\Local\Temp" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%SystemRoot%System32\Inetsrv" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%SystemDrive%\inetpub\temp\IIS Temporary Compressed Files\" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ExchangeInstallPath%\Logging\POP3" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ExchangeInstallPath%\Logging\IMAP4" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="SystemDrive%:\DAGFileShareWitnesses\<DAGFQDN>" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="C:\Program Files\Microsoft\Exchange Server\V15\Mailbox" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="C:\Program Files\Microsoft\Exchange Server\V15\Logging\Managed Folder Assistant" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ExchangeInstallPath%Mailbox" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ExchangeInstallPath%TransportRoles" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ExchangeInstallPath%TransportRoles\Logs" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ExchangeInstallPath%GroupMetrics" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ExchangeInstallPath%Logging" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ExchangeInstallPath%ExchangeOAB" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ExchangeInstallPath\Mailbox" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ExchangeInstallPath%TransportRoles\Data\Queue" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ExchangeInstallPath%ClientAccess\OAB" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ExchangeInstallPath%TransportRoles\Data\SenderReputation" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ExchangeInstallPath%Working\OleConverter" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ExchangeInstallPath%FIP-FS" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ExchangeInstallPath%TransportRoles\Logs\Mailbox" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ExchangeInstallPath%UnifiedMessaging\grammars" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ExchangeInstallPath%UnifiedMessaging\Prompts" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ExchangeInstallPath%UnifiedMessaging\voicemail" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ExchangeInstallPath%UnifiedMessaging\temp" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="C:\ExchangeVolumes" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="C:\ExchangeDatabases" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%TMP%" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%ExchangeInstallPath%TransportRoles\Logs\FrontEnd" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="%windir%\Cluster" Type="REG_DWORD" Disabled="false">0</AddValue>
    </AddKey>
    <AddKey Name="SOFTWARE\Policies\Microsoft\Microsoft Antimalware\Exclusions\Processes" Disabled="false">
     <AddValue Name="Cdb.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Cidaemon.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="fms.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Clussvc.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Dsamain.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="EdgeCredentialSvc.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="EdgeTransport.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="ExFBA.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Inetinfo.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Microsoft.Exchange.AntispamUpdateSvc.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="hostcontrollerservice.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="MSExchangeDagMgmt.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="MSExchangeMigrationWorkflow.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Microsoft.Exchange.AddressBook.Service.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Microsoft.Exchange.ContentFilter.Wrapper.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Microsoft.Exchange.Diagnostics.Service.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Microsoft.Exchange.Directory.TopologyService.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Microsoft.Exchange.EdgeCredentialSvc.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Microsoft.Exchange.EdgeSyncSvc.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Microsoft.Exchange.Imap4.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Microsoft.Exchange.Imap4service.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Microsoft.Exchange.Monitoring.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Microsoft.Exchange.Pop3.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Microsoft.Exchange.Pop3service.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Microsoft.Exchange.ProtectedServiceHost.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Microsoft.Exchange.RpcClientAccess.Service.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Microsoft.Exchange.Search.Service.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Microsoft.Exchange.Servicehost.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Microsoft.Exchange.Store.Service.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Microsoft.Exchange.Store.Worker.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Microsoft.Exchange.UM.CallRouter.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="MSExchangeDagMgmt.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="MSExchangeDelivery.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="MSExchangeFrontendTransport.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="MSExchangeHMHost.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="MSExchangeHMWorker.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="MSExchangeMailboxAssistants.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="MSExchangeMailboxReplication.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="MSExchangeRepl.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="MSExchangeSubmission.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="MSExchangeTransport.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="MSExchangeTransportLogSearch.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="MSExchangeThrottling.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="OleConverter.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Noderunner.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="ParserServer.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="Powershell.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="ScanEngineTest.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="ScanningProcess.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="TranscodingService.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="UmService.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="UmWorkerProcess.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="UpdateService.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
     <AddValue Name="W3wp.exe" Type="REG_DWORD" Disabled="false">0</AddValue>
    </AddKey>
    <AddKey Name="SOFTWARE\Policies\Microsoft\Microsoft Antimalware\Quarantine">
     <AddValue Name="PurgeItemsAfterDelay" Type="REG_DWORD">14</AddValue>
     <AddValue Name="LocalSettingOverridePurgeItemsAfterDelay" Type="REG_DWORD">0</AddValue>
    </AddKey>
    <AddKey Name="SOFTWARE\Policies\Microsoft\Microsoft Antimalware\Signature Updates">
     <AddValue Name="SignatureUpdateInterval" Type="REG_DWORD">8</AddValue>
     <AddValue Name="ScheduleDay" Disabled="true" Type="REG_DWORD">0</AddValue>
     <AddValue Name="ScheduleTime" Disabled="true" Type="REG_DWORD">0</AddValue>
     <AddValue Name="SignatureUpdateCatchupInterval" Type="REG_DWORD">1</AddValue>
     <AddValue Name="conAuGracePeriod" Type="REG_DWORD">24</AddValue>
     <AddValue Name="DefinitionUpdateFileSharesSources" Disabled="true" Type="REG_SZ"/>
     <AddValue Name="FallbackOrder" Type="REG_SZ">MicrosoftUpdateServer|MMPC</AddValue>
     <AddValue Name="SourceOrderOnly" Disabled="true" Type="REG_SZ">FileShares|InternalDefinitionUpdateServer|MicrosoftUpdateServer|MMPC</AddValue>
    </AddKey>
    <AddKey Name="SOFTWARE\Policies\Microsoft\Microsoft Antimalware\SpyNet">
     <AddValue Name="SpyNetReporting" Type="REG_DWORD">1</AddValue>
     <AddValue Name="LocalSettingOverrideSpyNetReporting" Type="REG_DWORD">0</AddValue>
    </AddKey>
   </LocalGroupPolicySettings>
  </PolicySection>
 <PolicySection Name="FEP.HostFirewallPolicy" Disabled="true">
  <WmiPropertySettings>
   <Namespace Name="Root\Microsoft\PolicyPlatform\WindowsFirewallConfiguration">
    <Class Name="Firewall_Profile_Domain">
     <Instance Identifier="@">
      <SetProperty Name="EnableFirewall">True</SetProperty>
      <SetProperty Name="BlockAllInboundTraffic">False</SetProperty>
      <SetProperty Name="DefaultInboundActionIsDeny">True</SetProperty>
      <SetProperty Name="DisableInboundNotifications">False</SetProperty>
      </Instance>
    </Class>
    <Class Name="Firewall_Profile_Private">
     <Instance Identifier="@">
      <SetProperty Name="EnableFirewall">True</SetProperty>
      <SetProperty Name="BlockAllInboundTraffic">False</SetProperty>
      <SetProperty Name="DefaultInboundActionIsDeny">True</SetProperty>
      <SetProperty Name="DisableInboundNotifications">False</SetProperty>
     </Instance>
    </Class>
    <Class Name="Firewall_Profile_Public">
     <Instance Identifier="@">
      <SetProperty Name="EnableFirewall">True</SetProperty>
      <SetProperty Name="BlockAllInboundTraffic">False</SetProperty>
      <SetProperty Name="DefaultInboundActionIsDeny">True</SetProperty>
      <SetProperty Name="DisableInboundNotifications">False</SetProperty>
     </Instance>
    </Class>
   </Namespace>
  </WmiPropertySettings>
 </PolicySection>
</SecurityPolicy>

Now that you have your template in hand, save it off(as an .xml file type of course) to a local drive. Now, we are going to leverage the below script to copy that file you created and apply it to your servers. The below script will take either a single server name, a list of server names from a csv file or take server from the PowerShell Pipeline.  If you choose not to input a username, the script will pass the currently logged in username into the script and you will need to input your password. Once the file is copied, it will invoke  a command to kick off the local ConfigSecurityPolicy.exe to import that xml. Below you will find examples of how to use the script, the script itself and a download of the script and xml as well.

Single Server Input – Default Method:

FEP_Server_PS

Server List from CSV:

FEP_CSV_PS

Pipeline Input:

FEP_Pipeline_PS

Set-ExchFEPExclusions:

#requires -version 3

Function Set-ExchFEPExclusions() {

<#
.SYNOPSIS
The following script will apply SCEP or FEP exclusions to a server

.DESCRIPTION
This script will copy the exclusion XML file to the servers in a list and then apply the XML to the server's AV client

.PARAMETER Csv
Specify the Csv file containing the servers you are targeting

.PARAMETER ServerName
Specifiy the Server Name you are targeting

.PARAMETER Source
Specify the path to the source XML you will be pushing to the servers

.PARAMETER Username
Used if you require elevated credentials on the target server. Defaults to the currently logged in user, but will still prompt for password

.LINK
Exchange 2013 FEP Exclusions
http://technet.microsoft.com/en-us/library/bb332342(v=exchg.150).aspx

.LINK
Editing a FEP Policy
http://technet.microsoft.com/en-us/library/gg398037.aspx

.INPUTS
You can pipe server names into the script

.OUTPUTS
Currently no outputs. Will be enabling logging in a future version

.EXAMPLE
Set-ExchFEPExclusions -Server Server1 -Source C:\temp\Exchange 2013 Server Policy.xml

.EXAMPLE
Set-ExchFEPExclusions -Csv C:\scripts\computers.txt -Source C:\temp\Exchange 2013 Server Policy.xml

.EXAMPLE
get-mailboxserver | ?{$_.databaseavailabilitygroup -eq "DAGName"} | Set-ExchFEPExclusions -Source 'C:\temp\Exchange 2013 Server Policy.xml' -UserName Domain\UserName

.Notes
Author: Mike DiVergilio
Date: 9/16/2014
Version: 2.0
#>

[CmdletBinding(DefaultParameterSetName = 'AVExclusionsByServer')]
Param
(
#List of servers
[Parameter(Mandatory=$false,Position=0,ParameterSetName='AVExclusionsByList',HelpMessage='Please input the full path to the list of servers you are targeting')]
[String]$Csv,

#Single Server
[Parameter(Mandatory=$false,Position=0,ParameterSetName='AVExclusionsByServer',ValueFromPipeline=$true,HelpMessage='Please input the server name you are targeting')]
[ValidateNotNullorEmpty()]
[String]$ServerName,

#Source XML file
[Parameter(Mandatory=$true,HelpMessage='Please input the full path to the XML file you will use for your exclusions')]
[ValidateNotNullorEmpty()]
[String]$Source,

#Input for Username in the form of Domain\username
[Parameter(Mandatory=$false,HelpMessage="Please input Domain\Username if current account doesn't have the neccessary access.")]
[String]$UserName
)

BEGIN {

$SecurePW = Read-Host -Prompt 'Enter Password.' -AsSecureString
$Credentials = New-Object System.Management.Automation.PSCredential -ArgumentList $UserName, $SecurePW
}

PROCESS {

Function Set-AVExclusions() {

[string]$Dest = "C$\Program Files\Microsoft Security Client"
[string]$File = Get-ChildItem $Source| Select-Object -ExpandProperty name

#Opens Session to target server.
$Session = New-PSSession -ComputerName $Server -Credential $Credentials

If (test-connection -Cn $Server -Quiet) {
Write-Host 'Executing Remote Command' -ForegroundColor Yellow

#Creates path and copies file to remote server
$Path = Join-Path -Path "\\$Server.corp.cox.com" -ChildPath $Dest
New-PSDrive -Credential $Credentials -PSProvider FileSystem -Root $Path -Name $Server
Write-Host "Copying Exclusion XML to $Server"
Copy-Item -Path $Source -Destination $Path

#Adjust path to pass variables via the invoke-command cmdlet
$ParentPath = $Dest.Replace('$',':')
$FilePath = Join-Path $ParentPath -ChildPath $File
Invoke-Command -session $Session -ScriptBlock {Param([string]$FilePath)& 'C:\Program Files\Microsoft Security Client\ConfigSecurityPolicy.exe' $FilePath} -ArgumentList $FilePath

#Cleanup
Remove-PSDrive -Name $Server
Remove-PSSession $Session

} else {

Write-host "$Server failed to connect for AV Exclusions"

}
}
Function Set-AVExclusionsByServer() {

$Servers = $ServerName

Foreach ($Server in $Servers) {
Write-Host "Setting Exclusions to $Server" -ForegroundColor Green
Set-AVExclusions
}
}

Function Set-AVExclusionsByList() {

[array]$ServerList = Import-csv -Path $Csv -Header ServerName | Select-Object -ExpandProperty Servername

ForEach ($Server in $ServerList) {
Write-Host "Setting Exclusions to $Server" -ForegroundColor Green
Set-AVExclusions
}
}

switch ($PSCmdlet.ParameterSetName ) {
'AVExclusionsByServer'{Set-AVExclusionsByServer}
'AVExclusionsByList'{Set-AVExclusionsByList}
}
}
END {

Write-Host "The Remote Execution of Exchange 2013 FEP Exclusions is Complete"
}
}

I hope you find this post and this script useful in your administration. I welcome any improvements you may have to make this script better.

Script and XML Download

 

Updated 10/20/2014 to include modifications to the A/V Exclusions TechNet, added Windows 2012 exclusions and added FEP Policy settings.

Mailbox Move from 2013 back to 2007 Failed Due to Mailbox Lock.

I recently ran into an issue migrating a mailbox from one version of Exchange to another. There have been instances where we had to move mailboxes back to Exchange 2007 due to 3rd party applications not fully supporting Exchange 2013. While the migration to Exchange 2013 went flawlessly, the move back stalled and threw an error I hadn’t seen before. If you ran a Get-MoveRequest –identity <Displayname> and piped that to a Get-MoveRequestStatistics you would see the following error.

folderheirarchy

MoveRequest-Message1

When I researched the above error, the only association I found was with migrating mailboxes to Office 365. Michael Van Horenbeeck wrote an article on this very issue. http://vanhybrid.com/2013/07/07/you-get-an-error-stalledduetomailboxlock-when-moving-mailboxes-to-office-365-in-a-hybrid-configuration/

But for me, I was moving from 2013 to 2007 and none of the problems that faced Michael’s scenario were playing a role here. To rule out some things I first activated the source DB to another server, I tested replication health and everything was as it should. I decided to run New-MoveRequest from the shell and see what additional information I could find.

FailedOther

posionjob

I keyed in on the StatusDetail that was saying the move request was stalled due to a lock on the mailbox

Now, I was confused, poisoned job? I’ve had poisoned messages but not a poisoned job. Could it be the mailbox was quarantined by Exchange due to some level of corruption in the mailbox? If I was to run a Disable-MailboxQuarantine, I would see that this wasn’t the case and the issue has to be in the mailbox itself. I went back to the original move request statistics message and looked at what stage in the move the mailbox was failing at. Looking at the SyncStage piece of the request I found that it was failing creating the folder hierarchy. My next step was to open the mailbox in outlook and see if there were folders with non approved characters or perhaps a folder name that was extremely long. But non of these proved to be the case so I decided to check under the hood via MFCMAPI. At first glance, I didn’t see any issue, but I decided to expand the Finders folder. I had previously had an issue with this hidden folder when a user had more then the than 75 search folders causing emails to bounce back with the following NDR.

Remote Server returned ‘554 5.2.0 STOREDRV.Deliver.Exception:StoragePermanentException.MapiExceptionMaxObjsExceeded; Failed to process message due to a permanent exception with message Cannot set search criteria in SearchFolder. Try using fewer keywords at the same time, reducing the number of users in the From, To, Cc, and Bcc fields, and reducing the number of mailboxes that are searched at the same time.

This was easily remediated by adding the /cleanfinders switch when calling the outlook executable. But in this instance, it was not due to a large number of folders, but either something in the search folder or how the folder was named as you can see the odd search folder names below.

MFCMAPI-PosionJob

From what I can tell, those restriction folders are corrupted search folders. I saw a similar restriction folder when I check skipped items during mailbox batches.

CorruptSearch

I just simply hard deleted the folders while in MFCMAPI and then submitted a new move request. I was happy to see the following message and begin my drive home.

requestcomplete

Exchange 2013 WMI May Crash on Windows 2012

I found an interesting issue when troubleshooting a problem with Managed Availability. A server in my environment went to an unhealthy state for many processes, but the server failed to bug check. If you restarted the Health Manager service, the problem would still pursist until rebooted. It was discovered, that the WmiPrvSE service running under the Network Service account was crashing several times a day and generating an Event ID: 5612.

Source: WMI
Event ID: 5612
Description:
Windows Management Instrumentation has stopped WMIPRVSE.EXE because a quota reached a warning value. Quota: HandleCount Value: 4140 Maximum value: 4096 WMIPRVSE PID: 30744 Providers hosted in this process: %systemroot%\system32\wbem\cimwin32.dll, %systemroot%\system32\wbem\ntevt.dll, %systemroot%\system32\wbem\mqmtprovider.dll, %systemroot%\system32\wbem\tscrqwmi.dll

The error above refered to exceeding the handle count of this process and therefor causing the process to crash and restart. If you were to add the handles column to the detail view of task manager, you would see the number of handles for this WMI service incrementing slowly. For some of my mailbox servers, this was happening around 7 times a day and all 20 Mailbox and 8 CAS were affected.

We needed to determine if this a problem with a low handle quota or is there a memory leak in the WMI service. I ran the following steps to double the quote limit and validate if a memory leak is present.

  • Go to Start–> Run and type wbemtest.exe.
  • Click Connect.
  • In the namespace text box type “root” (without quotes).
  • Click Connect.
  • Click Enum Instances…
  • In the Class Info dialog box enter Superclass Name as “__ProviderHostQuotaConfiguration” (without quotes) and press OK.
    Note:the Superclass name includes a double underscore at the front.
  • In the Query Result window, double-click “__ProviderHostQuotaConfiguration=@”
  • In the Object Editor window, double-click HandlesPerHost.
  • In the Value dialog, type in 8192.
  • Click Save Property.
  • Click Save Object.
  • Close Wbemtest.
  • Restart the computer.

After a period of time running the server at the higher quota, we again saw the event id and process crash. Though this validated a memory leak was present we still needed confirmation by getting a dump of this process through WinDbg by Premier. Once identified, Microsoft located a hotfix that resolved the issue (KB2790831). The cause of the memory leak per the KB, is

This issue occurs because, when Performance Data Helper log files are opened, Pdh.dll creates a new thread by using the CreateThread() API to process the log files. The CreateThread() API then returns a handle to the newly created thread. The handle remains open after these log files are closed, causing a handle leak.

This hotfix is a year old, so I do wonder why this hadn’t made it to a list of must haves for Exchange 2013, but at least it resolved my particular issue. I will say that my staging environment, which is completely running on VMWare was not affected, but it also had no production traffic.

Handle leak in WmiPrvSE.exe process on a Windows 8-based or Windows Server 2012-based computer
http://support.microsoft.com/kb/2790831

Scripting the Removal of all Databases from a DAG

Most of you may never need to do this task, but if you like I, have a staging or lab environment you may find yourself doing this from time to time. In my particular case, I decided to rebuild all the databases in all 3 of my DAGs once I installed SP1. I only have 100 users on the platform and once we move to SP1, we will begin migration of the remaining 33,000 mailboxes, so this would be my only chance to do this. Some may think this is unnecessary, but I’ve put these servers through a lot of hard testing and I want to ensure my users have a nice clean database when we move them.

Mailbox Migrations

When thinking of the best way to remove the all the databases, there were many questions I asked myself and then validated in my lab. One hurdle was how do I move all the users from a DAG? This can be done relatively easy, but in production I have 120 databases and I don’t want to cherry pick the target database. Exchange 2013 has a workflow that will choose the databases and I wanted to leverage that process in the move. Before I can allow this process I needed to isolate the DAGs I was targeting for deletion to ensure mailboxes were migrated to only the DAG I was not. This little script did the job nicely

Get-MailboxDatabase | ?{$_.MasterServerOrAvailabilityGroup -eq "DAG-NAME"}` 
| Set-MailboxDatabase -IsExcludedFromProvisioning $true 

Once isolated, I ran a script to dump all mailboxes to a CSV and import it back into a New-MigrationBatch. I’m not as proficient as PowerShell as I would like to be, so if anyone has some thoughts on simplification, I welcome the advice.

$Databases = Get-MailboxDatabase | ?{$_.MasterServerOrAvailabilityGroup -eq "DAG-NAME"}
$path = "C:\DAG-MBXExport.csv"

ForEach ($Database in $Databases) {
       Set-ADServerSettings -ViewEntireForest $true
       Get-Mailbox -Database $database.Name | ForEach-Object {
           $users = $_.PrimarySMTPAddress
           New-Object -TypeName PSObject -Property @{EmailAddress = $users
       } | Select-Object EmailAddress
   } | Export-Csv $path -NoTypeInformation -append

       Get-Mailbox -Database $database.Name -Arbitration | ForEach-Object {
           $Arbitration = $_.PrimarySMTPAddress
           New-Object -TypeName PSObject -Property @{EmailAddress = $Arbitration
       } | Select-Object EmailAddress
   } | Export-Csv $path -NoTypeInformation -append
}
New-MigrationBatch -Name "DAG-Migration-Batch" -local -CSVData ([System.IO.File]::ReadAllBytes($path))

If you want to autostart the batch and autocomplete, just add the parameters -AutoStart or -AutoComplete at the end of the New-MigrationBatch line. You can also add other parameters referenced in this link.

Database Removal

Every mailbox should now be removed from your DAG, so now you can prepare for deleting all database copies. Now, I could have come up with a couple of scripts to do the job, but I like doing things the hard way, so I did it all in one. The following script will disable circular logging if enabled and deleted all the databases currently not mounted, then go back and remove the last database copy for that DB. There is a known issue when removing a database and getting an error when trying to delete the monitoring mailbox, apparently Exchange Trusted Subsystem doesn’t have rights to the Monitoring Mailboxes OU, so it is on my list of things to do to add some functionality to compensate. I’ve tested this script on a DAG with 120 databases across 10 DAG members each having 5 copies and it worked really well.

<#
.Author: Mike DiVergilio, Senior Systems Engineer, Cox Communications

.Date: 4/4/2014

.Synopsis
    Script to remove all database copies from an Exchange 2013 DAG


    Current Build: 1.0

.Description
    This script performs the process of disabling circular logging on each database, blocking 
    activation to ensure a switchover does not occur during this time and then remove all 
    passive and active copies
#>

Function remove_lastcopy(){

ForEach($Server in $Servers){

    $LastDatabases = Get-MailboxDatabase -Server $Server.Identity
    foreach($LastDatabase in $LastDatabases){
        Write-Host "Removing Last Database Copy" $LastDatabase.Name "on Server $Server" -ForegroundColor Red
        Remove-MailboxDatabase $LastDatabase.Name -Confirm:$false
        Start-Sleep 5
        }
    }
}
Function disable_circularlogging(){

$CircularLogDBs = Get-MailboxDatabase -Server $Server.Name

    If($CircularLogDBs.CircularLoggingEnabled -eq $true){
    ForEach($CircularLogDB in $CircularLogDBs){
         Write-Host "Note: Disabling Circular Logging on database $CircularLogDB" -ForegroundColor Yellow
         Set-MailboxDatabase -Identity $CircularLogDB.name -CircularLoggingEnabled $false
         }
    }
    Write-Host "Note: Circular Logging Disabled on all Databases for Server $Server" -ForegroundColor Yellow
}

$Servers = Get-MailboxServer| ?{$_.DatabaseAvailabilityGroup -eq "DAGName" }

ForEach($Server in $Servers){

disable_circularlogging
Set-MailboxServer $Server.Identity -DatabaseCopyAutoActivationPolicy Blocked
$DatabaseCopies = Get-MailboxDatabaseCopyStatus -Server $Server.Identity | select DatabaseName,Name,Status

ForEach($Database in $DatabaseCopies){

    if($database.status -ne "Mounted"){
          Write-Host "Removing Database Copy" $Database.DatabaseName "on Server $Server" -ForegroundColor Green
          Remove-MailboxDatabaseCopy $Database.Name -Confirm:$false
          Start-Sleep 5
    }Else{
    Write-Host "Database" $Database.DatabaseName "is Mounted, Skipping..."
    }
  }
}
Start-sleep 300
remove_lastcopy

Once completed, you will need to remove the EDB and Log files if you decide not to just blow it all away via the Storage Calculator Diskpart script. I may also look into adding this in a future update. I hope someone out there benefits from this process, I know I did when faced with the option of doing this manually or through.

Undocumented SP1 fix is now documented.

I wrote an article concerning a fix in SP1 that was not on the original release notes. Since posting my article Microsoft has documented the issue on TechNet. The KB clearly states that the resolution is to install SP1 and also offers a workaround for those who have not installed SP1 or who have yet to migrate users to 2013. I will state that the workaround in the technet and annotated in my previous post does not work on all folders. I have first hand knowledge of some MRM 1.0 folders such as Junk Mail and Deleted items that still fail to delete when retention tags clearly show the item is expired. The image below is a view of an item currently in my deleted items folder.

expireditem

While the workaround to remove MRM 1.0 policies prior to migration is a recommended step, the full resolution will still require an SP1 install to your environment.