Informed of Undocumented Fix in Exchange 2013 SP1

For those of you migrating a user from 2007 to 2013, you may notice that email may not purge or intermittently purge from default folders based on assigned Retention Tags. It appears that Exchange 2013 cannot upgrade MRM 1.0 folders to MRM 2.0. I was told by Premier that you need to remove the Managed Folder Policies prior to migration, but even after running the below script, the issue was still occurring.

$users = Get-Content C:\Scripts\migration\Phase2.csv

ForEach ($user in $users)
$name = Get-mailbox $user

Set-Mailbox $name.alias -RemoveManagedFolderAndPolicy

The difference I experienced with running the above script than previous attempts was that I was now seeing some errors in the event logs when starting the MFA (Managed Folder Assistant) on a mailbox.

The MRM Assistant will skip processing mailbox ‘Display Exception details: ‘Microsoft.Exchange.Assistants.TransientMailboxException —> Microsoft.Exchange.InfoWorker.Common.ELC.ELCFolderSyncException: Failed to synchronize the messaging records management settings on managed folder Junk E-mail in mailbox with the settings in Active Directory.

After seeing this event ID, followed by some IDNA tracing, I received some interesting news from Microsoft. This error has been seen before and is fixed in an Interim Update for CU3. The title for this IU, is Interim Update for Exchange Server 2013 Cumulative Update 3 (KB2924875) 15.0.775.4. Since Exchange 2013 SP3 has a build of 15.0.775.0, it is reasonable to think that this is the 4th in a line of Interim Updates for this code release. But for me this didn’t provide me any value because I need to get on SP1.

Knowing I still needed a fix for my problem I begin building a business case to be sent to the product group to have an IU created for SP1. For those who have never had an IU built, it isn’t an overnight process. It takes some effort and time and more than likely, you are just the latest company to complain about the issue. For my issue, I received some surprising news. The code included in Interim Update for Exchange Server 2013 Cumulative Update 3 (KB2924875) 15.0.775.4 has been back ported into Exchange 2013 SP1.

This was about the only thing to have gone my way this entire project, but I am happy to take it. It will be a couple of weeks before I can get the schema extensions processed through our Directory Services team for an SP1 install to validate, but I’m very hopeful that there is truth in the information I’ve received. Now I can focus on the other 10 cases I have open to include MFA removing retention tags and folder policy after manually running MFA.


Trouble with the Apostrophe

apostI recently watched a movie with Clint Eastwood about a baseball scout who was the best in his prime but as he got older he began to have trouble seeing. At one point in the movie the Atlanta Braves are about to sign this big hitter to a nice fat contract, but Eastwood knows this kid can’t hit a curve ball. I bring this up because Microsoft the leader in enterprise software has it’s own curve ball; that is the Apostrophe character.

When I deployed Lync 2010 in my environment and we completed our end user migration, I noticed a trend in tickets from end users who had Irish and Italian last names. I myself being Italian took sympathy and assigned the ticket to myseld. Their ancestory has nothing to do with the problem, but rather was due to their last name having an apostrophe and in turn in their SIP address. It was documented in this KB It was an annoyance, but we were able to work around it and thankfully with 26,000 employees the percent affected was low. It was an interesting bug and provided some good jokes around the office, but I never thought I would see a similar case in a Microsoft product again…I was wrong!

Let’s move forward from the summer 2012 to today where I am in the middle of an Exchange 2013 deployment. I get an interesting call regarding a exec who is getting prompted for his credentials twice when logging into the Outlook Web App. We are currently running Exchange 2013 CU2 in coexistence with Exchange 2007 so Single Sign On(SSO) should be working yet we are seeing CU1 like behavior.

We went through all the basic troubleshooting steps. Look at where his mailbox was located, validated those servers. Removed the load balancer from the equation. Dumped all the attributes for the user account and still found nothing. We then went to the IIS logs, HTTP-Proxy logs…nothing. Exchange Troubleshooting Assistance A.K.A. ExTra was ran against various components and still nada. This caused us all to collectively scratch our heads, both here and at Microsoft.

The engineer I was working with wanted us to install Fiddler on the CAS server we were testing from. If you haven’t used fiddler before, let me give you a couple of tips. First, it doesn’t seem to capture the OWA connection when trying to connect to the server from a remote browser, at least I couldn’t get it to work. You need to open up IE from the CAS server itself to see the traffic. Secondly, follow the Fiddler steps for Windows 8 that are prompeted when running the app. Fiddler can be downloaded here.

When we ran fiddler and began capturing the traffic we saw something odd with regard to the users credentials. Yes you can decrypt the password in fiddler too! When logging into the Exchange 2013 CAS, we saw serveral attempts to proxy the credentials from /owa/auth.owa to the legacy OWA page on Exchange 2007.

The password for this mythical user should be bogus504′ not bogus 504%

When the browser finally redirects you to the legacy URL, you see the following in the fiddler text view. The apostrophe should be replaced with an ' which is the xml version of ” ‘ “.

It is interesting that the apostrophe is listed as an unsafe character per RFC1738 as it can be modified by various gateways or other transport agents.

As much as I love finding a smoking gun to a problem I’m working on, I didn’t get the same level of satisfaction finding this. How did this ever make it out of dogfood???? I even did some post resolution research on this problem and found reports of this with various past exchange versions. Example –

The really troubling part is that I won’t be expecting an resolution at least until Q2 2014 assuming this time it is approved. With CU3 out now and an expected SP1 release in Q1, we will have to wait until there is an SP1 CU1 at a minimum. Sadly hotfixes are like solid product Microsoft released…a distant memory.

KB Issued to Address Mounting More the 50 DBs Post Upgrade from E2K13 CU1 to CU2 .

Tim McMichael posted a KB yesterday to officially address the issue I reported back in October. The issue was a problem mounting a database when the number of copies exceeds 50 after upgrading to CU2 from CU1. In my case I was dealing with LAG copies, but is or all copies after 50. I have not checked to see if the issue still occurs upgrading from CU1 to CU3. If you are interested in Tim’s KB, please check it out.

Exchange 2013 to 2007 Outlook Anywhere Proxy Issue

Two weeks ago was a big step in our Exchange 2013 deployment as we cut over client connectivity to our Exchange 2013 CAS servers. As step in the process of installing Exchange 2013 is to cut over client access to proxy your 2007 or 2010 environment through 2013. This will pass traffic from your 2013 virtual directories to the legacy URL you create on your legacy exchange environment. For a more detailed understanding of this process a good place to start is Ross Smith’s article here. It was a long night and most of everything worked but Outlook Anywhere for users whose mailbox was hosted on Exchange 2007. Our first step in validation was to run the remote connectivity analyzer externally. This must have tool can be found at

When running our first set of tests, we received the following error:

Attempting to ping RPC proxy

 RPC Proxy can’t be pinged.

    Additional Details

  A Web exception occurred because an HTTP 400 – BadRequest response was received from Unknown.

Headers received:

Transfer-Encoding: chunked

request-id: fc24649c-e3f5-43ea-b84f-dca01c1bc8b2

X-CalculatedBETarget: Exchange 2007 CAS FQDN

Persistent-Auth: true

X-FEServer: Exchange 2013 CAS FQDN

Cache-Control: private

Content-Type: text/html; charset=us-ascii

Date: Thu, 21 Nov 2013 13:04:13 GMT

Server: Microsoft-IIS/8.0

X-AspNet-Version: 4.0.30319

X-Powered-By: ASP.NET

Elapsed Time: 541 ms. 

We then tried creating an outlook profile externally and while it worked for 2013, 2007 users could not. With my 2013 profile opened, I added my 2013 mailbox and watched the connection status in outlook and you could see the server attempt a connection on the 2007 MBX servers then drop for a minute and then retry again. This just made a long night longer.

We checked authentication settings on 2013 and 2007 and everything was set to the documented settings. With a team of 5 including my Microsoft PFE on the bridge, we couldn’t figure it out.

Premier Support Time!

Yes, it was time to call Premier and they actually assisted us in resolving the issue rather quickly. Let me say, what you are about to learn is not documented and should be added to the deployment process for all 2007 to 2013 coexistence implementation.

To determine the issue, we first went to the Httperr Logs on the 2013 CAS and found the below error.

400 1 BadRequest MSExchangeRpcProxyAppPool

 We then looked at the Failed-Request log files. There is a good article on troubleshooint failed requests using IIS7 here.


In the FR*.xml file, we saw this error.


The next step was to look at the Httperr Logs on a 2007 CAS server and this is where we found our issue.

 Buffer=”<!DOCTYPE HTML PUBLIC “-//W3C//DTD HTML 4.01//EN””“>


<META HTTP-EQUIV=”Content-Type” Content=”text/html; charset=us-ascii”></HEAD>

<BODY><h2>Bad Request – Request Too Long</h2>

<hr><p>HTTP Error 400. The size of the request headers is too long.</p>


The HTTP Error references a header that is too long for 2007 to accept.

When you install a 2013 Mailbox or MultiRole server, exchange adds two registry items:




But on 2007 CAS, we do not see the same entries and therefore the proxied requests are not making it through 2007 CAS.

The fix is to simply add the same keys and then reboot the server or at a minimum, restart the HTTP and iisadmin service.

Here is a script to add the keys for you.

$keyPath = “HKLM:\System\CurrentControlSet\Services\HTTP\Parameters”;

if (!(Get-Item $keyPath -ErrorAction SilentlyContinue)) { New-Item $keyPath -Force }

set-itemproperty -path $keyPath -Name “MaxFieldLength” -Value 65536 -Type DWORD -Force;

set-itemproperty -path $keyPath -Name “MaxRequestBytes” -Value 65536 -Type DWORD -Force;

It is my hope that these changes will find themselves in a future CU for Exchange 2007.

Lastly, let me send a big thanks to Phillip Denton, Senior Support Escalation Engineer at Microsoft who deserves all the credit for helping us resolve this issue.

Leftovers Uninstalling Exchange 2013 Manually

During my recent deployment of Exchange 2013 I came across an interesting error via managed availability. During our deployment we had some networking issues we had to work out and as part of my troubleshooting I had installed Exchange 2013 on a server on a different class of hardware to rule out local NIC or driver issues.  Sadly, I was in a hurry and the server wasn’t a clean OS build. When it came time to remove the server; the un-install went sideways and all the server would do was login with a black desktop and a command window, and no this was not a server core install.

As an Exchange Administrator, this is not what we like to see. We’ve all had to manually remove an exchange server from AD and while it is not an overly time consuming process it is not best practice, but I didn’t have this option and so off I went.

I removed the server from the Configuration container of Active Directory and the databases that was created. I then removed the computer object before rebuilding the server. The rebuild went smooth, but then I started getting alerts from SCOM telling me that it could not get a heartbeat from the machine. In my haste I forgot to have the AD team remove the server from SCOM which I quickly remedied.

A day later, I was gifted with the following alert via email from one of my existing Exchange 2013 servers.

Source: ServerName – RemoteMonitoring


Last modified by: System

Last modified time: 11/20/2013 3:57:20 PM Alert description: Machine ‘(removed server FQDN)’ has failed to heartbeat since ’11/14/2013 10:59:03 PM’, as observed by machine ‘Current 2013 FQDN’. Restarting the Exchange Health Manager Service did not fix the problem.

If you ran the below Exchange PowerShell Command you would see the Health Set as ‘Unhealthy’

Get-ServerHealth -Identity ‘Server Name’ -HealthSet ‘RemoteMonitoring’ 

This alert told me that managed availability still thinks the Exchange server exists despite the lack of AD objects. Since, not every server was giving me this error, he had to somewhat localized so I turned to the Managed Availability inside the Registry of the alerting server.

Local Managed Availability settings such as Overrides, ServerComponentStates and BugCheck Dates can be found in this location.


The Key I was looking for was another level down.


In this location I found a String Value with the old server name and the last notification date. I backed up the registry (we all do this right?) and deleted the string. And no more erroneous alerts in email or SCOM.


If you like to use remote powershell here is how you can do the above remotely.

Enter Remote PS Session

$credential = Get-Credential

Enter-PSSession –ComputerName <remote server> -Credential $credential

Backup Key

Reg export HKLM\Software\Microsoft\ExchangeServer\V15\ActiveMonitoring\Subjects export.reg

(This will export the key to the document library of your local user profile on the server)

Remove Key

Remove-ItemProperty –Path “Registry::HKLM\Software\Microsoft\ExchangeServer\V15\ActiveMonitoring\Subjects” –name “Server FQDN”

I hope you never have to uninstall Exchange 2013 the hard way, but if you do, I hope this removes one obstacle to a stable system.

Error in Exchange 2013 CU2 when adding LAG copy

During my deployment of Exchange 2013 I noticed an interesting error in my application logs that took me by surprise: Error MapiExceptionTooManyMountedDatabases. I would have expected this had I been on Exchange 2013 CU1, but I had recently upgrade all my servers to CU2 and this was totally unexpected. You all remember the big announcement in July on the features available in CU2, the most anticipated one was the increase of the number of databases per server to Exchange 2010 levels. This received praise from System Admins everywhere! So if CU2  can support 100 databases and I’m on CU2,  then what gives? Let’s first give you a brief description of my environment.

My 2013 environment consists of 20 Dell R810 mailbox servers running Windows 2012, these servers are split into 2 DAGs of 10 server each and those are further split across 2 datacenters in an Active/Active design of 5 mailbox servers each. Each DAG has a File Share Witness at a 3rd primary site and a secondary FSW at a 4th site for further redundancy. As far as databases go, I have 120 databases per DAG with 3 passive copies and a 4th lag for a total of 5 copies. Each server should have 60 DB copies so without CU2 I wasn’t able to add my lags. I do run separate CAS roles, but I will save that stuff for another article.

I had already run the Exchange 2013 scripts you find in the Exchange 2013 calculator prior to CU2, so I could only use them for the first 4 copies. For the LAG copy, I had to perform the arduous steps of doing this process manually…240 times. I am sure you are all asking why couldn’t I script it and to answer your question, it was due to all the diskpart, directory and database creation commands that went into the original calculator script and I decided it was quicker to do this rather than deconstruct the script to serve my needs. Plus I use PowerShell ISE and it is really easy to do this type of work in this GUI.

So, I begin my journey and fired off my cmdlets on server number one. All the databases were created but when I ran a Get-MailboxDatabaseCopyStatus I noticed that the LAGs showed a Failed and Suspended status on the copies I just created.


If you were to open the Exchange Admin Center, you would also see the database suspended and if you opened up the copy window, you will see an error stating there are too many database.


The same error above is also present in the logs.

 Log Name:      Application
Source:        MSExchangeRepl
Date:          9/24/2013 3:14:21 PM
Event ID:      4057
Task Category: Service
Level:         Error
Keywords:      Classic
User:          N/A
The Microsoft Exchange Replication service encountered an unexpected error in log replay for database ‘databasename\servername’. Error MapiExceptionTooManyMountedDatabases: Unable to mount database. (hr=0x8004060e, ec=-2147219954)
Diagnostic context:
Lid: 65256
Lid: 10722   StoreEc: 0x8004060E
Lid: 1494    —- Remote Context Beg —-
Lid: 37952   dwParam: 0x241EE58B
Lid: 39576   StoreEc: 0x977
Lid: 35200   dwParam: 0x95AC
Lid: 58864   StoreEc: 0x8004060E
Lid: 43248   StoreEc: 0x8004060E
Lid: 48432   StoreEc: 0x8004060E
Lid: 54336   dwParam: 0x241EF045
Lid: 1750    —- Remote Context End —-
Lid: 1047    StoreEc: 0x8004060E
Event Xml:
<Event xmlns=”“>
<Provider Name=”MSExchangeRepl” />
<EventID Qualifiers=”49156″>4057</EventID>
<TimeCreated SystemTime=”2013-09-24T19:14:21.000000000Z” />
<Security />
<Data>MapiExceptionTooManyMountedDatabases: Unable to mount database. (hr=0x8004060e, ec=-2147219954)
Diagnostic context:
Lid: 65256
Lid: 10722   StoreEc: 0x8004060E
Lid: 1494    —- Remote Context Beg —-
Lid: 37952   dwParam: 0x241EE58B
Lid: 39576   StoreEc: 0x977
Lid: 35200   dwParam: 0x95AC
Lid: 58864   StoreEc: 0x8004060E
Lid: 43248   StoreEc: 0x8004060E
Lid: 48432   StoreEc: 0x8004060E
Lid: 54336   dwParam: 0x241EF045
Lid: 1750    —- Remote Context End —-
Lid: 1047    StoreEc: 0x8004060E</Data>

These errors are not indicating that the doubling of the supported databases in CU2 are being applied. The only thing I could do at this point is contact support, open a ticket and then call Tim McMichael to help me out. I don’t like queues, they suck to be honest and since my company is Federated with Microsoft, I IM Escalation Engineers over Lync and see if they can take my ticket. Tim McMichael is the most informed individual on Exchange cluster then anyone I know and he is on my speed dial. I want to say that all the credit for this fix is his and the dev team at Microsoft, I am only documenting this find so others may save some time should this issue occur. For some great reading head over to Tim’s blog here.

Tim and I did all the basic troubleshooting, we escalated that to Time Traces on the server, several of them, but they weren’t showing anything productive. We next ran a TList debugger command on the server. If you haven’t run a TList before, it’s pretty cool. You need the windows debuggers installed, but it essentially gives you a list of all running processes, their PID, threads running in each process and any errors it may have. The full information on this debug tool is on the Windows Dev Center site.

In the output of this command, we were looking at the total number of Store Worker Processes running on the server.

     Command Line: “C:\Program Files\Microsoft\Exchange Server\V15\bin\Microsoft.Exchange.Store.Worker.exe” -id:217f0c3a-d483-425b-b7ea-71184cc3430e -pipe:3936 -readykey:Global\WorkerReadyKey-28d3fb96-6c8b-4b95-832e-f4b4383904aa
28316 Microsoft.Exchange.Store.Worker.exe

We confirmed that the server was only running 50 store worker processes not the 60 as expected. We then ran an LDP dump of the Servers Information Store in AD and this is where we received our aha moment. The setting in error is the msExchMaxStoresTotal attribute on the server as indicated in red below. This attribute is the total number of databases allowed per server should not equal 50, but rather 100. 

Dn: CN=InformationStore,CN=BrokeServer,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Cox,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=com
adminDisplayName: InformationStore;
cn: InformationStore;
distinguishedName: CN=InformationStore,BrokeServer,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Cox,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=com;
instanceType: 0x4 = ( WRITE );
msExchESEParamCircularLog: 0;
msExchESEParamCommitDefault: 0;
msExchESEParamDbExtensionSize: 256;
msExchESEParamEnableIndexChecking: TRUE;
msExchESEParamEnableOnlineDefrag: TRUE;
msExchESEParamLogFileSize: 5120;
msExchESEParamPageFragment: 8;
msExchESEParamPageTempDBMin: 0;
msExchESEParamZeroDatabaseDuringBackup: 0;
msExchMaxRestoreStorageGroups: 1;
msExchMaxStorageGroups: 100;
msExchMaxStoresPerGroup: 5;
msExchMaxStoresTotal: 50;
msExchMinAdminVersion: -2147453113;
msExchVersion: 4535486012416;
name: InformationStore;
objectCategory: CN=ms-Exch-Information-Store,CN=Schema,CN=Configuration,DC=COX,DC=com;
objectClass (3): top; container; msExchInformationStore;
objectGUID: 732ccbb0-e55c-4e71-924b-baa15c221108;
showInAdvancedViewOnly: TRUE;
uSNChanged: 886324806;
uSNCreated: 886292709;
whenChanged: 7/22/2013 3:31:02 PM Eastern Daylight Time;
whenCreated: 7/22/2013 12:00:13 PM Eastern Daylight Time;

If you are a GUI person, then open ADSI Edit and go to the Configuration > Services > Microsoft Exchange > Domain > Administrative Groups > Exchange Administrative Group (FYDIBOHF23SPDLT) > Servers > ServerName > Information Store. On the properties of the IS, you will see following.


All 20 of my production mailbox servers had 50 as the value and all 20 were CU1 to CU2 upgrades. The 6 mailboxes I have for my staging environment were direct CU2 installs and showed the attribute as it’s expected value. I confirmed my results by installing a new Exchange 2013 server with CU1, and prior to applying the enterprise product key the msExchMaxStoresTotal value was the expected 5, the value then changed to 50 after setting the key and after the installation of CU2 the attribute maintained the incorrect value.

The fix for this issue was easy after we located the fault, just increase the value to 100 on each of your servers in the DAG, as seen below, and reboot. Once you servers is online, resume your database copies and potentially restart the search service to clear the suspended indexes.


Being this is my first technical blog post, I welcome and appreciate your feedback.

New Blog

2013 has been a really busy year for me. I’m continuing to take courses towards my college degree, I renovated the basement, build two pieces of furniture for the house and deployed Exchange 2013 in my company’s environment. Needless to say, I have a hard time catching my breath with the above items coupled with family time. I’ve been so busy that I failed to use some PTO and when I mean some I mean just about all of it, so I have to use almost a month in the next 10 weeks. With all this spare time I decided It’s about time I begin a blog to discuss one of my favorite subjects; Microsoft Exchange. In this blog I hope to pass on information I’ve learned over the course of my IT career and hopefully in kind, learn from all of you. I hope you find this blog informative as well as entertaining. When I’m not writing in depth technical articles I will be using this forum to vent my frustrations with just about anything that rubs me in an uncomfortable manner. I’m sure there will be no short of criticism and I welcome it when done constructively. I have a lot to learn about blogging and writing in general but I am an eager student and look forward to the challenge. Thanks for your support!