KB Issued to Address Mounting More the 50 DBs Post Upgrade from E2K13 CU1 to CU2 .

Tim McMichael posted a KB yesterday to officially address the issue I reported back in October. The issue was a problem mounting a database when the number of copies exceeds 50 after upgrading to CU2 from CU1. In my case I was dealing with LAG copies, but is or all copies after 50. I have not checked to see if the issue still occurs upgrading from CU1 to CU3. If you are interested in Tim’s KB, please check it out. http://support.microsoft.com/kb/2897047

Advertisements

Exchange 2013 to 2007 Outlook Anywhere Proxy Issue

Two weeks ago was a big step in our Exchange 2013 deployment as we cut over client connectivity to our Exchange 2013 CAS servers. As step in the process of installing Exchange 2013 is to cut over client access to proxy your 2007 or 2010 environment through 2013. This will pass traffic from your 2013 virtual directories to the legacy URL you create on your legacy exchange environment. For a more detailed understanding of this process a good place to start is Ross Smith’s article here. It was a long night and most of everything worked but Outlook Anywhere for users whose mailbox was hosted on Exchange 2007. Our first step in validation was to run the remote connectivity analyzer externally. This must have tool can be found at https://testconnectivity.microsoft.com/.

When running our first set of tests, we received the following error:

Attempting to ping RPC proxy oa.domain.com.

 RPC Proxy can’t be pinged.

    Additional Details

  A Web exception occurred because an HTTP 400 – BadRequest response was received from Unknown.

Headers received:

Transfer-Encoding: chunked

request-id: fc24649c-e3f5-43ea-b84f-dca01c1bc8b2

X-CalculatedBETarget: Exchange 2007 CAS FQDN

Persistent-Auth: true

X-FEServer: Exchange 2013 CAS FQDN

Cache-Control: private

Content-Type: text/html; charset=us-ascii

Date: Thu, 21 Nov 2013 13:04:13 GMT

Server: Microsoft-IIS/8.0

X-AspNet-Version: 4.0.30319

X-Powered-By: ASP.NET

Elapsed Time: 541 ms. 

We then tried creating an outlook profile externally and while it worked for 2013, 2007 users could not. With my 2013 profile opened, I added my 2013 mailbox and watched the connection status in outlook and you could see the server attempt a connection on the 2007 MBX servers then drop for a minute and then retry again. This just made a long night longer.

We checked authentication settings on 2013 and 2007 and everything was set to the documented settings. With a team of 5 including my Microsoft PFE on the bridge, we couldn’t figure it out.

Premier Support Time!

Yes, it was time to call Premier and they actually assisted us in resolving the issue rather quickly. Let me say, what you are about to learn is not documented and should be added to the deployment process for all 2007 to 2013 coexistence implementation.

To determine the issue, we first went to the Httperr Logs on the 2013 CAS and found the below error.

400 1 BadRequest MSExchangeRpcProxyAppPool

 We then looked at the Failed-Request log files. There is a good article on troubleshooint failed requests using IIS7 here.

C:\inetpub\logs\FailedReqLogFiles\W3SVC1

In the FR*.xml file, we saw this error.

failedrequest

The next step was to look at the Httperr Logs on a 2007 CAS server and this is where we found our issue.

 Buffer=”<!DOCTYPE HTML PUBLIC “-//W3C//DTD HTML 4.01//EN””http://www.w3.org/TR/html4/strict.dtd“>

<HTML><HEAD><TITLE>Bad Request</TITLE>

<META HTTP-EQUIV=”Content-Type” Content=”text/html; charset=us-ascii”></HEAD>

<BODY><h2>Bad Request – Request Too Long</h2>

<hr><p>HTTP Error 400. The size of the request headers is too long.</p>

</BODY></HTML>

The HTTP Error references a header that is too long for 2007 to accept.

When you install a 2013 Mailbox or MultiRole server, exchange adds two registry items:

 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters]

“MaxFieldLength”=dword:00010000

“MaxRequestBytes”=dword:00010000

But on 2007 CAS, we do not see the same entries and therefore the proxied requests are not making it through 2007 CAS.

The fix is to simply add the same keys and then reboot the server or at a minimum, restart the HTTP and iisadmin service.

Here is a script to add the keys for you.

$keyPath = “HKLM:\System\CurrentControlSet\Services\HTTP\Parameters”;

if (!(Get-Item $keyPath -ErrorAction SilentlyContinue)) { New-Item $keyPath -Force }

set-itemproperty -path $keyPath -Name “MaxFieldLength” -Value 65536 -Type DWORD -Force;

set-itemproperty -path $keyPath -Name “MaxRequestBytes” -Value 65536 -Type DWORD -Force;

It is my hope that these changes will find themselves in a future CU for Exchange 2007.

Lastly, let me send a big thanks to Phillip Denton, Senior Support Escalation Engineer at Microsoft who deserves all the credit for helping us resolve this issue.

Leftovers Uninstalling Exchange 2013 Manually

During my recent deployment of Exchange 2013 I came across an interesting error via managed availability. During our deployment we had some networking issues we had to work out and as part of my troubleshooting I had installed Exchange 2013 on a server on a different class of hardware to rule out local NIC or driver issues.  Sadly, I was in a hurry and the server wasn’t a clean OS build. When it came time to remove the server; the un-install went sideways and all the server would do was login with a black desktop and a command window, and no this was not a server core install.

As an Exchange Administrator, this is not what we like to see. We’ve all had to manually remove an exchange server from AD and while it is not an overly time consuming process it is not best practice, but I didn’t have this option and so off I went.

I removed the server from the Configuration container of Active Directory and the databases that was created. I then removed the computer object before rebuilding the server. The rebuild went smooth, but then I started getting alerts from SCOM telling me that it could not get a heartbeat from the machine. In my haste I forgot to have the AD team remove the server from SCOM which I quickly remedied.

A day later, I was gifted with the following alert via email from one of my existing Exchange 2013 servers.

Source: ServerName – RemoteMonitoring

Path: FQDN; FQDN

Last modified by: System

Last modified time: 11/20/2013 3:57:20 PM Alert description: Machine ‘(removed server FQDN)’ has failed to heartbeat since ’11/14/2013 10:59:03 PM’, as observed by machine ‘Current 2013 FQDN’. Restarting the Exchange Health Manager Service did not fix the problem.

If you ran the below Exchange PowerShell Command you would see the Health Set as ‘Unhealthy’

Get-ServerHealth -Identity ‘Server Name’ -HealthSet ‘RemoteMonitoring’ 

This alert told me that managed availability still thinks the Exchange server exists despite the lack of AD objects. Since, not every server was giving me this error, he had to somewhat localized so I turned to the Managed Availability inside the Registry of the alerting server.

Local Managed Availability settings such as Overrides, ServerComponentStates and BugCheck Dates can be found in this location.

HKLM\Software\Microsoft\ExchangeServer\V15\ActiveMonitoring

The Key I was looking for was another level down.

HKLM\Software\Microsoft\ExchangeServer\V15\ActiveMonitoring\Subjects

In this location I found a String Value with the old server name and the last notification date. I backed up the registry (we all do this right?) and deleted the string. And no more erroneous alerts in email or SCOM.

BONUS:

If you like to use remote powershell here is how you can do the above remotely.

Enter Remote PS Session

$credential = Get-Credential

Enter-PSSession –ComputerName <remote server> -Credential $credential

Backup Key

Reg export HKLM\Software\Microsoft\ExchangeServer\V15\ActiveMonitoring\Subjects export.reg

(This will export the key to the document library of your local user profile on the server)

Remove Key

Remove-ItemProperty –Path “Registry::HKLM\Software\Microsoft\ExchangeServer\V15\ActiveMonitoring\Subjects” –name “Server FQDN”

I hope you never have to uninstall Exchange 2013 the hard way, but if you do, I hope this removes one obstacle to a stable system.

Error in Exchange 2013 CU2 when adding LAG copy

During my deployment of Exchange 2013 I noticed an interesting error in my application logs that took me by surprise: Error MapiExceptionTooManyMountedDatabases. I would have expected this had I been on Exchange 2013 CU1, but I had recently upgrade all my servers to CU2 and this was totally unexpected. You all remember the big announcement in July on the features available in CU2, the most anticipated one was the increase of the number of databases per server to Exchange 2010 levels. This received praise from System Admins everywhere! So if CU2  can support 100 databases and I’m on CU2,  then what gives? Let’s first give you a brief description of my environment.

My 2013 environment consists of 20 Dell R810 mailbox servers running Windows 2012, these servers are split into 2 DAGs of 10 server each and those are further split across 2 datacenters in an Active/Active design of 5 mailbox servers each. Each DAG has a File Share Witness at a 3rd primary site and a secondary FSW at a 4th site for further redundancy. As far as databases go, I have 120 databases per DAG with 3 passive copies and a 4th lag for a total of 5 copies. Each server should have 60 DB copies so without CU2 I wasn’t able to add my lags. I do run separate CAS roles, but I will save that stuff for another article.

I had already run the Exchange 2013 scripts you find in the Exchange 2013 calculator prior to CU2, so I could only use them for the first 4 copies. For the LAG copy, I had to perform the arduous steps of doing this process manually…240 times. I am sure you are all asking why couldn’t I script it and to answer your question, it was due to all the diskpart, directory and database creation commands that went into the original calculator script and I decided it was quicker to do this rather than deconstruct the script to serve my needs. Plus I use PowerShell ISE and it is really easy to do this type of work in this GUI.

So, I begin my journey and fired off my cmdlets on server number one. All the databases were created but when I ran a Get-MailboxDatabaseCopyStatus I noticed that the LAGs showed a Failed and Suspended status on the copies I just created.

copystatus

If you were to open the Exchange Admin Center, you would also see the database suspended and if you opened up the copy window, you will see an error stating there are too many database.

MapiError

The same error above is also present in the logs.

 Log Name:      Application
Source:        MSExchangeRepl
Date:          9/24/2013 3:14:21 PM
Event ID:      4057
Task Category: Service
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Servername.contoso.com
Description:
The Microsoft Exchange Replication service encountered an unexpected error in log replay for database ‘databasename\servername’. Error MapiExceptionTooManyMountedDatabases: Unable to mount database. (hr=0x8004060e, ec=-2147219954)
Diagnostic context:
Lid: 65256
Lid: 10722   StoreEc: 0x8004060E
Lid: 1494    —- Remote Context Beg —-
Lid: 37952   dwParam: 0x241EE58B
Lid: 39576   StoreEc: 0x977
Lid: 35200   dwParam: 0x95AC
Lid: 58864   StoreEc: 0x8004060E
Lid: 43248   StoreEc: 0x8004060E
Lid: 48432   StoreEc: 0x8004060E
Lid: 54336   dwParam: 0x241EF045
Lid: 1750    —- Remote Context End —-
Lid: 1047    StoreEc: 0x8004060E
Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event“>
<System>
<Provider Name=”MSExchangeRepl” />
<EventID Qualifiers=”49156″>4057</EventID>
<Level>2</Level>
<Task>1</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime=”2013-09-24T19:14:21.000000000Z” />
<EventRecordID>897512</EventRecordID>
<Channel>Application</Channel>
<Computer>BrokeServer.contoso.com</Computer>
<Security />
</System>
<EventData>
<Data>databasename\servername</Data>
<Data>MapiExceptionTooManyMountedDatabases: Unable to mount database. (hr=0x8004060e, ec=-2147219954)
Diagnostic context:
Lid: 65256
Lid: 10722   StoreEc: 0x8004060E
Lid: 1494    —- Remote Context Beg —-
Lid: 37952   dwParam: 0x241EE58B
Lid: 39576   StoreEc: 0x977
Lid: 35200   dwParam: 0x95AC
Lid: 58864   StoreEc: 0x8004060E
Lid: 43248   StoreEc: 0x8004060E
Lid: 48432   StoreEc: 0x8004060E
Lid: 54336   dwParam: 0x241EF045
Lid: 1750    —- Remote Context End —-
Lid: 1047    StoreEc: 0x8004060E</Data>
</EventData>
</Event>

These errors are not indicating that the doubling of the supported databases in CU2 are being applied. The only thing I could do at this point is contact support, open a ticket and then call Tim McMichael to help me out. I don’t like queues, they suck to be honest and since my company is Federated with Microsoft, I IM Escalation Engineers over Lync and see if they can take my ticket. Tim McMichael is the most informed individual on Exchange cluster then anyone I know and he is on my speed dial. I want to say that all the credit for this fix is his and the dev team at Microsoft, I am only documenting this find so others may save some time should this issue occur. For some great reading head over to Tim’s blog here.

Tim and I did all the basic troubleshooting, we escalated that to Time Traces on the server, several of them, but they weren’t showing anything productive. We next ran a TList debugger command on the server. If you haven’t run a TList before, it’s pretty cool. You need the windows debuggers installed, but it essentially gives you a list of all running processes, their PID, threads running in each process and any errors it may have. The full information on this debug tool is on the Windows Dev Center site.

In the output of this command, we were looking at the total number of Store Worker Processes running on the server.

     Command Line: “C:\Program Files\Microsoft\Exchange Server\V15\bin\Microsoft.Exchange.Store.Worker.exe” -id:217f0c3a-d483-425b-b7ea-71184cc3430e -pipe:3936 -readykey:Global\WorkerReadyKey-28d3fb96-6c8b-4b95-832e-f4b4383904aa
28316 Microsoft.Exchange.Store.Worker.exe

We confirmed that the server was only running 50 store worker processes not the 60 as expected. We then ran an LDP dump of the Servers Information Store in AD and this is where we received our aha moment. The setting in error is the msExchMaxStoresTotal attribute on the server as indicated in red below. This attribute is the total number of databases allowed per server should not equal 50, but rather 100. 

Dn: CN=InformationStore,CN=BrokeServer,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Cox,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=com
adminDisplayName: InformationStore;
cn: InformationStore;
distinguishedName: CN=InformationStore,BrokeServer,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Cox,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=com;
instanceType: 0x4 = ( WRITE );
msExchESEParamCircularLog: 0;
msExchESEParamCommitDefault: 0;
msExchESEParamDbExtensionSize: 256;
msExchESEParamEnableIndexChecking: TRUE;
msExchESEParamEnableOnlineDefrag: TRUE;
msExchESEParamLogFileSize: 5120;
msExchESEParamPageFragment: 8;
msExchESEParamPageTempDBMin: 0;
msExchESEParamZeroDatabaseDuringBackup: 0;
msExchMaxRestoreStorageGroups: 1;
msExchMaxStorageGroups: 100;
msExchMaxStoresPerGroup: 5;
msExchMaxStoresTotal: 50;
msExchMinAdminVersion: -2147453113;
msExchVersion: 4535486012416;
name: InformationStore;
objectCategory: CN=ms-Exch-Information-Store,CN=Schema,CN=Configuration,DC=COX,DC=com;
objectClass (3): top; container; msExchInformationStore;
objectGUID: 732ccbb0-e55c-4e71-924b-baa15c221108;
showInAdvancedViewOnly: TRUE;
uSNChanged: 886324806;
uSNCreated: 886292709;
whenChanged: 7/22/2013 3:31:02 PM Eastern Daylight Time;
whenCreated: 7/22/2013 12:00:13 PM Eastern Daylight Time;

If you are a GUI person, then open ADSI Edit and go to the Configuration > Services > Microsoft Exchange > Domain > Administrative Groups > Exchange Administrative Group (FYDIBOHF23SPDLT) > Servers > ServerName > Information Store. On the properties of the IS, you will see following.

InformationStore_Before

All 20 of my production mailbox servers had 50 as the value and all 20 were CU1 to CU2 upgrades. The 6 mailboxes I have for my staging environment were direct CU2 installs and showed the attribute as it’s expected value. I confirmed my results by installing a new Exchange 2013 server with CU1, and prior to applying the enterprise product key the msExchMaxStoresTotal value was the expected 5, the value then changed to 50 after setting the key and after the installation of CU2 the attribute maintained the incorrect value.

The fix for this issue was easy after we located the fault, just increase the value to 100 on each of your servers in the DAG, as seen below, and reboot. Once you servers is online, resume your database copies and potentially restart the search service to clear the suspended indexes.

InformationStore_After

Being this is my first technical blog post, I welcome and appreciate your feedback.

New Blog

2013 has been a really busy year for me. I’m continuing to take courses towards my college degree, I renovated the basement, build two pieces of furniture for the house and deployed Exchange 2013 in my company’s environment. Needless to say, I have a hard time catching my breath with the above items coupled with family time. I’ve been so busy that I failed to use some PTO and when I mean some I mean just about all of it, so I have to use almost a month in the next 10 weeks. With all this spare time I decided It’s about time I begin a blog to discuss one of my favorite subjects; Microsoft Exchange. In this blog I hope to pass on information I’ve learned over the course of my IT career and hopefully in kind, learn from all of you. I hope you find this blog informative as well as entertaining. When I’m not writing in depth technical articles I will be using this forum to vent my frustrations with just about anything that rubs me in an uncomfortable manner. I’m sure there will be no short of criticism and I welcome it when done constructively. I have a lot to learn about blogging and writing in general but I am an eager student and look forward to the challenge. Thanks for your support!