Trouble with the Apostrophe

apostI recently watched a movie with Clint Eastwood about a baseball scout who was the best in his prime but as he got older he began to have trouble seeing. At one point in the movie the Atlanta Braves are about to sign this big hitter to a nice fat contract, but Eastwood knows this kid can’t hit a curve ball. I bring this up because Microsoft the leader in enterprise software has it’s own curve ball; that is the Apostrophe character.

When I deployed Lync 2010 in my environment and we completed our end user migration, I noticed a trend in tickets from end users who had Irish and Italian last names. I myself being Italian took sympathy and assigned the ticket to myseld. Their ancestory has nothing to do with the problem, but rather was due to their last name having an apostrophe and in turn in their SIP address. It was documented in this KB http://support.microsoft.com/kb/2716467. It was an annoyance, but we were able to work around it and thankfully with 26,000 employees the percent affected was low. It was an interesting bug and provided some good jokes around the office, but I never thought I would see a similar case in a Microsoft product again…I was wrong!

Let’s move forward from the summer 2012 to today where I am in the middle of an Exchange 2013 deployment. I get an interesting call regarding a exec who is getting prompted for his credentials twice when logging into the Outlook Web App. We are currently running Exchange 2013 CU2 in coexistence with Exchange 2007 so Single Sign On(SSO) should be working yet we are seeing CU1 like behavior.

We went through all the basic troubleshooting steps. Look at where his mailbox was located, validated those servers. Removed the load balancer from the equation. Dumped all the attributes for the user account and still found nothing. We then went to the IIS logs, HTTP-Proxy logs…nothing. Exchange Troubleshooting Assistance A.K.A. ExTra was ran against various components and still nada. This caused us all to collectively scratch our heads, both here and at Microsoft.

The engineer I was working with wanted us to install Fiddler on the CAS server we were testing from. If you haven’t used fiddler before, let me give you a couple of tips. First, it doesn’t seem to capture the OWA connection when trying to connect to the server from a remote browser, at least I couldn’t get it to work. You need to open up IE from the CAS server itself to see the traffic. Secondly, follow the Fiddler steps for Windows 8 that are prompeted when running the app. Fiddler can be downloaded here.

When we ran fiddler and began capturing the traffic we saw something odd with regard to the users credentials. Yes you can decrypt the password in fiddler too! When logging into the Exchange 2013 CAS, we saw serveral attempts to proxy the credentials from /owa/auth.owa to the legacy OWA page on Exchange 2007.

destination=https%3A%2F%2Fexchange2013cas.contoso.com%2Fowa&flags=4&forcedownlevel=0&username=contoso%5Ctestuser100&password=bogus504%27&isUtf8=1

The password for this mythical user should be bogus504′ not bogus 504%

When the browser finally redirects you to the legacy URL, you see the following in the fiddler text view. The apostrophe should be replaced with an ' which is the xml version of ” ‘ “.

destination=https%3A%2F%2Flegacyexchangeurl.contoso.com%2Fowa&username=contoso%5Ctestuser100&flags=4&password=bogus504

It is interesting that the apostrophe is listed as an unsafe character per RFC1738 as it can be modified by various gateways or other transport agents. http://www.faqs.org/rfcs/rfc1738.html

As much as I love finding a smoking gun to a problem I’m working on, I didn’t get the same level of satisfaction finding this. How did this ever make it out of dogfood???? I even did some post resolution research on this problem and found reports of this with various past exchange versions. Example –http://social.technet.microsoft.com/Forums/exchange/en-US/6a622db1-2ed6-42f1-8712-d4e9abfb829c/legacy-silent-redirection-speech-mark-apostrophe-in-password?forum=exchange2010

The really troubling part is that I won’t be expecting an resolution at least until Q2 2014 assuming this time it is approved. With CU3 out now and an expected SP1 release in Q1, we will have to wait until there is an SP1 CU1 at a minimum. Sadly hotfixes are like solid product Microsoft released…a distant memory.

KB Issued to Address Mounting More the 50 DBs Post Upgrade from E2K13 CU1 to CU2 .

Tim McMichael posted a KB yesterday to officially address the issue I reported back in October. The issue was a problem mounting a database when the number of copies exceeds 50 after upgrading to CU2 from CU1. In my case I was dealing with LAG copies, but is or all copies after 50. I have not checked to see if the issue still occurs upgrading from CU1 to CU3. If you are interested in Tim’s KB, please check it out. http://support.microsoft.com/kb/2897047

Exchange 2013 to 2007 Outlook Anywhere Proxy Issue

Two weeks ago was a big step in our Exchange 2013 deployment as we cut over client connectivity to our Exchange 2013 CAS servers. As step in the process of installing Exchange 2013 is to cut over client access to proxy your 2007 or 2010 environment through 2013. This will pass traffic from your 2013 virtual directories to the legacy URL you create on your legacy exchange environment. For a more detailed understanding of this process a good place to start is Ross Smith’s article here. It was a long night and most of everything worked but Outlook Anywhere for users whose mailbox was hosted on Exchange 2007. Our first step in validation was to run the remote connectivity analyzer externally. This must have tool can be found at https://testconnectivity.microsoft.com/.

When running our first set of tests, we received the following error:

Attempting to ping RPC proxy oa.domain.com.

 RPC Proxy can’t be pinged.

    Additional Details

  A Web exception occurred because an HTTP 400 – BadRequest response was received from Unknown.

Headers received:

Transfer-Encoding: chunked

request-id: fc24649c-e3f5-43ea-b84f-dca01c1bc8b2

X-CalculatedBETarget: Exchange 2007 CAS FQDN

Persistent-Auth: true

X-FEServer: Exchange 2013 CAS FQDN

Cache-Control: private

Content-Type: text/html; charset=us-ascii

Date: Thu, 21 Nov 2013 13:04:13 GMT

Server: Microsoft-IIS/8.0

X-AspNet-Version: 4.0.30319

X-Powered-By: ASP.NET

Elapsed Time: 541 ms. 

We then tried creating an outlook profile externally and while it worked for 2013, 2007 users could not. With my 2013 profile opened, I added my 2013 mailbox and watched the connection status in outlook and you could see the server attempt a connection on the 2007 MBX servers then drop for a minute and then retry again. This just made a long night longer.

We checked authentication settings on 2013 and 2007 and everything was set to the documented settings. With a team of 5 including my Microsoft PFE on the bridge, we couldn’t figure it out.

Premier Support Time!

Yes, it was time to call Premier and they actually assisted us in resolving the issue rather quickly. Let me say, what you are about to learn is not documented and should be added to the deployment process for all 2007 to 2013 coexistence implementation.

To determine the issue, we first went to the Httperr Logs on the 2013 CAS and found the below error.

400 1 BadRequest MSExchangeRpcProxyAppPool

 We then looked at the Failed-Request log files. There is a good article on troubleshooint failed requests using IIS7 here.

C:\inetpub\logs\FailedReqLogFiles\W3SVC1

In the FR*.xml file, we saw this error.

failedrequest

The next step was to look at the Httperr Logs on a 2007 CAS server and this is where we found our issue.

 Buffer=”<!DOCTYPE HTML PUBLIC “-//W3C//DTD HTML 4.01//EN””http://www.w3.org/TR/html4/strict.dtd“>

<HTML><HEAD><TITLE>Bad Request</TITLE>

<META HTTP-EQUIV=”Content-Type” Content=”text/html; charset=us-ascii”></HEAD>

<BODY><h2>Bad Request – Request Too Long</h2>

<hr><p>HTTP Error 400. The size of the request headers is too long.</p>

</BODY></HTML>

The HTTP Error references a header that is too long for 2007 to accept.

When you install a 2013 Mailbox or MultiRole server, exchange adds two registry items:

 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters]

“MaxFieldLength”=dword:00010000

“MaxRequestBytes”=dword:00010000

But on 2007 CAS, we do not see the same entries and therefore the proxied requests are not making it through 2007 CAS.

The fix is to simply add the same keys and then reboot the server or at a minimum, restart the HTTP and iisadmin service.

Here is a script to add the keys for you.

$keyPath = “HKLM:\System\CurrentControlSet\Services\HTTP\Parameters”;

if (!(Get-Item $keyPath -ErrorAction SilentlyContinue)) { New-Item $keyPath -Force }

set-itemproperty -path $keyPath -Name “MaxFieldLength” -Value 65536 -Type DWORD -Force;

set-itemproperty -path $keyPath -Name “MaxRequestBytes” -Value 65536 -Type DWORD -Force;

It is my hope that these changes will find themselves in a future CU for Exchange 2007.

Lastly, let me send a big thanks to Phillip Denton, Senior Support Escalation Engineer at Microsoft who deserves all the credit for helping us resolve this issue.