I recently watched a movie with Clint Eastwood about a baseball scout who was the best in his prime but as he got older he began to have trouble seeing. At one point in the movie the Atlanta Braves are about to sign this big hitter to a nice fat contract, but Eastwood knows this kid can’t hit a curve ball. I bring this up because Microsoft the leader in enterprise software has it’s own curve ball; that is the Apostrophe character.
When I deployed Lync 2010 in my environment and we completed our end user migration, I noticed a trend in tickets from end users who had Irish and Italian last names. I myself being Italian took sympathy and assigned the ticket to myseld. Their ancestory has nothing to do with the problem, but rather was due to their last name having an apostrophe and in turn in their SIP address. It was documented in this KB http://support.microsoft.com/kb/2716467. It was an annoyance, but we were able to work around it and thankfully with 26,000 employees the percent affected was low. It was an interesting bug and provided some good jokes around the office, but I never thought I would see a similar case in a Microsoft product again…I was wrong!
Let’s move forward from the summer 2012 to today where I am in the middle of an Exchange 2013 deployment. I get an interesting call regarding a exec who is getting prompted for his credentials twice when logging into the Outlook Web App. We are currently running Exchange 2013 CU2 in coexistence with Exchange 2007 so Single Sign On(SSO) should be working yet we are seeing CU1 like behavior.
We went through all the basic troubleshooting steps. Look at where his mailbox was located, validated those servers. Removed the load balancer from the equation. Dumped all the attributes for the user account and still found nothing. We then went to the IIS logs, HTTP-Proxy logs…nothing. Exchange Troubleshooting Assistance A.K.A. ExTra was ran against various components and still nada. This caused us all to collectively scratch our heads, both here and at Microsoft.
The engineer I was working with wanted us to install Fiddler on the CAS server we were testing from. If you haven’t used fiddler before, let me give you a couple of tips. First, it doesn’t seem to capture the OWA connection when trying to connect to the server from a remote browser, at least I couldn’t get it to work. You need to open up IE from the CAS server itself to see the traffic. Secondly, follow the Fiddler steps for Windows 8 that are prompeted when running the app. Fiddler can be downloaded here.
When we ran fiddler and began capturing the traffic we saw something odd with regard to the users credentials. Yes you can decrypt the password in fiddler too! When logging into the Exchange 2013 CAS, we saw serveral attempts to proxy the credentials from /owa/auth.owa to the legacy OWA page on Exchange 2007.
The password for this mythical user should be bogus504′ not bogus 504%
When the browser finally redirects you to the legacy URL, you see the following in the fiddler text view. The apostrophe should be replaced with an ' which is the xml version of ” ‘ “.
It is interesting that the apostrophe is listed as an unsafe character per RFC1738 as it can be modified by various gateways or other transport agents. http://www.faqs.org/rfcs/rfc1738.html
As much as I love finding a smoking gun to a problem I’m working on, I didn’t get the same level of satisfaction finding this. How did this ever make it out of dogfood???? I even did some post resolution research on this problem and found reports of this with various past exchange versions. Example –http://social.technet.microsoft.com/Forums/exchange/en-US/6a622db1-2ed6-42f1-8712-d4e9abfb829c/legacy-silent-redirection-speech-mark-apostrophe-in-password?forum=exchange2010
The really troubling part is that I won’t be expecting an resolution at least until Q2 2014 assuming this time it is approved. With CU3 out now and an expected SP1 release in Q1, we will have to wait until there is an SP1 CU1 at a minimum. Sadly hotfixes are like solid product Microsoft released…a distant memory.