Technology for Mortals

I have been struggling with a problem with ActiveSync for ages now, I kept getting a 401.2 error sent to the client when they would try to get their mail (but they could get their calendar). The problem was in the ISA server configuration. When I changed ISA’s access rule to be set to “Basic” authentication instead of “No delegation, but client may authenticate directly” the problem was solved.

J.Ja

I think we all remember the “Inconceivable!” routine from The Princess Bride. That’s about how I feel after the last few weeks. I have an extremely high end RAID controller in a box, a 20 port (16 internal, 4 external) device. It’s in a monster SuperMicro case, with 2 small drives for the OS, and 14 1 TB drives for storage. Over the last few weeks, we’ve been having a host of bizarre behavior, from the pair of new 2 TB drives magically disappearing and reappearing on the controller every 11 minutes (on the dot) to the system having LEDs and sirens as if there is a severe failure but the RAID controller software showing 100% optimal state. Update: Just to make it clear, we are (and have been) working with the vendor on this issue. When we find out precisely what the issue is, I’ll post a new item. Also, I forgot one angle of this when I first posted the story. These bizarre failings (not the two 2 TB drives, the other one) turned out to have been caused by (get this) bad sectors on the drives. But those drives (or the controller) are supposed to automatically handle and work around bad sector errors! And why would that kind of error blow out the RAID to the point where the controller is sounding alarms, but not to the point where the software is aware? “Inconceivable!”

Tonight took the cake, though. My on-site person deliberately broke the RAID. We had planned to do this; we wanted to take one of the mirrored drives and put it into our backup chassis to help diagnose a problem with the backup unit. One of my “Inconceivable!” moments two weeks ago, was when we wanted to move to the backup chassis, the system went into an endless reboot cycle, even though it worked fine a few months prior and hasn’t been touched since. The plan was simple: pull the drive, put one of our spares in, and let the RAID (it’s a RAID 1) sync. No big deal. Well, the system decides to BSOD, in a definite “Inconceivable!” moment. Let’s get this straight. A RAID controller that we paid $1,500 – $2,000 for (I can’t recall the number offhand) decides to panic so badly that the entire OS comes crashing down, over a simple hot swap of hard drives? Inconceivable!

After the reboot, users start complaining that they can’t get their email, so I get a call. Yet another “Inconceivable!” moment… I had just sat down at a restaurant to celebrate my wife’s birthday with about TWENTY friends and family. We look at the Exchange server (a VM on the machine that BSOD’ed). After some diagnoses, it looks like the Exchange databases managed to get corrupted and refuse to recover themselves. Once again… “Inconceivable!” I spent the entire dinner (including bathroom breaks, ordering, and eating) on the phone. My only break was when we ran some repairs that took a while, just long enough to have a few moments of conversation and sing Happy Birthday. I’m on the phone throughout the goodbyes. And of course, Thursday is the night when I usually do the food shopping. To make matters worse, I deliberately ran out of food this afternoon, so food shopping was not an option. Valuable troubleshooting time, and I need to be in the food store. On top of that, I can’t stay up all night and sleep in, because I’ve been watching our son in the morning as my wife has returned to work, and he wakes up  early. “Inconceivable!” So I must get this resolved before, say, midnight.

I eventually give up with my on-site guy, and resign myself to a very long night. My boss calls while I am in the food store, and he decides to give the recovery another try. See, the previous recovery attempts failed, with error codes that were not found on Google or Bing. “Inconceivable!” Well, the new recovery attempts all fail. At the last moment, we decide to try a different command line switch. The whole thing took an hour and a half, finished up (after fixing the corrupted database file), and after a restart of the Information Store service, Exchange is working just fine again.

So, to add up all of the “inconceivable” events:

• The enterprise grade RAID controller wet its pants over a simple hot swap
• A hard drive hot swap BSOD’ed Windows
• A “power off” failure put the Exchange database in a state that it could not automatically recover from
• This all happened on one of the three nights a year that I cannot be at my home office for, oh, four hours, and during the one three week period of the year that I can’t stay up all night and sleep during the day
• None of the obvious recovery choices worked

Obviously, “Inconceivable” does not mean what I think it does!

J.Ja

If you’re going to be installing BlackBerry Enterprise Server (BES) 5.0 on a Windows Server 2008 machine, you better be ready to call technical support or read this blog post at a minimum.  I spend 5 hours on the phone with BlackBerry customer support over the course of two days to work out all the installation problems that should have been automated by the installer and documented in the pre-installation guide.  I will give Research In Motion support credit for helping me through these problems.

The main different between BES 5.0 and BES 4.1.x is that the administration interface is Java and ActiveX web based only, which irritates me to no end.  BES 4.1 gives you a real interface that works without a browser and isn’t dependent on some complicated JBoss+Apache web server setup.  This web based aspect of BES 5.0 was the source of some major installation and configuration headaches.  With BES 4.1.x, you don’t have to put up with any of this nonsense.  The reasoning behind 5.0 using a web administration interface is that now the users can manage their own BlackBerry Enterprise accounts though I really didn’t need this feature.

Just getting the basic preparation work done for a BES 5.0 install is daunting enough because you have to go through this pre-installation guide.  That involves setting up a BES service account, the local server permissions, the active directory permissions, the exchange server permissions, and other software that needs to be pre-installed.  It gets a bit confusing and I definitely recommend just using the SQL Server 2005 express they include in their BES 5.0 installer and then you don’t have to worry about additional JDBC drivers for remote SQL databases.  Don’t bother installing SQL express yourself as it will only make things more complicated.  You will need to install the Microsoft Exchange MAPI client before you install BES.

One pre-installation step that is missing is that you need to make sure that IPv6 on Windows Server 2008 is disabled.  Un-checking IPv6 in the network interface is not sufficient, and you’ll need to follow the instructions in this Microsoft KB article (Jeremy in the comment section below pointed out that the updated KB article is here) and edit the registry.  [NOTE - Disabling IPv6 will break Windows Small Business Server (SBS) which means you don't mix BES with SBS].  Once you’ve done this along with all the other pre-installation procedures above, you can proceed to install BES.  Note that during installation, be sure to select BlackBerry Server authentication for the web administration interface and not Active Directory (AD) integrated authentication.  BlackBerry technical support couldn’t figure out how to get this working and they told me to reinstall from scratch on a clean machine and don’t use AD authentication for web administration.

There’s more pain after you’ve completed the whole installation.  The web administration interface also needs access to the SQL database which doesn’t work out of the box and can frustrate you to no end.  Because the BES installer didn’t bother to nail down the SQL ports from dynamic to fixed TCP 1433, the web admin page refuses to come up.  You need to go into the SQL Server Configuration Manager and manually set the TCP ports to 1433 as shown below.

Internet Explorer 8.0 in Windows Server 2008 also requires you to enable “compatibility mode” for the administration page and you must also put the administration URL in the trusted site list.  The other problem you’ll find is that because the BES 5.0 installer generated its own SSL certificate rather than using the one that is either already on the server or easily obtained in a Windows network environment with a Certificate Authority in place, the browser throws up error messages that it doesn’t trust the SSL certificate.  You can fix this by right clicking on Internet Explorer and choose “Run as administrator”.  Then you can view the certificate and install it.  However, it’s installed in the wrong place in the personal user certificate store and you’ll have to export the certificate and import it into the computer certificate.  A better option is to import the certificate into Active Directory trusted certificates using this procedure I wrote up in 2006 and that solves your problem for every computer within the active directory that needs to use the BES web administration page.

Anyhow, it’s all working for me right now and I hope this document helps you avoid the headaches I went through.

To put it plainly, someone completely forgot to document Exchange 2007. The OAB (Offline Address Book) issues that were present in previous editions are still there. The sad part is, the problem is documentation, not technical. Over the last month or so, I have been wrestling with OAB issues for at least 20 hours per week. About 30 minutes ago, I conquered them for good. Here is everything that I have learned along the way, hopefully it will spare you some trouble.

The error that we usually get in Outlook is “an object cannot be found”. The Synchronization Log shows that it could not get the Offline Address Book (to see the log, go to the “Folders” view and select the “Sync Issues” folder). Additionally, you’ll possibly get an error code of 0x8004010F during a Send/Receive. Here are some of the causes I have found for this:

The OAB does not exist at all
Verify that the OAB has been created. Turn up the logging level on OAB in Exchange and force a re-creation of it. Verify that there are no errors. If there are errors, fix them.

The OAB is in the wrong place or inaccessible
After creating an OAB, you need to restart Outlook to get the new location. If you still get the error, check the IIS logs on the server hosting the Exchange Mailbox role. Look for entries trying to do a GET on /OAB/* which will be Outlook trying to get the OAB. If there are errors, fix them. A few common problems with this:

• The OAB directory requiring SSL; all of the other Exchange directories require SSL, but OAB should not.
• The OAB virtual directory is pointing to the wrong spot. I deleted the default OAB and created a new one, but the new one was in a physical path different from the physical path of the old one (not just the GUID of the OAB, but the entire path was different!). I had to re-target the OAB virtual directory in IIS to get it looking in the right spot.

The path is being blocked by a firewall/ISA
If your problem only crops up when the user is trying to get access via “Outlook Anywhere” (or RPC or HTTP), make sure that what you’ve published in the firewall or ISA server is allowing the path /OAB/* through to the correct server.

Missing entry for autodiscover in DNS
This was my final “gotcha” to overcome. After spending hours doing packet captures, I noticed that on a clean DNS table and a fresh Outlook start (in other words, close Outlook 100%, do “ipconfig /flushdns” and start Outlook) I saw that Outlook was trying to lookup “autoconfigure.domain.com” as a host, and when that failed, it was hitting “_autoconfigure._tcp.domain.com” as an SRV record. Making an A or a CNAME record for autoconfigure.domain.com is silly, since I would need to spend money on another SSL certificate for it. Actually, that is debatable, since the only clients trying to hit that address should be clients that already trust my internal root CA, so I could self-issue one. But I digress. To make things slightly less chaotic (do I really need to publish Yet Another Host Name in ISA anyways?), I simply added an SRV record for “_autoconfigure._tcp.domain.com” with the data of “0 0 443 mail.domain.com” (mail.domain.com is the external address for the Exchange server, which already has a cert on it and it is the name of the Exchange proxy server, etc.). Once I did that, the last of the problems were solved.

I hope this helps someone. I’ve seen some of these items listed elsewhere (sadly, the bits on autoconfigure turned up on my final search, in tabs that I checked after I resolved the problem!), but never all of them in one spot.

J.Ja

Today I was trying to export an Exchange 2007 mailbox to a PST file. The command gave me an error:

Cannot open the log file ‘C:Program FilesMicrosoftExchange ServerLoggingMigrationLogsexport-Mailbox20081019-110212-4778281.log’.
At line:1 char:1

This error is because I am on Vista; you need to run the Exchange Command Shell as Administrator to have the rights to that directory. To fix the problem, right-click the Exchange Command Shell and choose “Run as Administrator” and try again.

J.Ja

I’ve been slaving away over a hot Exchange 2007 server on Windows 2008 for a week now. Great product, except for the Outlook Anywhere component, which does not install itself properly. The issues are well documented by everyone except for Microsoft, it seems like. In any event, over the course of my work, I started seeing event ID 4001 in the logs a lot, with a message that said, “A transient failure has occured. The problem may resolve itself in a while. The service will retry in 56 seconds.”

Another symptom I saw was that OWA (Outlook Web Access) would state that there was a transient failure as well when people tried to access it. The diagnostic information mumbled some stuff about not being able to open a mailbox. This is caused by an inability for the OWA server to communicate properly with the CAS, usually itself. In this case, while following some troubleshooting documents, I had added lines like:

192.168.1.XYZ   NETBIOSNAME
192.168.1.XYZ   FQDN

to my hosts file. Taking those out and running “ipconfig /flushdns” corrected the issue. I did not see any information on the Web about this, so I figured I should post some.

J.Ja

Today I set about installing Exchange 2007 in our organization. As a little bit of background, we had initially had a windows 2003 SBS Server in place, which means that our Active Directory had the Exchange changes made to it. When we moved off of that system, I manually uninstalled Exchange before wiping the server, but in my ignorance I did not remove Exchange from Active Directory. Today, I found a way to do that, despite errors from trying “update /removeOrg” in the Exchange SP2 system.

In a nutshell, the proper way to get Exchange out of Active Directory is to run “update /removeOrg” from an SP2 installer. Sometimes it doesn’t work. For me, it was spewing errors about trying to prepare the forest (why, when I’m trying to unprepare the forest?). After a day of search, it seemed like I was the only one ever to need to manually extract Exchange from Active Directory. What I ended up doing was quite simple. I ran ADSI, and removed the Microsoft Exchange OU in the default schema. That’s the obvious one. But that wasn’t enough. I also needed to switch to the “Configuration” schema (right click the domain name in ADSI on the left-hand tree and choose “Settings”, and change the “Select a well known naming context” dropdown to “Configuration”), drill down to “Services” and remove the “Microsoft Exchange” CN there as well.

Presto, no more Exchange!

If you want/need to, you should also remove the groups in Active Directory, but this appears to be unnecessary for moving to Exchange 2007.

J.Ja