Category Archives: Windows Server 2008

Solution for an empty “Network Connections” in Windows

Yesterday, I had to do some work on our Forefront Threat Management Gateway machine. When I brought up the TMG console, it gave me a strange error: “Refresh failed” with an error code of 0x80004005. It was inexplicable. A few days earlier, we noticed that the “Network Connections” in control panel showed no connections at all, but ipconfig showed them as expected. I ended up placing a call to Microsoft support. They suspected that the TMG console issue was caused by the inability to enumerate the network connections, and I was inclined to agree. Their specialist for these things said that there’s a registry key which sometimes gets corrupted, and you can delete it and reboot the server to fix the issue. After carefully reviewing to ensure that nothing else was the issue, that’s just what we did. After the reboot, the network connections showed, and the TMG console issues were solved as well. To do this fix yourself (the usual disclaimer: back up your registry before editing!), look up the following key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Network and delete the “Config” value.


Fix for 0x80072f0c error (502.3 – Bad Gateway) for reverse proxy to SSL with IIS

I’ve spent most of a week struggling with this error. I set up IIS to reverse proxy to a backend server using the URL Rewrite module and the Application Request Routing (ARR) module. The first problem I encountered was that when using the “Reverse Proxy” wizard/template under URL Rewrite, it kept blowing up, giving me an error 500. The solution for this was to first go to “Server Variables” and add “HTTP_ACCEPT_ENCODING” as an allowed server variable. Next, I had to go into the configuration and set HTTP_ACCEPT_ENCODING to be passed to the destination server with an EMTPY value. You can’t do this direction from the configuration screen, because that demands a value. You can do it in web.config (or anywhere in the configuration chain). I did it by going to the “configuration editor” in IIS Manager to edit the value raw with no validation.

The next problem was much trickier. The reverse proxy template was able to handle carrying over SSL just fine to the backend server, but when I tried to access those links, it would blow up, giving me an error 502.3. Turning on detailed error reporting showed me an error code of 0x80072f0c and the text “HTTP Error 502.3 – Bad Gateway”. Full details showed more confusion under “possible causes”:

The CGI application did not return a valid set of HTTP errors.
A server acting as a proxy or gateway was unable to process the request due to an error in a parent gateway.

This made no sense to me at all. After hours of work on this issue, I finally found the problem. The virtual directory on the destination server (the one BEHIND the proxy) had been set to “Accept” client SSL certificates; this needs to be set to “Ignore”. While the site itself was set to “Ignore”, the virtual directory had been created with “Accept”, causing the problems.


How to perform a P2V conversion for FreeBSD to run on Hyper-V

One of my big projects with my personal server setup, was to turn my current physical FreeBSD server into a Hyper-V VM. Why would I do this? Don’t ask, because I don’t want to start a religous war here… let’s just say that as much as I like FreeBSD for a lot of purposes, I do not like living with it as a sys admin without a paycheck attached.

So, here’s how I went from FreeBSD on a physical machine (garbage x64 hardware) to a Hyper-V VM (Windows 2008 R2 on garbage x64 hardware).

  1. Upgrade the FreeBSD machine to version 8.0-RELEASE. This is mandatory.
  2. Get Hyper-V installed and configured, including enabling Intel VT in the BIOS.
  3. Shut down both machines. Transfer the physical hard drive from the FreeBSD machine to the Windows 2008 R2 machine. Turn on the 2008 R2 machine, and verify in Disk Management that the transferred drive is visible.
  4. Create a new Hyper-V VM for the machine, but do not specify a hard disk. Go back into the settings, and remove the NIC that was put into the VM. Do “Add new hardware” and select “Legacy Network Adapter”, and connect the new NIC to the network of choice.
  5. Create a new virtual disk. Select “Fixed” type, and on the next page in the wizard, tell it to copy the contents of a physical disk. Choose the disk you transferred from the FreeBSD machine.
  6. Go eat dinner, walk the dog, read a magazine. You’ll be here a while during the disk copy. To be on the safe side, go download the “Live FS” FreeBSD ISO appropriate for your installed FreeBSD version.
  7. Once the new virtual disk has been created, go back into the VM settings, move the optical drive to postion 1 on the IDE chain, and then add the newly created disk to the VM on position 0 on the IDE chain.
  8. Start the VM. If you receive errors like “Invalid slice”, you need to do the following:
    1. Insert the Live FS ISO into the virtual DVD drive and reboot the VM.
    2. Go to “Configure” and then “Fdisk”. Set the main drive slice (the big one) to be bootable, and then press “W” to write the information to disk. Before it writes, it will ask about a boot loader; choose the standard one, unless you have a good reason not to and know what you are doing.
    3. Exit the Live FS system, eject the ISO, and reboot the VM.

    This should take care of the “bad” boot loader.

  9. If the physical disk in the original server was not device “ad0” (for example, it was a SCSI drive or a RAID 1 member), then the system will spaz when you boot and drop to single user mode. Not to worry! In single user mode, do the following: (note: if you can’t even get into single user mode, boot off of the Live FS CD and use the “Fixit” shell)
    1. Re-mount the root partition as writeable with:mount -u /
      mount -a
      Likewise, mount /usr and /tmp with:
      mount /dev/ad0s1f /usr
      mount /dev/ad0s1e /tmp
    2. Now you can actually use your text editor of choice to edit /etc/fstab and set the references to the old drive to be references to the new drive as ad0. Do that and reboot.
  10. You are in the home stretch now! You should be booted into FreeBSD, albeit a crippled one, because the NIC isn’t configured. Go edit /etc/rc.conf and change the reference to your old NIC to be a reference to de0 (the NIC that Hyper-V provides). Reboot again, and you should be done!

This is what I did… it might not work 100% for you, for better or for worse.


Hyper-V in Server 2008 R2 has one super new feature

I put together a Windows Server 2008 R2 box over the weekend (my old Vista machine is now the server). I spent part of today working on trying to do a P2V conversion of my FreeBSD server to bring it onto the box in Hyper-V. At first, I tried using Acronis Home 2009 to clone the disk, then re-clone into the VM, like I’ve done before. For whatever reason, it did not like the RAID in the FreeBSD box, and wouldn’t read the data from it. Along the way, I decided to prep the new Hyper-V VM, and lo and behold, I discovered it’s super new feature: when creating a new virtual hard drive, you can copy an existing physical drive (not “file system”, the entire drive!) as the contents of the virtual drive. This means that you can take the disk out of the old system, hook it up to the new system, clone it into the VM really quick, and be on your merry way. The only real drawbacks are that you cannot do a dynamically expanding disk like this, so the new VHD is the same size as the physical disk it was clone from (although it will be fast), and that it takes forever because it does a sector-by-sector copy of the disk. While this is still not a proper substitute for a true P2V agent-based conversion, this is pretty darned close, especially for OS’s that are not mainstream enough to justify someone writing the conversion agent.


Temporary workaround for Windows SMBv2 zero-day

The Windows SMBv2 zero-day vulnerability (disclosed vulnerability with no software fix) appears to be more dangerous than initially thought.  The vulnerability does not affect the Release to Manufacturing (RTM) version of Windows 7 or Windows Server 2008 R2, but it does affects Windows Vista and Windows Server 2008.  The danger is no longer just a system crash or reboot, it can lead to a full system compromise.

In the absence of a patch, Microsoft released some instructions for disabling SMBv2.  For your convenience, I’ve packaged two REG files that you can download that enable and disable SMBv2 in Windows Vista and Windows Server 2008.  So until a software patch is available, you need to disable SMBv2 double clicking the disable-SMBv2.reg file and then rebooting.  The workaround does not break your ability to serve files, but it does reduce your SMB file serving speeds down to Windows XP and Windows Server 2003 levels which would result in a moderate decrease in performance.  When the patch becomes available and you have applied the patch, just run the enable-SMBv2.reg file and reboot.

BlackBerry Enterprise Server 5 installation nightmares

If you’re going to be installing BlackBerry Enterprise Server (BES) 5.0 on a Windows Server 2008 machine, you better be ready to call technical support or read this blog post at a minimum.  I spend 5 hours on the phone with BlackBerry customer support over the course of two days to work out all the installation problems that should have been automated by the installer and documented in the pre-installation guide.  I will give Research In Motion support credit for helping me through these problems.

The main different between BES 5.0 and BES 4.1.x is that the administration interface is Java and ActiveX web based only, which irritates me to no end.  BES 4.1 gives you a real interface that works without a browser and isn’t dependent on some complicated JBoss+Apache web server setup.  This web based aspect of BES 5.0 was the source of some major installation and configuration headaches.  With BES 4.1.x, you don’t have to put up with any of this nonsense.  The reasoning behind 5.0 using a web administration interface is that now the users can manage their own BlackBerry Enterprise accounts though I really didn’t need this feature.

Just getting the basic preparation work done for a BES 5.0 install is daunting enough because you have to go through this pre-installation guide.  That involves setting up a BES service account, the local server permissions, the active directory permissions, the exchange server permissions, and other software that needs to be pre-installed.  It gets a bit confusing and I definitely recommend just using the SQL Server 2005 express they include in their BES 5.0 installer and then you don’t have to worry about additional JDBC drivers for remote SQL databases.  Don’t bother installing SQL express yourself as it will only make things more complicated.  You will need to install the Microsoft Exchange MAPI client before you install BES.

One pre-installation step that is missing is that you need to make sure that IPv6 on Windows Server 2008 is disabled.  Un-checking IPv6 in the network interface is not sufficient, and you’ll need to follow the instructions in this Microsoft KB article (Jeremy in the comment section below pointed out that the updated KB article is here) and edit the registry.  [NOTE – Disabling IPv6 will break Windows Small Business Server (SBS) which means you don’t mix BES with SBS].  Once you’ve done this along with all the other pre-installation procedures above, you can proceed to install BES.  Note that during installation, be sure to select BlackBerry Server authentication for the web administration interface and not Active Directory (AD) integrated authentication.  BlackBerry technical support couldn’t figure out how to get this working and they told me to reinstall from scratch on a clean machine and don’t use AD authentication for web administration.

There’s more pain after you’ve completed the whole installation.  The web administration interface also needs access to the SQL database which doesn’t work out of the box and can frustrate you to no end.  Because the BES installer didn’t bother to nail down the SQL ports from dynamic to fixed TCP 1433, the web admin page refuses to come up.  You need to go into the SQL Server Configuration Manager and manually set the TCP ports to 1433 as shown below.

BlackBerry Enterprise Server 5.0 SQL configuration

Internet Explorer 8.0 in Windows Server 2008 also requires you to enable “compatibility mode” for the administration page and you must also put the administration URL in the trusted site list.  The other problem you’ll find is that because the BES 5.0 installer generated its own SSL certificate rather than using the one that is either already on the server or easily obtained in a Windows network environment with a Certificate Authority in place, the browser throws up error messages that it doesn’t trust the SSL certificate.  You can fix this by right clicking on Internet Explorer and choose “Run as administrator”.  Then you can view the certificate and install it.  However, it’s installed in the wrong place in the personal user certificate store and you’ll have to export the certificate and import it into the computer certificate.  A better option is to import the certificate into Active Directory trusted certificates using this procedure I wrote up in 2006 and that solves your problem for every computer within the active directory that needs to use the BES web administration page.

Anyhow, it’s all working for me right now and I hope this document helps you avoid the headaches I went through.

My experiences with Windows Server 2008

In July, we put up our first Windows Server 2008 server. Since then, I have migrated the domain from NT 4 to the new domain (which was upgraded from 2003 to 2008 too), put in Office Communications Server 2007 (OCS), Exchange 2007, SharePoint 2007, and I am in the process of getting Dynamics CRM 4.0 in place. After all of this, I think I can fairly report on Windows Server 2008.

First of all, it works, it works better than any other Microsoft OS I have encountered. They significantly cleaned up and reorganized the interface so that logically related tasks are “closer” to each other and are sometimes in the same tool now, as opposed to using 4 separate tools with wildly different interfaces to get things done. PowerShell is suddenly a “big deal”, and many of the GUI management tools really are just wizards to construct & run PowerShell scripts on the fly (Exchange 2007 Management Console is a great example, it even shows you the PowerShell command it is about to run). Now that I’ve had to use PowerShell, I like it, but only because I am familiar with *Nix; it’s like someone took the *Nix model of pipes and indirection, and instead of letting all of the commands be developed hodgepodge by different people with no common naming conventions, format conventions, etc., it was all centrally managed and therefore, logical.

Things I don’t like? Application incompatability. A lot of apps (ISA, Office Communication Server) don’t work on it. Others require odd modifications (Exchange 2007, pre-rollup 4, needed some bizarre hacks to disable IPv6 in some common scenarios) because the applications don’t work “right” with it. I’m unhappy that Microsoft is pushing some products to 64-bit only (like Exchange 2007), while others refuse to run under 64-bit (Office Communication Server). This mandates that you have at least a 64-bit and a 32-bit install, and possibly (probably) a 32-bit Windows Server 2003 machine around too. I’m furious that many of the updated products are 64-bit only, which means that instead of a simple upgrade, you need to bring the new version into the “pool”, transfer responsibility to it, then gracefully disable the old 32-bit server from the pool, and then uninstall it; this is the “upgrade” path for OCS 2007 R2, in a nutshell. And Exchange 2007, for that matter. Not a nice thing to do. Personally, I find that it is easier to see what my options are when they are in a GUI than a command line, I just walk the menu tree and go into every dialog. But that is a personal thing. At the same time, many tasks can only be done in PowerShell, so if you don’t like the command line… tough.

Hyper-V ROCKS. I can’t compare it to VMWare ESX server, but I can tell you that it beats VMWare Server, Virtual PC 2007, and Virtual Server 2005 with the ugly end of the ugly stick, and leaves them half dead in a Moscow alleyway. The biggest gripe I have, is that its missing features are in the (just RTMed) System Center Virtual Machine Manager product. Granted, much, if not most of that functionality is already available via PowerShell (once again, the idea that GUI tools are now just PowerShell script construction wizards). But still, it would have been nice to easily get P2V and V2V conversions up front with a GUI tool.


Broken Search in Windows Server 2008

Windows Server 2008 has the most broken feature I have seen in a Micrisoft product in a long time: file searching. Like Vista, Server 2008 has some really nice file searching features that make heavy use of the file indexing. Sadly, some idiot decided to make the system search only indexed items by default, and then turned off the indexing by default!

The end result? A hopelessly broken search system that is insanely frustrating to use. You either need to manually tell each search to use non-indexed content, turn on indexing (you may very well not want to do this on a server!), or change the default settings to always use non-indexed content. Whoever set up these defaults is a fool. It’s a lot like the problem in Server 2003, where the checkbox for “Search Tape Backups” also seems to regulate whether or not it will actually search on a network drive, even if you’ve specified a network drive as being where to search.


NT 4 to Windows 2008 Migration – almost ready!

The monster project on my plate (I’ve been building up to it since around March) is to migrate our existing NT 4 domain to Windows 2008. This project has been joy and pain, and it is finally nearly done.

For the last few months, I’ve been getting the new domain ready, like upgrading the domain controller to server 2008, getting a new SQL Server install in place, SharePoint, and so on. I still need to do Exchange, CRM 4,0, and Office Communications Server, but we agreed that those items need to wait until after the migration.

When I went to do the initial batch of migrations, though, I hit a snag. The Active Directory Migration Tool (ADMT) version 3.0 supports migrating from NT 4, but not migrating to 2008. The newest version 3.1, supports 2008 as a target but not NT 4 as a source. So we needed to get the NT 4 server upgraded to 2000 or higher. For safety’s sake, we decided to use a VMWare image of the server for this.

The VMWare conversion process worked fine, but when we fired up the VM, it claimed that there was no system disk. This wasn’t a huge surprise – the original machine is an ancient Compaq Proliant with an EISA SCSI controller in it; the MBR points to a tiny 36 MB partition on the RAID 1, which contains the SCSI tools to get into the card’s firmware, and then boot off of the true C drive. Gotta love 1990’s technology. After contacting VMWare, we decided that the best route was to do the following:

  1. Take a drive image of the original server, and copy it to the VMWare computer’s local drive.
  2. Create a new VM with disks of the appropriate size (I made the C drive 20 GB larger than the original, to provide ample space for the upgrade to take place), and also mount the local drive with the image file as a disk in the VM.
  3. Start the VM and boot off of the imaging software’s CD.
  4. Blast the image onto the virtual drives.
  5. Copy the VM back to the NT 4 server (to ensure the same version of NTFS) and run the virtual machine conversion wizard.

This worked great, except for when it didn’t work. Why not? Well, we still didn’t have an MBR pointing to the right place, since we weren’t going to get the EISA SCISI tool partition (we tried it once on a drive image, it complained about jumpers…). So what did we do? We made floppy images of the NT 4 install floppies, and a floppy image of a fresh NT 4 Emergency Repair Disk, and ran the NT 4 recovery mode, to “Inspect Boot Sector”. That fixed the MBR issue!

Now, we got NTLDR issues. So we brought the VM back to the NT 4 server, and tried to run the conversion utility. It groused about not being able to identify the OS. Huh? Looking at the VMWare converter logs, we found the problem. It turns out that the VM was still set to mount the local drive of the VMWare workstation; removing that virtual drive solved that problem, and the conversion continued.

And lo and behold, it worked! We actually managed to virtualize a server that I originally built when I was (I beleive) a sophomore in college. This machine was my introduction to SCSI, TCP/IP, NT 4 (I had experience with NT 3.51, 3.5, and 3.1 before that, as well as NetWare), multi-CPU machines (it had 2 CPUs, amazing for the time), and a lot of other technologies (DNS and DHCP come to mind immediately). This machine really got me started hardcore in systems administration. And now it is a VMWare VM.

But I digress.

We then performed the upgrade to Windows Server 2003 R2. This went extremely well; the only hiccups we had were remembering to make a floppy image containing the VMWare SCSI controller driver to feed to the Windows setup program, and then remembering to disconnect the floppy image before the next reboot (we got another scary NTLDR error… woops!).

On a side note, we needed to make the Active Directory install post-upgrade be in a completely separate forest, since the 2003 domain can’t participate in the 2008 forest.

But we are now finally ready to migrate this domain, and I can’t wait. If our NT 4 domain could last from 1997 to 2008, I shouldn’t have to upgrade this domain until around 2019. :)


Windows Search 4.0 for XP, Vista, and Server 2008

Betanews has a good article about Windows Search 4.0 which is now listed as “important” in Windows Vista Update.  You can manually download a copy here for Windows Vista and Server 2008 and you can download Search 4.0 for Windows XP.  Couple this with Microsoft’s free enterprise search and you have a complete search solution for your whole business or organization at little or no additional cost.