Ever since I put up an ISA Server 2006 deployment, we noticed that Web access through it was incredibly slow. When we connected test machines directly to the Internet connection (the FiOS line that I've mentioned before), it was blazing fast. But outbound Web access through the ISA Server was slow slow slow. Well, I fixed it!
When I started troubleshooting it, I first looked to the Event Viewer for failures. Other than a few minor items about DNS, I didn't see anything in there at all. We tried all of the usual diagnostics, and it was clear that network latency was not the problem. I also tried a good number of other tests, including DNS lookups, all of which looked good.
Performance monitoring showed that the local SQL Server installation (for catching the logs) was Hoovering RAM. So I got into SQL Server and capped its memory usage at 512 MB of RAM. We saw that the overall RAM consumption was down; it now ran with plenty of free memory, but the Web access was still slow. Outgoing access for all other protocols was super fast, incoming traffic was super fast. What gives?
Next, I decided to take a look at the DNS problems in Event Viewer. You can never be too careful, and who wants to have errors anyways? I fixed the problems with DNS up, and some sites seemed to improve, but the overall access was still quite slow. The trend now seemed to be that frequently accessed sites (based on hostname) were speedy, but everything else was slow. This deepened my suspicions of DNS problems.
At this point, I had nowhere else to turn to. Monitoring and performance logging told me what I already knew, that everything was fine except the retrieval of non-cached Web pages. I was typing up an email about the problem, enumerating all of the potential points of slowness and why or why I could rule them out. Everything could be ruled out easily, except for the DNS situation. Who knows what the internal DNS stack is really doing, especially when the ISA Server is its own DNS server?
So I decided to take yet another look at the DNS configuration. The configuration of the local DNS server was perfect. No problems found, no errors in Event Viewer. On a whim, I checked the DNS entries of all of the NICs. The only entry of note, was that the NIC on the LAN had an entry for an alternate DNS server, an entry for a machine that was no longer in use. That entry was put in as a "fallback" entry when the server was first installed, and it pointed to the previous firewall, in case the previous firewall had a DNS entry that we had not moved to the new DNS nameservers yet.
But how much trouble could this cause, right? After all, the lookup should be occuring on the WAN NIC, not the LAN NIC, and this is the alternate DNS server, which should never be hit anyways! Well, I removed the entry just for correctness, and BAM. Problem solved.
Microsoft, sometimes you baffle me. Why in the world would the alternate DNS server for the LAN NIC affect perforfmance on the WAN NIC, especially when the primary DNS server for all NICs is localhost, and localhost's DNS server forwards to the WAN ISP's DNS servers when needed? And why would this only affect Web access, and not FTP, SMTP, etc.? Regardless, if you are seeing insanely slow Web throughput on your ISA Server 2006 install, check your DNS subsystem as a whole completely before giving up.
J.Ja