Kernel Confusion

Last night I had to upgrade our existing Threat Management Gateway RC machine to the final version of the product. According to TechNet this seemed to be a simple task, only a few steps are needed:

1. Exporting the Forefront TMG RC configuration.
2. Uninstalling Forefront TMG RC from the server.
3. Installing Forefront TMG RTM on the server.
4. Importing the Forefront TMG RC configuration into Forefront TMG RTM.

Step one was simple enough. Just follow the TechNet instructions:

To export the Forefront TMG RC configuration

The deinstallation was not that easy, however. First, the TMG itself was deinstalled. Next the SQL Server had it’s turn, but failed with some wired errors. Unfortunately, I don’t have the logs anymore, so I can’t post them here. As the TMG wasn’t listed in the contral panel at “Installed Software”, I guessed I could try to install the RTM right away… and was wrong. It failed AGAIN when installing the SQL Server.

To solve this, I had to manually remove any SQL components left over and I renamed any SQL related directories under %programfiles% and %programfiles(x86)%. This time the setup did it’s work as expected and I imported the system-configuration back into the Firewall. At the first start, cancel the wizard and follow these steps:

To import the Forefront TMG RC configuration

At a first look, it all worked well. Internet-Access was available again and the Exchange started to receive and send E-Mails again. But my Microsoft Office Communicator 2007 R2 was unable to connect. Also, my virtual test-machine failed to establish the IPHTTPS tunnel for Direct Access while 6to4 apparently worked. The IPHTTPS tunnel the the most use way for us, so it had it’s importance.

Solution to the unresponsive Office Communication Server (OCS)

As a matter of fact, all settings were imported, but apparently NOT IN THE RIGHT ORDER. While the normal policies looked right, the network rules were ordered randomly. The rule regulating the traffic between DMZ and internal LAN (routing) was below a NAT rule and thus not functional. Restoring the original rule order solved the connection problem.

Solution to the nonfunctional DirectAccess (DA)

Let me note here, that we have both the TMG and the DA on the same machine, so this problem might be unique to this environment. I tried to open the IPHTTPS URL in the Browser and got a certificate error. As you may already know, certificates are a pain and absolutely important for any DA connection. I found out the wrong cert was presented to the client. So I checked the DirectAccess MMC and made sure the setting were correct. I even went through all four configuration panels and applied the newly generated config XML. But the certificate didn’t change. After endless tries, I surprisingly messed up the config so badly, that the wizard wasn’t able to apply it anymore and told me to undo the current configuration. I did as told and even had to manually remove both the DA GPOs left over. After that, I rebuild the config (with the exact same details as before)… and it worked. New GPOs were created and the right certificate was published. I don’t really know what went wrong, but this is how you can solve it.