In this post, I discuss some of my activities for the next couple of months. These include the INTERalliance TechOlympics, SharePoint Saturday Michigan, and continuing efforts to get the SharePoint 2010 Disaster Recovery Guide ready for product launch.
2010 is in full-swing, and there seems to be no shortage of activities for me to jump into! If anything, I need more free time to take on some of the stuff I really want to sink my teeth into (such as a SharePoint 2010 CodePlex project I want to have ready for RTM). Until I have something more tangible in hand, though, I’ll avoid talking about that topic any further.
Here are some of the things occupying my free time in the short-to-mid term:
TechOlympics Expo 2010
The TechOlympics Expo is the type of event every adult geek wishes they had when they were in high school – a weekend lock-in featuring technical competitions, cool toys, games of every imaginable sort, and pretty much everything else that would get a teenage gearhead jazzed-up. The underlying goal of the event is to get high school kids interested in technology, careers in technology, and technical opportunities in the Cincinnati area.
The event (on March 5-7) is being put on by the INTERalliance of Greater Cincinnati, and my involvement in the event is kind of a curious thing. My primary client of the past 2+ years is a big backer of (and heavily invested in) the INTERalliance, so naturally they kick-in help whenever events come up. I helped the INTERalliance through a last-minute (and somewhat ugly) technical hurdle involving SMS voting for their PharaohFest event last October, and I suspect that played a part in my being asked to help out with the TechOlympics.
With the TechOlympics, I’m part of a team that’s working to make all the “technical stuff” (behind-the-scenes and otherwise) happen. My responsibilities seem to shift a bit each day, but the bulk of what I’ve been working on is coordinating network logistics and services, translating “the vision” into technical infrastructure, providing some guidance on applications being written to support the event, and generally doing my best at “collision avoidance” to ensure that we don’t miss anything important for the event.
I’m confident that the event is going to be incredible, and it’s been a lot of fun doing the planning thus far. Seeing everything come together is going to be neat – both for me and for everyone else who has been laboring to make the magic happen!
SharePoint Saturday Michigan
What would an “Upcoming Activities” post be without a SharePoint Saturday announcement! The next one I’ll be attending is SharePoint Saturday Michigan in Ann Arbor on March 13th. I’ll be presenting “Saving SharePoint,” the disaster recovery talk that John Ferringer and I have been delivering at various SharePoint Saturday events around the region. I’ll be flying solo this time around, though, as John has some other things going on that weekend.
As always, SharePoint Saturday events are free and open to the public. If you have any interest in learning more about SharePoint, getting some free training, or simply networking and meeting other professionals in the SharePoint space, please sign up!
SharePoint 2010 Disaster Recovery Guide
This announcement is last, but it’s definitely not least. Some of you are aware, but for those who aren’t: John and I have been working on the SharePoint 2010 Disaster Recovery Guide for a while now. I’m not going to lie – it’s slow going. Personally, I’m a very slow writer, and the process itself is exceptionally labor-intensive. Nevertheless, we’re making progress – one page at a time.
Our goal (and Cengage’s goal for us) is to have the book ready for SharePoint 2010 RTM. I haven’t seen or heard anything official from Microsoft, but rumor has it that SharePoint 2010 will probably be out sometime in June. If that’s the case, then John and I are on-track.
If you have suggestions for us, particularly if you read the first book, we would love to hear them. We’re incorporating a few that we already received (for example, a chapter that covers some real world use-cases), but our ears are open and listening. We know that DR isn’t a topic that gets everyone overly hot and bothered (unless they’ve lost everything at some point, of course), but our goal is to make the book as useful as possible. We’d love your help!
In this post, I discuss a couple of DCOM/RPC snags I ran into while configuring Microsoft’s Data Protection Manager (DPM) 2007 client protection agent to run on my new Forefront Threat Management Gateway (TMG) 2010 server. I also walk through the troubleshooting approach I took to resolve the issues that appeared.
I completed the move about a week ago, and I’ve been very happy with TMG thus far. TMG’s ISP redundancy and load balancing features have been fantastic, and they’ve allowed me to use my Internet connections much more effectively.
As a user of ISA since its original 2000 release, I also had no problem jumping in and working with the TMG management GUI. It was all very familiar from the get-go. Call me “very satisfied” thus far.
This afternoon, I took a few moments to un-join the old ISA servers from the domain, power them down, and clean things up. I had also planned to take a little time integrating the new TMG box into my Data Protection Manager (DPM) 2007 backup rotation. Unfortunately, though, the DPM integration took a bit longer than expected …
Backup Brigade
For those unfamiliar with the operation of DPM, I’ll take a couple of moments to explain a bit about how it works. In order for DPM to do its thing, any computer that is going to be protected must have the DPM 2007 Protection Agent installed on it. Once the DPM Protection Agent is installed and configured, the DPM server communicates through the agent (which operates as a Windows service) to leverage the protected system’s Volume Shadow Copy Service (VSS) for backups.
Installing the DPM agent typically isn’t a big challenge for common client computers, and it can be accomplished directly from within the DPM management GUI itself. When the agent is installed through the GUI, DPM connects to the computer to be protected, installs the agent, and configures it to point back to the DPM server. No manual intervention is required.
On some systems, though, it’s simply easier to install and configure the agent directly from the to-be-protected system itself. A locked-down server (like a TMG box) falls into this category, so I manually put the agent installation package on the TMG server, ran it, and carried out the follow-up PowerShell Attach-ProductionServer (from the DPM Management Shell) on the DPM server. The install proceeded without issue, and the associated attach (on the DPM server) went off without a hitch. I thought I was good to go.
I fired up the management GUI on the DPM Server, went into the Agents tab under Management, and discovered that I couldn’t connect to the TMG server.
The Checklist
The fact that I couldn’t connect to the TMG server (SS-TMG1) from my DPM box was a bit of an eyebrow lifter, but it wasn’t entirely unexpected. Communication between a DPM server and the DPM agent leverages DCOM, and I’d had to jump through a few hoops to ensure that the DPM server could communicate with the ISA boxes previously.
I suspected that an RPC/DCOM issue was in play, but I was having trouble seeing where the problem might be. So, I reviewed where I was at.
Without an exception, Windows Firewall will block communication between a DPM server and its agents. I confirmed that Windows Firewall wasn’t in play and that TMG itself was handling all of the firewall action.
Examining TMG, I confirmed that I had a rule in place that permitted all traffic between my DPM server (SS-TOOLS1) and the TMG box itself.
Strict RPC compliance is another potential problem with DPM on both ISA and TMG, as requiring strict compliance blocks any DCOM traffic. DCOM (and any other traffic that doesn’t explicitly begin RPC exchanges by communicating with the RPC endpoint mapper on the target server) gets dropped by the RPC Filter unless the checkbox for Enforce strict RPC compliance is unchecked. I confirmed that my rule wasn’t requiring strict compliance (as shown on the right).
I made sure that my DPM server wasn’t listed as a member of either the Enterprise Remote Management Computers or Remote Management Computers Computer Sets in TMG. These two Computer Sets are specially impacted by a couple of TMG System Policy rules that can impact their ability to call into TMG via RPC and DCOM.
I reviewed all System Policy rules that might impact inbound RPC calls to the TMG server, and I couldn’t find any that would (or should) be influencing DPM’s ability to connect to its agent.
I also went out to the Forefront TMG Product Team’s Blog to see what advice they might have to offer, and I found this excellent article on RPC and TMG – well worth a read if you’re trying to troubleshoot RPC problems. Unfortunately, it didn’t offer me any tips that would help in my situation.
Watching The Traffic
I may have simply had tunnel vision, but I was still obsessed with the notion that strict RPC checking was causing my problems. To see if it was, I decided to fire-up TMG’s live logging and see what was happening when DPM tried to connect to its agent. I set the logging to display only traffic that was originated from the DPM box, and this is what I saw.
There was nothing wrong that I could see. My access rule was clearly being utilized (the one that doesn’t enforce strict RPC checking), and I wasn’t seeing any errors – just connection initiations and closes. Traffic from DPM to TMG looked clean.
Taking A Step Back
I was frustrated, and I clearly needed to consider the possibility that I didn’t have a good read on the problem. So, I went to the Windows Application event log to see if it might provide some insight. I probably should have started with the event logs instead of TMG itself and firewall rules … but better late than never, I figured.
Popping open Event Viewer, I was greeted with the image you see on the left. What I saw was enlightening, for I did have a problem with communication between the DPM agent and the DPM server. The part that intrigued me, though, was the fact that the problem was with outbound communication (that is, from TMG server to DPM server) – not the other way around as I had originally suspected. All of my focus had been on troubleshooting traffic coming into TMG because I’d been interpreting the errors I’d seen to mean that the DPM server couldn’t reach the agent – not that the agent couldn’t “phone home,” so to speak.
I knew for a fact that the DPM Server, SS-TOOLS1, didn’t have the Windows Firewall service running. Since the service wasn’t running, there was no way that the agent’s attempts to communicate with DPM could (or rather, should) be getting blocked at the destination. That left the finger of blame pointing at TMG.
On The Way Out
I decided to repeat the traffic watching exercise I’d conducted earlier, but instead of watching traffic coming into the TMG box from my DPM server, I elected to watch traffic going the other direction – from TMG to DPM. Here’s what I saw:
The “a-ha” moment for me came when I saw the firewall rule that was actually governing RPC traffic to the DPM box from TMG. It wasn’t the DPM All <=> SS-TMG1 rule I’d established — it was a system policy rule called [System] Allow RPC from Forefront TMG to trusted servers.
System policy rules are normally hidden in the firewall policy tab, so I had to explicitly show them to review them. Once I did, there it was – rule 22.
Note that this rule applies to all traffic from the TMG server to the Internal network; I’ll be talking about that more in a bit.
I couldn’t edit the rule in-place; I needed to use the System Policy editor. So, I fired up the System Policy Editor and traced the rule back to its associated configuration group. As it turned out, the rule was tied to the Active Directory configuration group under Authentication Services.
As the picture on the left clearly shows, the Enforce strict RPC compliance checkbox was checked. Once I unchecked it and applied the configuration change, the DPM agent began communicating with the DPM server without issue. Problem solved.
What Happened?
I was fairly sure that I hadn’t experienced this sort of trouble installing the DPM Protection Agent under ISA Server 2006, so I tried to figure out what might have happened.
I hadn’t recalled having to adjust the target system policy under ISA when installing the DPM agent originally, but a quick boot and check of my old ISA server revealed that the checkbox was indeed unchecked (meaning that strict RPC compliance wasn’t being enforced). I’d apparently made the change at some point and forgotten about it. I suspect I’d messed with it at some point in the distant past while working on passing AD information through ISA, getting VPN functionality up-and-running, or perhaps something else.
Implications
Bottom line: TMG enforces strict compliance for RPC traffic that originates on the TMG server (Local Host) and is destined for the Internal network. Since System Policy Rules are applied before administrator-defined Firewall Policy Rules, RPC traffic from the TMG server to the Internal network will always be governed by the system policy unless that policy is disabled.
In this particular scenario, the DPM 2007 Protection Agent’s operation was impacted. Even though I’d created a rule that I thought would govern interactions between DPM and TMG, the reality is that it only governed RPC traffic coming into TMG – not traffic going out.
In reality, any service or application that sends DCOM traffic originating on the TMG server to the Internal network is going to be affected by the Allow RPC from Forefront TMG to trusted servers rule unless the associated system policy is adjusted.
Conclusion
The core findings of this post have been documented by others (in a variety of forms/scenarios) for ISA, but this go-round with TMG and the DPM association caught me off-guard such that I thought it would be worth sharing my experience with other firewall administrators. If anyone else moving to TMG takes the “build it from the ground up” approach that I did, then the system policy I’ve been discussing may get missed. Hopefully this post will serve as a good lesson (or reminder for veteran firewall administrators).
UPDATE (10/26/2010)
Thomas K. H. Bittner (former MVP from Germany who runs the Windows Server System Reference Architecture blog) contacted me and shared a blog post he wrote on configuring DPM 2010 and TMG communication; the post can be found here: http://msmvps.com/blogs/wssra/archive/2010/10/20/configure-the-forefront-tmg-2010-to-allow-dpm-2010-communication.aspx. Thomas’ post is fantastic in that it is extremely granular in its configuration of communication channels between TMG and DPM. If you would prefer to lock things down more securely than I demonstrate, then I highly recommend checking out the post that Thomas wrote.
As always, SharePoint Saturday events are free and open to the public. If you have any interest in learning more about SharePoint, getting some free training, or simply networking and meeting other professionals in the SharePoint space, please sign up!
