DPM, RPC, and DCOM with Forefront TMG 2010

In this post, I discuss a couple of DCOM/RPC snags I ran into while configuring Microsoft’s Data Protection Manager (DPM) 2007 client protection agent to run on my new Forefront Threat Management Gateway (TMG) 2010 server. I also walk through the troubleshooting approach I took to resolve the issues that appeared.

In a recent post, I was discussing my impending move to Microsoft’s Forefront Threat Management Gateway (TMG) 2010 on my home network.  As part of the move, I was going to decommission two Microsoft Internet Security and Acceleration (ISA) 2006 servers and an old Windows Server 2008 remote access services (RAS) box and replace them with a single TMG 2010 server – a big savings in terms of server maintenance and power consumption.

I completed the move about a week ago, and I’ve been very happy with TMG thus far.  TMG’s ISP redundancy and load balancing features have been fantastic, and they’ve allowed me to use my Internet connections much more effectively.

As a user of ISA since its original 2000 release, I also had no problem jumping in and working with the TMG management GUI.  It was all very familiar from the get-go.  Call me “very satisfied” thus far.

This afternoon, I took a few moments to un-join the old ISA servers from the domain, power them down, and clean things up.  I had also planned to take a little time integrating the new TMG box into my Data Protection Manager (DPM) 2007 backup rotation.  Unfortunately, though, the DPM integration took a bit longer than expected …

Backup Brigade

For those unfamiliar with the operation of DPM, I’ll take a couple of moments to explain a bit about how it works.  In order for DPM to do its thing, any computer that is going to be protected must have the DPM 2007 Protection Agent installed on it.  Once the DPM Protection Agent is installed and configured, the DPM server communicates through the agent (which operates as a Windows service) to leverage the protected system’s Volume Shadow Copy Service (VSS) for backups.

Installing the DPM agent typically isn’t a big challenge for common client computers, and it can be accomplished directly from within the DPM management GUI itself.  When the agent is installed through the GUI, DPM connects to the computer to be protected, installs the agent, and configures it to point back to the DPM server.  No manual intervention is required.

On some systems, though, it’s simply easier to install and configure the agent directly from the to-be-protected system itself.  A locked-down server (like a TMG box) falls into this category, so I manually put the agent installation package on the TMG server, ran it, and carried out the follow-up PowerShell Attach-ProductionServer (from the DPM Management Shell) on the DPM server.  The install proceeded without issue, and the associated attach (on the DPM server) went off without a hitch.  I thought I was good to go.

I fired up the management GUI on the DPM Server, went into the Agents tab under Management, and discovered that I couldn’t connect to the TMG server.

DPM Agent Issue

The Checklist

The fact that I couldn’t connect to the TMG server (SS-TMG1) from my DPM box was a bit of an eyebrow lifter, but it wasn’t entirely unexpected.  Communication between a DPM server and the DPM agent leverages DCOM, and I’d had to jump through a few hoops to ensure that the DPM server could communicate with the ISA boxes previously.

I suspected that an RPC/DCOM issue was in play, but I was having trouble seeing where the problem might be. So, I reviewed where I was at.

  • Without an exception, Windows Firewall will block communication between a DPM server and its agents.  I confirmed that Windows Firewall wasn’t in play and that TMG itself was handling all of the firewall action.
  • Examining TMG, I confirmed that I had a rule in place that permitted all traffic between my DPM server (SS-TOOLS1) and the TMG box itself.
    DPM-TMG Access Rule 
  • DPM-TMG Access Rule (with RPC)Strict RPC compliance is another potential problem with DPM on both ISA and TMG, as requiring strict compliance blocks any DCOM traffic.  DCOM (and any other traffic that doesn’t explicitly begin RPC exchanges by communicating with the RPC endpoint mapper on the target server) gets dropped by the RPC Filter unless the checkbox for Enforce strict RPC compliance is unchecked.  I confirmed that my rule wasn’t requiring strict compliance (as shown on the right).
  • I made sure that my DPM server wasn’t listed as a member of either the Enterprise Remote Management Computers or Remote Management Computers Computer Sets in TMG.  These two Computer Sets are specially impacted by a couple of TMG System Policy rules that can impact their ability to call into TMG via RPC and DCOM.
  • I reviewed all System Policy rules that might impact inbound RPC calls to the TMG server, and I couldn’t find any that would (or should) be influencing DPM’s ability to connect to its agent.

I also went out to the Forefront TMG Product Team’s Blog to see what advice they might have to offer, and I found this excellent article on RPC and TMG – well worth a read if you’re trying to troubleshoot RPC problems.  Unfortunately, it didn’t offer me any tips that would help in my situation.

Watching The Traffic

I may have simply had tunnel vision, but I was still obsessed with the notion that strict RPC checking was causing my problems.  To see if it was, I decided to fire-up TMG’s live logging and see what was happening when DPM tried to connect to its agent.  I set the logging to display only traffic that was originated from the DPM box, and this is what I saw.

Watching Traffic from DPM

There was nothing wrong that I could see.  My access rule was clearly being utilized (the one that doesn’t enforce strict RPC checking), and I wasn’t seeing any errors – just connection initiations and closes.  Traffic from DPM to TMG looked clean.

Taking A Step Back

I was frustrated, and I clearly needed to consider the possibility that I didn’t have a good read on the problem.  So, I went to the Windows Application event log to see if it might provide some insight.  I probably should have started with the event logs instead of TMG itself and firewall rules … but better late than never, I figured.

DPM Agent Can't CommunicatePopping open Event Viewer, I was greeted with the image you see on the left.  What I saw was enlightening, for I did have a problem with communication between the DPM agent and the DPM server.  The part that intrigued me, though, was the fact that the problem was with outbound communication (that is, from TMG server to DPM server) – not the other way around as I had originally suspected.  All of my focus had been on troubleshooting traffic coming into TMG because I’d been interpreting the errors I’d seen to mean that the DPM server couldn’t reach the agent – not that the agent couldn’t “phone home,” so to speak.

I knew for a fact that the DPM Server, SS-TOOLS1, didn’t have the Windows Firewall service running.  Since the service wasn’t running, there was no way that the agent’s attempts to communicate with DPM could (or rather, should) be getting blocked at the destination.  That left the finger of blame pointing at TMG.

On The Way Out

I decided to repeat the traffic watching exercise I’d conducted earlier, but instead of watching traffic coming into the TMG box from my DPM server, I elected to watch traffic going the other direction – from TMG to DPM.  Here’s what I saw:

Watching Traffic to DPM

The “a-ha” moment for me came when I saw the firewall rule that was actually governing RPC traffic to the DPM box from TMG.  It wasn’t the DPM All <=> SS-TMG1 rule I’d established — it was a system policy rule called [System] Allow RPC from Forefront TMG to trusted servers.

System policy rules are normally hidden in the firewall policy tab, so I had to explicitly show them to review them.  Once I did, there it was – rule 22.

System Policy Rule 22

Note that this rule applies to all traffic from the TMG server to the Internal network; I’ll be talking about that more in a bit.

System Policy EditorI couldn’t edit the rule in-place; I needed to use the System Policy editor.  So, I fired up the System Policy Editor and traced the rule back to its associated configuration group.  As it turned out, the rule was tied to the Active Directory configuration group under Authentication Services.

As the picture on the left clearly shows, the Enforce strict RPC compliance checkbox was checked.  Once I unchecked it and applied the configuration change, the DPM agent began communicating with the DPM server without issue.  Problem solved.

What Happened?

I was fairly sure that I hadn’t experienced this sort of trouble installing the DPM Protection Agent under ISA Server 2006, so I tried to figure out what might have happened.

I hadn’t recalled having to adjust the target system policy under ISA when installing the DPM agent originally, but a quick boot and check of my old ISA server revealed that the checkbox was indeed unchecked (meaning that strict RPC compliance wasn’t being enforced).  I’d apparently made the change at some point and forgotten about it.  I suspect I’d messed with it at some point in the distant past while working on passing AD information through ISA, getting VPN functionality up-and-running, or perhaps something else.

Implications

Bottom line: TMG enforces strict compliance for RPC traffic that originates on the TMG server (Local Host) and is destined for the Internal network.  Since System Policy Rules are applied before administrator-defined Firewall Policy Rules, RPC traffic from the TMG server to the Internal network will always be governed by the system policy unless that policy is disabled.

In this particular scenario, the DPM 2007 Protection Agent’s operation was impacted.  Even though I’d created a rule that I thought would govern interactions between DPM and TMG, the reality is that it only governed RPC traffic coming into TMG – not traffic going out.

In reality, any service or application that sends DCOM traffic originating on the TMG server to the Internal network is going to be affected by the Allow RPC from Forefront TMG to trusted servers rule unless the associated system policy is adjusted.

Conclusion

The core findings of this post have been documented by others (in a variety of forms/scenarios) for ISA, but this go-round with TMG and the DPM association caught me off-guard such that I thought it would be worth sharing my experience with other firewall administrators.  If anyone else moving to TMG takes the “build it from the ground up” approach that I did, then the system policy I’ve been discussing may get missed.  Hopefully this post will serve as a good lesson (or reminder for veteran firewall administrators).

UPDATE (10/26/2010)

Thomas K. H. Bittner (former MVP from Germany who runs the Windows Server System Reference Architecture blog) contacted me and shared a blog post he wrote on configuring DPM 2010 and TMG communication; the post can be found here: http://msmvps.com/blogs/wssra/archive/2010/10/20/configure-the-forefront-tmg-2010-to-allow-dpm-2010-communication.aspx.  Thomas’ post is fantastic in that it is extremely granular in its configuration of communication channels between TMG and DPM.  If you would prefer to lock things down more securely than I demonstrate, then I highly recommend checking out the post that Thomas wrote.

Additional Reading and References

  1. Recent Post: Portrait of a Basement Datacenter
  2. Microsoft: Forefront Threat Management Gateway 2010
  3. Microsoft: Internet Security and Acceleration Server 2006
  4. Microsoft: System Center Data Protection Manager
  5. Forefront TMG Product Team Blog: RPC Filter and “Enable strict RPC compliance”

Author: Sean McDonough

I am the Chief Technology Officer for PAIT Group, a Cincinnati-based Microsoft partner providing SharePoint solutions and consulting. I have been developing software, building infrastructures, and solving technical problems for over two decades. I have acquired a diverse and expansive skill set as consultant, product manager, solution architect, team lead, administrator, and platform evangelist through the years. I am an author, blogger, and frequent speaker/instructor at technical conferences, user groups, and similar events all over North America. I am also a Microsoft MVP in both the Office Development and the Office Servers and Services categories. I can be reached through my blog (http://SharePointInterface.com) or Twitter (@spmcdonough).

26 thoughts on “DPM, RPC, and DCOM with Forefront TMG 2010”

  1. Fantastic article – thanks!
    I have been trying to solve this problem with DPM 2007 (and 2010RC) and EBS 2008 Security Server (with TMG) for weeks. Hugely appreciated!
    :-)

  2. You’re very welcome, Pat; glad the article was of assistance!

    I’m looking forward to moving to DPM 2010 once it actually goes RTM. I’m not a big fan of all the babysitting DPM 2007 seems to require, and I’ve been told that 2010 is quite a bit more “self-sufficient.” I hope that’s the case.

    Thanks for taking the time to share your thoughts.

  3. I’m having an issue with this. I’ve done what you’ve said in your blog however I can’t attached the Client.

    Attach protected computer failed:
    Error 31144: The server could not be found in Active Directory.
    Recommended action: Ping this server to verify that it is accessible and then retry the operation. If the server is accessible, verify that the DPM service is running and that the current user has a domain account and is a member of the local administrators group. If the problem persists, contact your DPM administrator.

    In ISA the only denied connection I see is

    Denied Connection 10/8/2010 3:47:27 PM
    Log type: Firewall service
    Status: A non-SYN packet was dropped because it was sent by a source that does not have an established connection with the Forefront TMG computer.
    Rule: None – see Result Code
    Source: Internal (x.x.x.x:56803)
    Destination: Local Host (x.x.x.x:445)
    Protocol: Microsoft CIFS (TCP)

    Result Code is 0xc0040017 FWX_E_TCP_NOT_SYN_PACKET_DROPPED

    Any ideas?

  4. Matt,

    I’ll start by asking the obvious question: is your TMG box a member of the domain that your DPM server is joined to? If not, then you may have the source of your problem. If the DPM and TMG servers are in different domains, you’ll need a trust relationship between the domains.

    If the TMG box isn’t a domain member at all, you should be able be able to attach to the server (in workgroup mode) based on this article (http://www.scdpmonline.org/111protecting-a-workgroup-computer-with-dpm-2010-easy-.aspx). I have not tried this procedure, though, so I can’t confirm that it works as advertised.

    Double-check and make sure that you’re not enforcing strict RPC compliance. Since the dropped packet is a CIFS packet (and not an RPC packet), though, I doubt RPC compliance is the issue.

    If you’ve tried all of that, then make sure that you’ve got all the necessary ports open to your domain controller(s) for authentication and intra-domain communications. ISAServer.org has a good overview of this at http://www.isaserver.org/tutorials/Configuring-Domain-Members-Back-to-Back-ISA-Firewall-DMZ-Part2.html. Specifically look at the rules that are established for intra-domain communication between domain controllers (DCs) and DMZ servers. Though the article is for ISA server, the ports are relevant for TMG, as well.

    Assuming it’s a straight shot from your TMG box to your DCs, the aforementioned rules shouldn’t be an issue. If you happen to have back-to-back TMG servers, though, and the “problem server” is the front-edge server, you’ll definitely need to make sure that the back-end TMG/ISA box is passing all the required AD traffic between the DCs and the front-edge server.

    Anything of interest in your event logs? DPM server and client logs?

    Hopefully some of this is stuff that may at least get you to the next step :-)

    – Sean

  5. Sean,

    Been working on this problem all day until I found your blog post. That solved my problem. Many, many thanks for the writeup!

  6. This also came in handy to fix another DCOM issue that had nothing to do with DPM. We are trying to monitor the health of the TMG server using PRTG and were having errors. This seems to have fixed it.

    Thanks!

  7. You’re welcome, Alex, and thanks for sharing that bit of information. DCOM and the RPC Filter tend to clash more often than not, and I’m glad to hear that the post provided some food for thought in resolving your particular scenario.

  8. Thank you very much, your expertise saved my day.
    I had exactly the same problem and exactly your fix took care of it.
    Smart people are important no matter what others say :)

  9. I appreciate the comment, Alberto. When I said “I was fairly sure …,” I actually meant that I didn’t think I’d had to disable strict RPC compliance enforcement on ISA 2006. My memory proved wrong, though, as I indicated a few lines later. :-)

    Thanks for pointing out the manual agent install link!

  10. Thank you, Sean, it’s very interesting and helpful article. It helps me solve the problem of accessing to DPM agent on notebooks in the another network through the VPN site-to-site connection.

  11. Thanks a lot, i too had got the RPC Server is unavailable error and found that changing the system policy did the trick. You adding the screenshots is very helpful, however i couldn’t follow where in the event viewer did you see the logs where you got the ‘aha’ moment.

  12. krazieneurons,

    The revelation for me wasn’t a specific entry in the TMG application event log, but rather the presence of entries specifying that the TMG box couldn’t properly communicate with the DPM server (i.e., outbound traffic from TMG => DPM). Previously, I had only been watching the TMG firewall logs for traffic coming *into* the TMG box. After seeing the application event log entries, I shifted to watching the *outbound* traffic on the TMG box … and that’s when I saw the RPC-related issues.

    I hope that explains things a bit better.

  13. Hi Sean,

    I am having the same issue on one of my file servers. I checked the event logs on the file server but could not find any DPMRA related entries. However, I did find this in Application Logs:
    Event Type: Information
    Event Source: MsiInstaller
    Event Category: None
    Event ID: 11724
    Date: 6/28/2012
    Time: 4:51:15 PM
    User: CORPORATEROOT\adm_905393
    Computer: NMGBS107
    Description:
    Product: Forefront TMG Client — Removal completed successfully.

    Data:
    0000: 7b 31 39 39 42 37 46 37 {199B7F7
    0008: 38 2d 36 39 42 37 2d 34 8-69B7-4
    0010: 37 43 35 2d 38 44 34 42 7C5-8D4B
    0018: 2d 41 33 45 44 31 33 39 -A3ED139
    0020: 31 46 42 36 42 7d 1FB6B}

    I also found this error in System Logs :
    Event Type: Error
    Event Source: DCOM
    Event Category: None
    Event ID: 10010
    Date: 6/26/2012
    Time: 10:06:21 AM
    User: N/A
    Computer: NMGBS107
    Description:
    The server {DA6AA17A-D61C-4E9C-8CEA-DB25DEA52A95} did not register with DCOM within the required timeout.

    The server was backed up last Thursday latest and has stopped since. I verified that the file server is a member of the following groups on DPM server : MSDPMTrustedMachines, DPMRADmTrustedMachines and Distributed COM Users. I also verified that DPM server is a member of the following on the File Server : Distributed COM Users, DPMRADCOMTrustedMachines and DPMRADmTrustedMachines. Ping is working fine both ways. Is there anything alse I should be checking? Please help.

    Regards,
    Pavit

    1. Pavit: admittedly, it’s been a long time since I’ve had to troubleshoot this. If backups had been working without issue and then stopped suddenly, I’d be looking very hard at what changed in your environment at the point of stoppage. The event you cited (for application removal) – did you have the TMG client installed on the TMG server itself? If so, try putting it back on to see if changes things. Personally, though, I’ve not seen the client run directly on a TMG server itself … and I’m inclined to believe that this wouldn’t be something you’d want to do given how the client wires itself into WinSock …

      http://www.isaserver.org/tutorials/forefront-threat-management-gateway-tmg-2010-firewall-client-features-benefits.html

      Ignoring the TMG Client for a second, I’d go beyond the simple ping tests that you’re doing to actually look at traffic between the DPM server and the TMG server. Is TMG reporting that communications are blocked, going out? Coming in? You should have the tools you need on TMG itself to monitor conversations between servers, and those conversations should shed some light on the current issue.

  14. Thanks. Struggling for 1 day to promote a windows 2012 dc attached with site to site vpn to the domain. I knew it was something in VPN tunnel. Disabling strict RPC compliance did the trick.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s