Background

The BlobCacheFarmFlush solution itself has a handful of moving parts, and the element I’m going to focus on in this post is the administration page (BlobCacheFarmFlush.aspx) that gets added to the farm upon Feature activation.  In particular, I want to share some of the lessons I learned while figuring out how to get the page’s navigational (breadcrumb) support operating properly.

Unlike “standard” content pages that one might deploy through a SharePoint Feature or solution package, application pages (also called “layouts pages” because they go into the LAYOUTS folder within SharePoint’s 12 hive) don’t come with wired-up breadcrumb support.  An example of the type of breadcrumb to which I’m referring appears below (circled in red):

Unless additional steps are taken during the installation of your application pages (beyond simply placing them in the LAYOUTS folder), breadcrumbs like the one shown above will not appear.  It’s not that application pages (which derive from LayoutsBasePage or UnsecuredLayoutsBasePage) don’t include support for breadcrumbs – they do.  The reason breadcrumbs fail to show is because the newly added application pages themselves are not integrated into the sitemap files that describe the navigational hierarchy of the layouts pages.

Wiring Up Breadcrumb Support

Getting breadcrumbs to appear in your own application pages requires that you update the layouts sitemap files for each of the (IIS) sites serving up content on each of the SharePoint web front-end (WFE) servers in your farm.  The files to which I’m referring are named layouts.sitemap and appear in the _app_bin folder of each IIS site folder on the WFE.  An example of one such file (in its _app_bin folder) appears below.

I’m a “best practices” kind of guy, so when I was doing research for my BlobCacheFarmFlush solution, I was naturally interested in trying to make the required sitemap modifications in a way that was both easy and supported.  It didn’t take much searching on the topic before I came across Jan Tielens’ blog post titled “Adding Breadcrumb Navigation To SharePoint Application Pages, The Easy Way.”  In his blog post, Jan basically runs through the scenario I described above (though in much greater detail than I presented), and he mentions that another reader (Brian Staton) turned him onto a very simple and straightforward way of making the required sitemap modifications.  I’ll refer you to Jan’s blog post for the specifics, but the two-step quick summary goes like this:

1. Create a layouts.sitemap.*.xml file that contains your sitemap navigation additions and deploy it to the LAYOUTS folder within SharePoint’s 12 hive on a server.
2. Execute code that implements one of the two approaches shown below (typically on Feature activation) :

// Approach #1: Top-down starting at the SPFarm level
SPFarm.Local.Services.GetValue<SPWebService>().ApplyApplicationContentToLocalServer();

// Approach #2: Applying to the sites within an SPWebApplication
myWebApp.WebService.ApplyApplicationContentToLocalServer();

This isn’t much code, and it’s pretty clear that the magic rests with the ApplyApplicationContentToLocalServer method.  This method carries out a few operations, but the one in which we’re interested involves taking the new navigation nodes in the layouts.sitemap.*.xml file and integrating them into the layouts.sitemap file for each IIS site residing under a target SPWebService instance.  With the new nodes (which tie the new application pages into the navigational hierarchy) present within each layouts.sitemap file, breadcrumbs appear at the top of the new application pages when they are rendered.

I took this approach for a spin, and everything looked great!  My sitemap additions were integrated as expected, and my breadcrumb appeared on the BlobCacheFarmFlush.aspx page.  All was well .. until I actually deployed my solution to its first multi-server SharePoint environment.  That’s when I encountered my first problem.

Problem #1: The “Local” Part of the ApplyApplicationContentToLocalServer Method

When I installed and activated the BlobCacheFarmFlush solution in a multi-server environment, the breadcrumbs failed to appear on my application page.  It took a little legwork, but I discovered that the ApplyApplicationContentToLocalServer method has “Local” in its name for a reason: the changes made through the method’s actions only impact the server on which the method is invoked.

This contrasts with the behavior that SharePoint objects commonly exhibit.  The changes that are made through (and to) many SharePoint types impact data that is actually stored in SQL Server, and changes made through any farm member get persisted back to the appropriate database and become available through all servers within the farm.  The ApplyApplicationContentToLocalServer method, on the other hand, carries out its operations directly against the files and folders of the server on which the method is called, and the changes that are made do not “automagically” appear on or through other farm members.

The Central Administration host server for the farm in which I was activating my Feature wasn’t one of the WFEs serving up my application page.  When I activated my Feature from within Central Admin, my navigation additions were incorporated into the affected sites on the local (Central Admin) host … but the WFEs serving up actual site pages (and my application page) were not updated.  Result: no breadcrumb on my application page.

This issue is one of those problems that wouldn’t normally be discovered in a typical development environment.  Most of the SharePoint developers I know do their work within a virtual machine (VM) of some sort, so it’s not until one moves out of such an environment and into a multi-server environment that this type of deployment problem even makes itself known.  This issue only serves to underscore how important it is to test Features and solutions in a typical target deployment environment before releasing them for general use.

Putting my thinking cap back on, I worked to come up with another way to integrate the sitemap changes I needed in a way that was multi-server friendly.  The ApplyApplicationContentToLocalServer method still seemed like a winner given all that it did for a single line of code; perhaps all I needed to do was create and run a one-time custom timer job (that is, schedule a custom SPJobDefinition subclass) on each server within the farm and have that timer job execute the ApplyApplicationContentToLocalServer method locally.

I whipped-up a custom timer job to carry out this action and took it for a spin.  That’s when I ran into my second problem.

Problem #2: Rights Required for ApplyApplicationContentToLocalServer Method Invocation

The documentation for the ApplyApplicationContentToLocalServer method ends with this one line:

Only local administrators can call this method.

Prior to the creation of the custom timer job that I was going to use to update the sitemap files on each of the WFEs, I had basically ignored this point.  The local administrator requirement quickly became a barricade for my custom timer job, though.

Timer jobs, both SharePoint-supplied and custom, are executed within the context of the SharePoint Timer Service (OWSTIMER.EXE).  The Timer Service runs in an elevated security context with regard to the SharePoint farm, but its privileges shouldn’t extend beyond the workings of SharePoint.  Though some SharePoint administrators mistakenly believe that the Timer Service account (also known as the “database access account” or “farm service account”) requires local administrator rights on each server within the SharePoint farm, Microsoft spells out that this is neither required nor recommended.

The ApplyApplicationContentToLocalServer method works during Feature activation when the activating user is a member of the Local Administrators group on the server where activation is taking place – a common scenario.  The process breaks down, however, if the method call occurs within the context of the SharePoint Timer Service account because it isn’t (or shouldn’t be) a member of the Local Administrators group.  Attempts to call the ApplyApplicationContentToLocalServer method from within a timer job fail and result in an “Access Denied” message being written to the Application Event Log.  A quick look at the first section of code inside the method itself (using Reflector) makes this point pretty clearly:

if (!SPAdministrationServiceUtilities.IsCurrentUserMachineAdmin())
{
    throw new SecurityException(SPResource.GetString("AccessDenied", new object[0]));
}

This revelation told me that the ApplyApplicationContentToLocalServer method simply wasn’t going to cut the mustard for my purposes unless I wanted to either (a) require that the Timer Service account be added to the Local Administrators group on each server in the farm, or (b) require that an administrator manually execute an STSADM command or custom command line application to carry out the method call.  Neither of these were acceptable to me.

Method Deconstruction

Since I couldn’t use the ApplyApplicationContentToLocalServer method directly, I wanted to dissect it to the extent that I could in order to build my own process in a manner that replicated the method’s actions as closely as possible.  Performing the dissection (again via Reflector), I discovered that the method was basically iterating through each SPIisWebSite in each SPWebApplication within the SPWebService object being targeted.  As implied by its type name, each SPIisWebSite represents a web site within IIS – so each SPIisWebSite maps to a physical web site folder within the file system at C:\Inetpub\wwwroot\wss\VirtualDirectories (by default if IIS folders haven’t been redirected).

Once each of the web site folder paths is known, it isn’t hard to drill down a bit further to each layouts.sitemap file within the _app_bin folder for a given IIS web site.  With the fully qualified path to each layouts.sitemap file computed, it’s possible to carry out a programmatic XML merge with the new sitemap data from a layouts.sitemap.*.xml file that is deployed with a custom Feature or solution.  The ApplyApplicationContentToLocalServer method carries out such a merge through the private (and obfuscated) MergeAspSiteMapFiles method of the SPAspSiteMapFile internal type, but only after it has created a backup copy of the current layouts.sitemap file using the SPAspSiteMapFile.Copy method.

The Solution

With an understanding of the process that is carried out within the ApplyApplicationContentToLocalServer method, I proceeded to create my own class that effectively executed the same set of steps.  The result was the UpdateLayoutsSitemapTimerJob custom timer job definition that is part of my BlobCacheFarmFlush solution.  This class mimics the enumeration of SPWebApplication and SPIisWebSite objects, the backup of affected layouts.sitemap files, and the subsequent XML sitemap merge of the ApplyApplicationContentToLocalServer method.  The class is without external dependencies (beyond the SharePoint object model), and it is reusable in its current form.  Simply drop the class into a SharePoint project and call its DeployUpdateTimerJobs static method with the proper parameters – typically from the FeatureActivated method of a custom SPFeatureReceiver.  The class then takes care of provisioning a timer job instance that will update the layouts.sitemap navigational hierarchy for affected sites on each of the servers within the farm.

As an aside: while putting together the UpdateLayoutsSitemapTimerJob, there were times when I thought I had to be missing something.  On a handful of occasions, I found myself thinking, “Certainly there had to be a multi-server friendly version of the ApplyApplicationContentToLocalServer method.”  When I didn’t find one (after much searching), I had the good fortune of stumbling upon Vincent Rothwell’s “Configuring the breadcrumb for pages in _layouts” blog post.  Vincent’s post predates my own by a hefty two and a half years, but in it he describes a process that is very similar to the one I eventually ended up implementing in my custom timer job.  Seeing his post helped me realize I wasn’t losing my mind and that I was on the right track.  Thank you, Vincent.

Conclusion

I can sum up the contents of this post pretty simply: when developing application pages that entail sitemap updates, avoid using the ApplyApplicationContentToLocalServer method unless you’re (a) certain that your Feature will be installed into single server environments only, or (b) willing to direct those doing the installation and activation to carry out some follow-up administration on each WFE in the SharePoint farm.

Why does the ApplyApplicationContentToLocalServer method exist?  I did some thinking, and my guess is that it is leveraged primarily when service packs, hotfixes, and other additions are configured via the SharePoint Products and Technologies Configuration Wizard.  Anytime a SharePoint farm is updated with a patch or hotfix, the wizard is run on each server by a local administrator.

An examination of the LAYOUTS folder on one of my farm members provided some indirect support for this notion.  In my LAYOUTS folder, I found the layouts.sitemap.search.xml file, and it was dated 3/25/2008.  I believe (I’m not positive) that this file was deployed with the SharePoint Infrastructure Updates in the middle of 2008, and those updates introduced a number of new search admin pages for MOSS.  Since the contents of the layouts.sitemap.search.xml file include quite a few new search-related navigation nodes, my guess is that the ApplyApplicationContentToLocalServer method was leveraged to merge the navigation nodes for the new search pages when the configuration wizard was run.

In the meantime, if you happen to find a way to use this method in a multi-server deployment scenario that doesn’t involve the configuration wizard, I’d love to hear about it!  The caveat, of course, is that it has to be a best-practices approach – no security changes, no extra manual work/steps for farm administrators, etc.

Additional Reading and References

1. MSDN: Caching In Office SharePoint 2007
2. CodePlex: MOSS 2007 Farm-Wide BLOB Cache Flushing Solution
3. Jan Tielens: Adding Breadcrumb Navigation To SharePoint Application Pages, The Easy Way
4. MSDN: SPWebService.ApplyApplicationContentToLocalServer Method
5. TechNet: Plan for administrative and service accounts (Office SharePoint Server)
6. Red Gate Software: .NET Reflector
7. CodePlex: UpdateLayoutsSitemapTimerJob class
8. Vincent Rothwell: Configuring the breadcrumb for pages in _layouts