In this post, I share a PowerShell script that I recently wrote to help out a friend. The script allows you to search and identify content items in SharePoint site collections by object ID (GUID). The script isn’t something you’d probably use every day, but it might be handy to keep in the script library “just in case.”
Here’s another blog post to file in the “I’ll probably never need it, but you never know” bucket of things you’ve seen from me.
Admittedly, I don’t get to spend as much time as I’d like playing with PowerShell and assembling scripts. So when the opportunity to whip-up a “quick hit” script comes along, I usually jump at it.
The Situation
I feel very fortunate to have made so many friends in the SharePoint community over the last several years. One friend who has been with me since the beginning (i.e., since my first presentation at the original SharePoint Saturday Ozarks) is Kirk Talbot. Kirk has become something of a “regular” on the SharePoint Saturday circuit, and many of you may have seen him at a SharePoint Saturday event someplace in the continental United States. To tell you the truth, I’ve seen Kirk as far north as Michigan and as far south as New Orleans. Yes, he really gets around.
Kirk and I keep up fairly regular correspondence, and he recently found himself in a situation where he needed to determine which objects (in a SharePoint site collection) were associated with a handful of GUIDs. Put a different way: Kirk had a GUID (for example, 89b66b71-afc8-463f-b5ed-9770168996a6) and wanted to know – was it a web? A list? A list item? And what was the identity of the item?
PowerShell to the Rescue
I pointed Kirk to a script I had previously written (in my Finding Duplicate GUIDs in Your SharePoint Site Collection post) and indicated that it could probably be adapted for his purpose. Kirk was up to the challenge, but like so many other SharePoint administrators was short on time.
I happened to find myself with a bit of free time in the last week and was due to run into Kirk at SharePoint Saturday Louisville last weekend, so I figured “what the heck?” I took a crack at modifying the script I had written earlier so that it might address Kirk’s need. By the time I was done, I had basically thrown out my original script and started over. So much for following my own advice.
The Script
The PowerShell script that follows is relatively straightforward in its operation. You supply it with a site collection URL and a target object GUID. The script then searches through the webs, lists/libraries, and list items of the site collection for an object with an ID that matches the GUID specified. If it finds a match, it reports some information about the matching object.
A sample run of the script appears below. In the case of this example, a list item match was found in the target site collection for the supplied GUID.
This script leverages the SharePoint object model directly, so it can be used with either SharePoint 2007 or SharePoint 2010. Its search algorithm is relatively efficient, as well, so match results should be obtained in seconds to maybe minutes – not hours.
[sourcecode language=”powershell”]
<#
.SYNOPSIS
FindObjectByGuid.ps1
.DESCRIPTION
This script attempts to locate a SharePoint object by its unique ID (GUID) within
a site collection. The script first attempts to locate a match by examining webs;
following webs, lists/libraries are examined. Finally, individual items within
lists and libraries are examined. If an object with the ID is found, information
about the object is reported back.
.NOTES
Author: Sean McDonough
Last Revision: 27-July-2012
.PARAMETER SiteUrl
The URL of the site collection that will be searched
.PARAMETER ObjectGuid
The GUID that identifies the object to be located
.EXAMPLE
FindObjectByGuid -SiteUrl http://mysitecollection.com -ObjectGuid 91ce5bbf-eebb-4988-9964-79905576969c
#>
param
(
[string]$SiteUrl = "$(Read-Host ‘The URL of the site collection to search [e.g. http://mysitecollection.com]’)",
[Guid]$ObjectGuid = "$(Read-Host ‘The GUID of the object you are trying to find [e.g. 91ce5bbf-eebb-4988-9964-79905576969c]’)"
)
function FindObject($startingUrl, $targetGuid)
{
# To work with SP2007, we need to go directly against the object model
Add-Type -AssemblyName "Microsoft.SharePoint, Version=12.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c"
# Grab the site collection and all webs associated with it to start
$targetSite = New-Object Microsoft.SharePoint.SPSite($startingUrl)
$matchObject = $false
$itemsTotal = 0
$listsTotal = 0
$searchStart = Get-Date
Clear-Host
Write-Host ("INITIATING SEARCH FOR GUID: {0}" -f $targetGuid)
# Step 1: see if we can find a matching web.
$allWebs = $targetSite.AllWebs
Write-Host ("`nPhase 1: Examining all webs ({0} total)" -f $allWebs.Count)
foreach ($spWeb in $allWebs)
{
$listsTotal += $spWeb.Lists.Count
if ($spWeb.ID -eq $targetGuid)
{
Write-Host "`nMATCH FOUND: Web"
Write-Host ("- Web Title: {0}" -f $spWeb.Title)
Write-Host ("- Web URL: {0}" -f $spWeb.Url)
$matchObject = $true
break
}
$spWeb.Dispose()
}
# If we don’t yet have match, we’ll continue with list iteration
if ($matchObject -eq $false)
{
Write-Host ("Phase 2: Examining all lists and libraries ({0} total)" -f $listsTotal)
$allWebs = $targetSite.AllWebs
foreach ($spWeb in $allWebs)
{
$allLists = $spWeb.Lists
foreach ($spList in $allLists)
{
$itemsTotal += $spList.Items.Count
if ($spList.ID -eq $targetGuid)
{
Write-Host "`nMATCH FOUND: List/Library"
Write-Host ("- List Title: {0}" -f $spList.Title)
Write-Host ("- List Default View URL: {0}" -f $spList.DefaultViewUrl)
Write-Host ("- Parent Web Title: {0}" -f $spWeb.Title)
Write-Host ("- Parent Web URL: {0}" -f $spWeb.Url)
$matchObject = $true
break
}
}
if ($matchObject -eq $true)
{
break
}
}
$spWeb.Dispose()
}
# No match yet? Look at list items (which includes folders)
if ($matchObject -eq $false)
{
Write-Host ("Phase 3: Examining all list and library items ({0} total)" -f $itemsTotal)
$allWebs = $targetSite.AllWebs
foreach ($spWeb in $allWebs)
{
$allLists = $spWeb.Lists
foreach ($spList in $allLists)
{
try
{
$listItem = $spList.GetItemByUniqueId($targetGuid)
}
catch
{
$listItem = $null
}
if ($listItem -ne $null)
{
Write-Host "`nMATCH FOUND: List/Library Item"
Write-Host ("- Item Name: {0}" -f $listItem.Name)
Write-Host ("- Item Type: {0}" -f $listItem.FileSystemObjectType)
Write-Host ("- Site-Relative Item URL: {0}" -f $listItem.Url)
Write-Host ("- Parent List Title: {0}" -f $spList.Title)
Write-Host ("- Parent List Default View URL: {0}" -f $spList.DefaultViewUrl)
Write-Host ("- Parent Web Title: {0}" -f $spWeb.Title)
Write-Host ("- Parent Web URL: {0}" -f $spWeb.Url)
$matchObject = $true
break
}
}
if ($matchObject -eq $true)
{
break
}
}
$spWeb.Dispose()
}
# No match yet? Too bad; we’re done.
if ($matchObject -eq $false)
{
Write-Host ("`nNO MATCH FOUND FOR GUID: {0}" -f $targetGuid)
}
# Dispose of the site collection
$targetSite.Dispose()
Write-Host ("`nTotal seconds to execute search: {0}`n" -f ((Get-Date) – $searchStart).TotalSeconds)
# Abort script processing in the event an exception occurs.
trap
{
Write-Warning "`n*** Script execution aborting. See below for problem encountered during execution. ***"
$_.Message
break
}
}
Again, I don’t envision this being something that everyone needs. I want to share it anyway. One thing I learned with the “duplicate GUID” script I referenced earlier is that I generally underestimate the number of people who might find something like this useful.
Have fun with it, and please feel free to share your feedback!
Recent failures with Microsoft Office Picture Manager and SharePoint Explorer View led me to dive under-the-hood to better understand how SharePoint 2007’s WebDAV and IIS7’s WebDAV Publishing role service interact. This post summarizes my findings, as well as how I eliminated my 405 errors.
Several months ago, I decided that a rebuild of my primary MOSS environment here at home was in order. My farm consisted of a couple of Windows Server 2003 R2 VMs (one WFE, one app server) that were backed by a non-virtualized SQL Server. I wanted to free up some cycles on my Hyper-V boxes, and I had an “open physical box” … so, I elected to rebuild my farm on a single, non-virtualized box running (the then newly released) Windows Server 2008 R2.
The rebuild went relatively smoothly, and bringing my content databases over from the old farm posed no particular problems. Everything was good.
The Situation
Fast forward to just a few weeks ago.
One of the site collections in my farm is used to store and share pictures that we take of our kids. The site collection is, in effect, a huge multimedia repository …
… and allow me a moment to address the concerns of the savvy architects and administrators out there. I do understand SharePoint BLOB (binary large object) storage and the implications (and potential effects) that large multimedia libraries can have on scalability. I wouldn’t recommend what I’m doing to most clients – at least not until remote BLOB storage (RBS) gets here with SharePoint 2010. Remember, though, that my wife and I are just two people – not a company of hundreds or thousands. The benefits of centralized, tagged, searchable, nicely presented content outweigh scalability and performance concerns for us.
Back to the pictures site. I was getting set to upload a batch of pictures, so I did what I always do: I went into the Upload menu of the target pictures library in the site collection and selected Upload Multiple Pictures as shown on the right. For those who happen to have Microsoft Office 2007 installed (as I do), this action normally results in the Microsoft Office Picture Manager getting launched as shown below.
From within the Microsoft Office Picture Manager, uploading pictures is simply a matter of navigating to the folder containing the pictures, selecting the ones that are to be pushed into SharePoint, and pressing the Upload and Close button. From there, the application itself takes care of rounding up the pictures that have been selected and getting them into the picture library within SharePoint. SharePoint pops up a page that provides a handy “Go back to …” link that can then be used to navigate back to the library for viewing and working with the newly uploaded pictures.
Upon selecting the Upload Multiple Pictures menu item, SharePoint navigated to the infopage.aspx page shown above. I waited, and waited … but the Microsoft Office Picture Manager never launched. I hit my browser’s back button, and tried the operation again. Same result: no Picture Manager.
Trouble In River City
Picture Manager’s failure to launch was obviously a concern, and I wanted to know why I was encountering problems … but more than anything, I simply wanted to get my pictures uploaded and tagged. My wife had been snapping pictures of our kids left and right, and I had 131 JPEG files waiting for me to do something.
I figured that there was more than one way to skin a cat, so I initiated my backup plan: Explorer View. If you aren’t familiar with SharePoint’s Explorer View, then you need not look any further than the name to understand what it is and how it operates. By opening the Actions menu of a library (such as a Document Library or Picture Library) and selecting the Open with Windows Explorer menu item as shown on the right, a Windows Explorer window is opened to the library. The contents of the library can then be examined and manipulated using a file system paradigm – even though SharePoint libraries are not based in (or housed in) any physical file system.
The mechanisms through which the Explorer View are prepared, delivered, and rendered are really quite impressive from a technical perspective. I’m not going to go into the details, but if you want to learn more about them, I highly recommend a whitepaper that was authored by Steve Sheppard. Steve is an escalation engineer with Microsoft who I’ve worked with in the past, and his knowledge and attention to detail are second to none – and those qualities really come through in the whitepaper.
Unfortunately for me, though, my attempts to open the picture library in Explorer View also led nowhere. Simply put, nothing happened. I tried the Open with Windows Explorer option several times, and I was greeted with no action, error, or visible sign that anything was going on.
SharePoint and WebDAV
I was 0 for 2 on my attempts to get at the picture library for uploading. I wasn’t sure what was going on, but I was pretty sure that WebDAV (Web Distributed Authoring and Versioning) was mixed-up in the behavior I was seeing. WebDAV is implemented by SharePoint and typically employed to provide the Explorer View operations it supports. I was under the impression that the Microsoft Office Picture Manager leveraged WebDAV to provide some or all of its upload capabilities, too.
After a few moments of consideration, the notion that WebDAV might be involved wasn’t a tremendous mental leap. In rebuilding my farm on Windows Server 2008 R2, I had moved from Internet Information Services (IIS) version 6 (in Windows Server 2003 R2) to IIS7. WebDAV is different in IIS7 versus previous versions … I just hadn’t heard about SharePoint WebDAV-based functions operating any differently.
Playing a Client-Side Tune
My gut instincts regarding WebDAV hardly qualified as “objective troubleshooting information,” so I fired-up Fiddler2 to get a look at what was happening between my web browser and the rebuilt SharePoint farm. When I attempted to execute an Open with Windows Explorer against the picture library, I was greeted with a bunch of HTTP 405 errors.
405 ?!?!
To be completely honest, I’d never actually seen an HTTP 405 status code before. It was obviously an error (since it was in the 400-series), but beyond that, I wasn’t sure. A couple of minutes of digging through the W3C’s status code definitions, though, revealed that a 405 status code is returned whenever a requested method or verb isn’t supported.
I dug a little deeper and compared the request headers my browser had sent with the response headers I’d received from SharePoint. Doing that spelled-out the problem pretty clearly.
Here’s an example of one of the HTTP headers that was sent:
PROPFIND was the method that my browser was passing to SharePoint, and the request was failing because the server didn’t include the PROPFIND verb in its list of supported methods as stated in the Allow: portion of the response. PROPFIND was further evidence that WebDAV was in the mix, too, given its limited usage scenarios (and since the bulk of browser web requests employ either the GET or POST verb).
So what was going on? The operations I was attempting had worked without issue under II6 and Windows Server 2003 R2, and I was pretty confident that I hadn’t seen any issues with other Windows Server 2008 (R2 and non-R2) farms running IIS7. I’d either botched something on my farm rebuild or run into an esoteric problem of some sort; experience (and common sense) pointed to the former.
Doing Some Legwork
I turned to the Internet to see if anyone else had encountered HTTP 405 errors with SharePoint and WebDAV. Though I quickly found a number of posts, questions, and other information that seemed related to my situation, none of it really described my particular scenario or what I was seeing.
After some additional searching, I eventually came across a discussion on the MSDN forums that revolved around whether or not WebDAV should be enabled within IIS for servers that serve-up SharePoint content. The back and forth was a bit disjointed, but my relevant take-away was that enabling WebDAV within IIS7 seemed to cause problems for SharePoint.
I decided to have a look at the server housing my rebuilt farm to see if I had enabled the WebDAV Publishing role service. I didn’t think I had, but I needed to check. I opened up the Server Manager applet and had a look at Role Services that were enabled for the Web Server (IIS). The results are shown in the image on right; apparently, I had enabled WebDAV Publishing. My guess is that I did it because I thought it would be a good idea, but it was starting to look like a pretty bad idea all around.
The Test
I was tempted to simply remove the WebDAV Publishing role service and cross my fingers, but instead of messing with my live “production” farm, I decided to play it safe and study the effects of enabling and disabling WebDAV Publishing in a controlled environment. I fired up a VM that more-or-less matched my production box (Windows Server 2008 R2, 64-bit, same Windows and SharePoint patch levels) to play around.
When I fired-up the VM, a quick check of the enabled role services for IIS showed that WebDAV Publishing was not enabled – further proof that I got a bit overzealous in enabling role services on my rebuilt farm. I quickly went into the VM’s SharePoint Central Administration site and created a new web application (http://spsdev:18480). Within the web application, I created a team site called Sample Team Site. Within that team site, I then created a picture library called Sample Picture Library for testing.
When It Works (without the WebDAV Publishing Role Service)
I fired up Fiddler2 in the VM, opened Internet Explorer 8, navigated to the Sample Picture Library, and attempted to execute an Open with Windows Explorer operation. Windows Explorer opened right up, so I knew that things were working as they should within the VM. The pertinent capture for the exchange between Internet Explorer and SharePoint (from Fiddler2) appears below.
Reviewing the dialog between client and server, there appeared to be two distinct “stages” in this sequence. The first stage was an HTTP request that was made to the root of the site collection using the OPTIONS method, and the entire HTTP request looked like this:
In response to the request, the SharePoint server passed back an HTTP 200 status that looked similar to the block that appears below. Note the permitted methods/verbs (as Allow:) that the server said it would accept, and that the PROPFIND verb appeared within the list:
After this initial request and associated response, all subsequent requests (“stage 2”) were made using the PROPFIND verb and have a structure that was similar to the following:
Each of the requests returned a 207 HTTP status (WebDAV multi-status response) and some WebDAV data within an XML document (slightly modified for readability).
It was these PROPFIND requests (or rather, the 207 responses to the PROPFIND requests) that gave the client-side WebClient (directed by Internet Explorer) the information it needed to determine what was in the picture library, operations that were supported by the library, etc.
When It Doesn’t Work (i.e., WebDAV Publishing Enabled)
When the WebDAV Publishing role service was enabled within IIS7, the very same request (to open the picture library in Explorer View) yielded a very different series of exchanges (again, captured within Fiddler2):
The initial OPTIONS request returned an HTTP 200 status that was identical to the one previously shown, and it even included the PROPFIND verb amongst its list of accepted methods:
Even though the PROPFIND verb was supposedly permitted, though, subsequent requests resulted in an HTTP 405 status and failure:
[sourcecode language=”text”]
HTTP/1.1 405 Method Not Allowed
Allow: GET, HEAD, OPTIONS, TRACE
Server: Microsoft-IIS/7.5
Persistent-Auth: true
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 12.0.0.6510
Date: Mon, 21 Dec 2009 22:04:31 GMT
Content-Length: 0
[/sourcecode]
Unfortunately, these behind-the-scenes failures didn’t seem to generate any noticeable error or message in client browsers. While testing (locally) in the VM environment, I was at least prompted to authenticate and eventually shown a form of “unsupported” error message. While connecting (remotely) to my production environment, though, the failure was silent. Only Fiddler2 told me what was really occurring.
The Solution
The solution to this issue, it seems, is to ensure that the WebDAV Publishing role service is not installed on WFEs serving up SharePoint content in Windows Server 2008 / IIS7 environments. The mechanism by which SharePoint 2007 handles WebDAV requests is still something of a mystery to me, but it doesn’t appear to involve the IIS7-based WebDAV Publishing role service at all.
Steve Sheppard’s troubleshooting whitepaper (introduced earlier) mentions that enabling or disabling the WebDAV functionality supplied by IIS6 (under Windows Server 2003) has no appreciable effect on SharePoint operation. Steve even mentions that SharePoint’s internal WebDAV implementation is provided by an ISAPI filter that is housed in Stsfilt.dll. Though this was true in WSSv2 and SharePoint Portal Server 2003 (the platforms addressed by Steve’s whitepaper), it’s no longer the case with SharePoint 2007 (WSSv3 and MOSS 2007). The OPTIONS and PROPFIND verbs are mapped to the Microsoft.SharePoint.ApplicationRuntime.SPHttpHandler type in SharePoint web.config files (see below) – Stsfilt.dll library doesn’t even appear anywhere within the file system of MOSS servers (or at least in mine).
Regardless of how it is implemented, the fact that the two verbs of interest (OPTIONS and PROPFIND) are mapped to a SharePoint type indicates that WebDAV functionality is still handled privately within SharePoint for its own purposes. When the WebDAV Publishing role is enabled in IIS7, IIS7 takes over (or at least takes precedence for) PROPFIND requests … and that’s where things appear to break.
To Sum Up
After toggling the WebDAV Publishing role service on and off a couple of times in my VM, I became convinced that my production environment would start behaving the way I wanted it to if I simply disabled IIS7’s WebDAV Publishing functionality. I uninstalled the WebDAV Publishing role service, and both Microsoft Office Picture Manager and Explorer View started behaving again.
I also made a note to myself to avoid installing role services I thought I might need before I actually needed them :-)
RPO (recovery point objective) targets and RTO (recovery time objective) targets are critical to have in hand prior to the start of disaster recovery (DR) planning for SharePoint. This post discusses RPO and RTO to build an understanding of what they are and how they impact DR decision making.
Years ago, before I began working with SharePoint, I spent some time working as an application architect for a Fortune 500 financial services company based here in Cincinnati, Ohio. While at the company, I was awarded the opportunity to serve as a disaster recovery (DR) architect on the team that would build (from ground up) the company’s first DR site implementation. It was a high-profile role with little definition – the kind that can either boost a career or burn it down. Luckily for me, the outcome leaned towards the former.
Admittedly, though, I knew very little about DR before starting in that position. I only knew what management had tasked me with doing: ensuring that mission-critical applications would be available and functional at the future DR site in the event of a disaster. If you aren’t overly familiar with DR, then that target probably sounds relatively straightforward. As I began working and researching avenues of attack for my problem set, though, I quickly realized how challenging and unusual disaster recovery planning was as a discipline – particularly for a “technically minded” person like myself.
Understanding the “Technical Tendency”
When it comes to DR, folks with whom I’ve worked have heard me say the following more than a few times:
It is the nature of technical people to find and implement technical solutions to technical problems. At its core, disaster recovery is not a technical problem; it is a business problem. Approaching disaster recovery with a purely technical mindset will result in a failure to deliver an appropriate solution more often than not.
What do I mean by that? Well, technical personnel tend to lump DR plans and activities into categories like “buying servers,” “taking backups,” and “acquiring off-site space.” These activities can certainly be (and generally are) part of a DR plan, but if they are the starting point for a DR strategy, then problems are likely to arise.
Let me explain by way of a simplistic and fictitious example.
Planning for SharePoint DR in a Vacuum
Consider the plight of Larry. Larry is an IT professional who possesses administrative responsibility for his company’s SharePoint-based intranet. One day, Larry is approached by his manager and instructed to come up with a DR strategy for the SharePoint farm that houses the intranet. Like most SharePoint administrators, Larry’s never really “done” DR before. He’s certain that he will need to review his backup strategy and make sure that he’s getting good backups. He’ll probably need to talk with the database administrators, too, because it’s generally a good idea to make sure that SQL backups are being taken in addition to SharePoint farm (catastrophic) backups.
Larry’s been told that off-site space is already being arranged by the facilities group, so that’s something he’ll be able to take off of his plate. He figures he’ll need to order new servers, though. Since the company’s intranet farm consists of four servers (including database servers), he plans to play it safe and order four new servers for the DR site. In his estimation, he’ll probably need to talk with the server team about the hardware they’ll be placing out at the DR site, he’ll need to speak with the networking team about DNS and switching capabilities they plan to include, etc.
Larry prepares his to-do list, dives in, and emerges three months later with an intranet farm DR approach possessing the following characteristics:
The off-site DR location will include four servers that are setup and configured as a new, “warm standby” SharePoint farm.
Every Sunday night, a full catastrophic backup of the SharePoint farm will be taken; every other night of the week, a differential backup will be taken. After each nightly backup is complete, it will be remotely copied to the off-site DR location.
In the event of a disaster, Larry will restore the latest full backup and appropriate differential backups to the standby farm that is running at the DR site.
Once the backups have been restored, all content will be available for users to access – hypothetically speaking, of course.
There are a multitude of technical questions that aren’t answered in the plan described above. For example, how is patching of the standby farm handled? Is the DR site network a clone of the existing network? Will server name and DNS hostname differences be an issue? What about custom solution packages (WSPs)? Ignoring all the technical questions for a moment, take a step back and ask yourself the question of greatest importance: will Larry’s overall strategy and plan meet his DR requirements?
If you’re new to DR, you might say “yes” or “no” based on how you view your own SharePoint farm and your experiences with it. If you’ve previously been involved in DR planning and are being honest, though, you know that you can’t actually answer the question. Neither can Larry or his manager. In fact, no one (on the technical side, anyway) has any idea if the DR strategy is a good one or not – and that’s exactly the point I’m trying to drive home.
The Cart Before the Horse
Assuming Larry’s company is like many others, the SharePoint intranet has a set of business owners and stakeholders (collectively referred to as “The Business” hereafter) who represent those who use the intranet for some or all of their business activities. Ultimately, The Business would issue one of three verdicts upon learning of Larry’s DR strategy:
Verdict 1: Exactly What’s Needed
Let’s be honest: Larry’s DR plan for intranet recovery could be on-the-money. Given all of the variables in DR planning and the assumptions that Larry made, though, the chance of such an outcome is slim.
Verdict 2: DR Strategy Doesn’t Offer Sufficient Protection
There’s a solid chance that The Business could judge Larry’s DR plan as falling short. Perhaps the intranet houses areas that are highly volatile with critical data that changes frequently throughout the day. If an outage were to occur at 4pm in the afternoon, an entire day’s worth of data would basically be lost because the most recent backup would likely be 12 or so hours old (remember: the DR plan calls for nightly backups). Loss of that data could be exceptionally costly to the organization.
At the same time, Larry’s recovery strategy assumes that he has enough time to restore farm-level backups at the off-site location in the event of a disaster. Restoring a full SharePoint farm-level backup (with the potential addition of differential backups) could take hours. If having the intranet down costs the company $100,000 per hour in lost productivity or revenue, you can bet that The Business will not be happy with Larry’s DR plan in its current form.
Verdict 3: DR Strategy is Overkill
On the flipside, there’s always the chance that Larry’s plan is overkill. If the company’s intranet possesses primarily static content that changes very infrequently and is of relatively low importance, nightly backups and a warm off-site standby SharePoint farm may be overkill. Sure, it’ll certainly allow The Business to get their intranet back in a timely fashion … but at what cost?
If a monthly tape backup rotation and a plan to buy hardware in the event of a disaster is all that is required, then Larry’s plan is unnecessarily costly. Money is almost always constrained in DR planning and execution, and most organizations prioritize their DR target systems carefully. Extra money that is spent on server hardware, nightly backups, and maintenance for a warm off-site SharePoint farm could instead be allocated to the DR strategies of other, more important systems.
Taking Care of Business First
No one wants to be left guessing whether or not their SharePoint DR strategy will adequately address DR needs without going overboard. In approaching the challenge his manager handed him without obtaining any additional input, Larry fell into the same trap that many IT professionals do when confronted with DR: he failed to obtain the quantitative targets that would allow him to determine if his DR plan would meet the needs and expectations established by The Business. In their most basic form, these requirements come in the form of recovery point objectives (RPOs) and recovery time objectives (RTOs).
The Disaster Recovery Timeline
I have found that the concepts of RPO and RTO are easiest to explain with the help of illustrations, so let’s begin with a picture of a disaster recovery timeline itself:
The diagram above simply shows an arbitrary timeline with an event (a “declared disaster”) occurring in the middle of the timeline. Any DR planning and preparation occurs to the left of the event on the timeline (in the past when SharePoint was still operational), and the actual recovery of SharePoint will happen following the event (that is, to the right of the event on the timeline in the “non-operational” period).
This DR timeline will become the canvas for further discussion of the first quantitative DR target you need to obtain before you can begin planning a SharePoint DR strategy: RPO.
RPO: Looking Back
As stated a little earlier, RPO is an acronym for Recovery Point Objective. Though some find the description distasteful, the easiest way to describe RPO is this: it’s the maximum amount of data loss that’s tolerated in the event of a disaster. RPO targets vary wildly depending on volatility and criticality of the data stored within the SharePoint farm. Let’s add a couple of RPO targets to the DR timeline and discuss them a bit further.
Two RPO targets have been added to the timeline: RPO1 and RPO2. As discussed, each of these targets marks a point in the past from which data must be recoverable in the event of a disaster. In the case of our first example, RPO1, the point in question is 48 hours before a declared disaster (that is, “we have a 48 hour RPO”). RPO2, on the other hand, is a point in time that is a mere 30 minutes prior to the disaster event (or a “30 minute target RPO”).
At a minimum, any DR plan that is implemented must ensure that all of the data prior to the point in time denoted by the selected RPO can be recovered in the event of a disaster. For RPO1, there may be some loss of data that was manipulated in the 48 hours prior to the disaster, but all data older than 48 hours will be recovered in a consistent state. RPO2 is more stringent and leaves less wiggle room; all data older than 30 minutes is guaranteed to be available and consistent following recovery.
If you think about it for a couple of minutes, you can easily begin to see how RPO targets will quickly validate or rule-out varying backup and/or data protection strategies. In the case of RPO1, we’re “allowed” to lose up to two days (48 hours) worth of data. In this situation, a nightly backup strategy would be more than adequate to meet the RPO target, since a nightly backup rotation guarantees that available backup data is never more than 24 hours old. Whether disk or tape based, this type of backup approach is very common in the world of server management. It’s also relatively inexpensive.
The same nightly backup strategy would fail to meet the RPO requirement expressed by RPO2, though. RPO2 states that we cannot lose more than 30 minutes of data. With this type of RPO, most standard disk and tape-based backup strategies will fall short of meeting the target. To meet RPO2’s 30 minute target, we’d probably need to look at something like SQL Server log shipping or mirroring. Such a strategy is going to generally require a greater investment in database hardware, storage, and licensing. Technical complexity also goes up relative to the aforementioned nightly backup routine.
It’s not too hard to see that as the RPO window becomes increasingly more narrow and approaches zero (that is, an RPO target of real-time failover with no data loss permitted), the cost and complexity of an acceptable DR data protection strategy climbs dramatically.
RTO: Thinking Ahead
If RPO drives how SharePoint data protection should be approached prior to a disaster, RTO (or Recovery Time Objective) denotes the timeline within which post-disaster farm and data recovery must be completed. To illustrate, let’s turn once again to the DR timeline.
As with the previous RPO example, we now have two RTO targets on the timeline: RTO1 and RTO2. Analogous to the RPO targets, the RTO targets are given in units of time relative to the disaster event. In the case of RTO1, the point in time in question is two hours after a disaster has been declared. RTO2 is designated as t+36 hours, or a day and a half after the disaster has been declared.
In plain English, an RTO target is the maximum amount of time that the recovery of data and functionality can take following a disaster event. If the overall DR plan for your SharePoint farm were to have an RTO that matches RTO2, for instance, you would need to have functionality restored (at an agreed-upon level) within a day and half. If you were operating with a target that matches RTO1, you would have significantly less time to get everything “up and running” – only two hours.
RTO targets vary for the same reasons that RPO targets vary. If the data that is stored within SharePoint is highly critical to business operations, then RTOs are generally going to trend towards hours, minutes, or maybe even real-time (that is, an RTO that mandates transferring to a hot standby farm or “mirrored” data center for zero recovery time and no interruption in service). For SharePoint data and farms that are less business critical (maybe a publishing site that contains “nice to have” information), RTOs could be days or even weeks.
Just like an aggressive RPO target, an aggressive RTO target is going to limit the number of viable recovery options that can possibly address it – and those options are generally going to lean towards being more expensive and technically more complex. For example, attempting to meet a two hour RTO (RTO1) by restoring a farm from backup tapes is going to be a gamble. With very little data, it may be possible … but you wouldn’t know until you actually tried with a representative backup. At the other extreme, an RTO that is measured in weeks could actually make a ground-up farm rebuild (complete with new hardware acquisition following the disaster) a viable – and rather inexpensive (in up-front capital) – recovery strategy.
Whether or not a specific recovery strategy will meet RTO targets in advance of a disaster is oftentimes difficult to determine without actually testing it. That’s where the value of simulated disasters and recovery exercises come into play – but that’s another topic for another time.
Closing Words
This post was intended to highlight a common pitfall affecting not only SharePoint DR planning, but DR planning in general. It should be clear by now that I deliberately avoided technical questions and issues to focus on making my point about planning. Don’t interpret my “non-discussion” of technical topics to mean that I think that their place with regard to SharePoint DR is secondary. That’s not the case at all; the fact that John Ferringer and I wrote a book on the topic (the “SharePoint 2007 Disaster Recovery Guide”) should be proof of this. It should probably come as no surprise that I recommend our book for a much more holistic treatment of SharePoint DR – complete with technical detail.
There are also a large number of technical resources for SharePoint disaster recovery online, and the bulk of them have their strong and weak points. My only criticism of them in general is that they equate “disaster recovery” to “backup/restore.” While the two are interrelated, the latter is but only one aspect of the former. As I hope this post points out, true disaster recovery planning begins with dialog and objective targets – not server orders and backup schedules.
If you conclude your reading holding onto only one point from this post, let it be this: don’t attempt DR until you have RPOs and RTOs in hand!
This post explores the SPWebService’s ApplyApplicationContentToLocalServer method, the constraints one faces when using it, and an alternative to its use when updating application page sitemap files.
Caching capabilities that are available (or exposed) through MOSS are something I spend a fair number of working hours focusing on. MOSS publishing farms can make use of quite a few caching options, and wise administrators find ways to leverage them all for maximum scalability and performance. While helping a client work through some performance and scalability issues recently, I ran into some annoying problems with disk-based caching – also known as BLOB (Binary Large OBject) caching. These problems inspired me to create the BlobCacheFarmFlush solution that I’ve shared on CodePlex, and it was during the creation of this solution that I wrangled with the ApplyApplicationContentToLocalServer method.
Background
The BlobCacheFarmFlush solution itself has a handful of moving parts, and the element I’m going to focus on in this post is the administration page (BlobCacheFarmFlush.aspx) that gets added to the farm upon Feature activation. In particular, I want to share some of the lessons I learned while figuring out how to get the page’s navigational (breadcrumb) support operating properly.
Unlike “standard” content pages that one might deploy through a SharePoint Feature or solution package, application pages (also called “layouts pages” because they go into the LAYOUTS folder within SharePoint’s 12 hive) don’t come with wired-up breadcrumb support. An example of the type of breadcrumb to which I’m referring appears below (circled in red):
Unless additional steps are taken during the installation of your application pages (beyond simply placing them in the LAYOUTS folder), breadcrumbs like the one shown above will not appear. It’s not that application pages (which derive from LayoutsBasePage or UnsecuredLayoutsBasePage) don’t include support for breadcrumbs – they do. The reason breadcrumbs fail to show is because the newly added application pages themselves are not integrated into the sitemap files that describe the navigational hierarchy of the layouts pages.
Wiring Up Breadcrumb Support
Getting breadcrumbs to appear in your own application pages requires that you update the layouts sitemap files for each of the (IIS) sites serving up content on each of the SharePoint web front-end (WFE) servers in your farm. The files to which I’m referring are named layouts.sitemap and appear in the _app_bin folder of each IIS site folder on the WFE. An example of one such file (in its _app_bin folder) appears below.
I’m a “best practices” kind of guy, so when I was doing research for my BlobCacheFarmFlush solution, I was naturally interested in trying to make the required sitemap modifications in a way that was both easy and supported. It didn’t take much searching on the topic before I came across Jan Tielens’ blog post titled “Adding Breadcrumb Navigation To SharePoint Application Pages, The Easy Way.” In his blog post, Jan basically runs through the scenario I described above (though in much greater detail than I presented), and he mentions that another reader (Brian Staton) turned him onto a very simple and straightforward way of making the required sitemap modifications. I’ll refer you to Jan’s blog post for the specifics, but the two-step quick summary goes like this:
Create a layouts.sitemap.*.xml file that contains your sitemap navigation additions and deploy it to the LAYOUTS folder within SharePoint’s 12 hive on a server.
Execute code that implements one of the two approaches shown below (typically on Feature activation) :
// Approach #1: Top-down starting at the SPFarm level
SPFarm.Local.Services.GetValue<SPWebService>().ApplyApplicationContentToLocalServer();
// Approach #2: Applying to the sites within an SPWebApplication
myWebApp.WebService.ApplyApplicationContentToLocalServer();
This isn’t much code, and it’s pretty clear that the magic rests with the ApplyApplicationContentToLocalServer method. This method carries out a few operations, but the one in which we’re interested involves taking the new navigation nodes in the layouts.sitemap.*.xml file and integrating them into the layouts.sitemap file for each IIS site residing under a target SPWebService instance. With the new nodes (which tie the new application pages into the navigational hierarchy) present within each layouts.sitemap file, breadcrumbs appear at the top of the new application pages when they are rendered.
I took this approach for a spin, and everything looked great! My sitemap additions were integrated as expected, and my breadcrumb appeared on the BlobCacheFarmFlush.aspx page. All was well .. until I actually deployed my solution to its first multi-server SharePoint environment. That’s when I encountered my first problem.
Problem #1: The “Local” Part of the ApplyApplicationContentToLocalServer Method
When I installed and activated the BlobCacheFarmFlush solution in a multi-server environment, the breadcrumbs failed to appear on my application page. It took a little legwork, but I discovered that the ApplyApplicationContentToLocalServer method has “Local” in its name for a reason: the changes made through the method’s actions only impact the server on which the method is invoked.
This contrasts with the behavior that SharePoint objects commonly exhibit. The changes that are made through (and to) many SharePoint types impact data that is actually stored in SQL Server, and changes made through any farm member get persisted back to the appropriate database and become available through all servers within the farm. The ApplyApplicationContentToLocalServer method, on the other hand, carries out its operations directly against the files and folders of the server on which the method is called, and the changes that are made do not “automagically” appear on or through other farm members.
The Central Administration host server for the farm in which I was activating my Feature wasn’t one of the WFEs serving up my application page. When I activated my Feature from within Central Admin, my navigation additions were incorporated into the affected sites on the local (Central Admin) host … but the WFEs serving up actual site pages (and my application page) were not updated. Result: no breadcrumb on my application page.
This issue is one of those problems that wouldn’t normally be discovered in a typical development environment. Most of the SharePoint developers I know do their work within a virtual machine (VM) of some sort, so it’s not until one moves out of such an environment and into a multi-server environment that this type of deployment problem even makes itself known. This issue only serves to underscore how important it is to test Features and solutions in a typical target deployment environment before releasing them for general use.
Putting my thinking cap back on, I worked to come up with another way to integrate the sitemap changes I needed in a way that was multi-server friendly. The ApplyApplicationContentToLocalServer method still seemed like a winner given all that it did for a single line of code; perhaps all I needed to do was create and run a one-time custom timer job (that is, schedule a custom SPJobDefinition subclass) on each server within the farm and have that timer job execute the ApplyApplicationContentToLocalServer method locally.
I whipped-up a custom timer job to carry out this action and took it for a spin. That’s when I ran into my second problem.
Problem #2: Rights Required for ApplyApplicationContentToLocalServer Method Invocation
Prior to the creation of the custom timer job that I was going to use to update the sitemap files on each of the WFEs, I had basically ignored this point. The local administrator requirement quickly became a barricade for my custom timer job, though.
Timer jobs, both SharePoint-supplied and custom, are executed within the context of the SharePoint Timer Service (OWSTIMER.EXE). The Timer Service runs in an elevated security context with regard to the SharePoint farm, but its privileges shouldn’t extend beyond the workings of SharePoint. Though some SharePoint administrators mistakenly believe that the Timer Service account (also known as the “database access account” or “farm service account”) requires local administrator rights on each server within the SharePoint farm, Microsoft spells out that this is neither required nor recommended.
The ApplyApplicationContentToLocalServer method works during Feature activation when the activating user is a member of the Local Administrators group on the server where activation is taking place – a common scenario. The process breaks down, however, if the method call occurs within the context of the SharePoint Timer Service account because it isn’t (or shouldn’t be) a member of the Local Administrators group. Attempts to call the ApplyApplicationContentToLocalServer method from within a timer job fail and result in an “Access Denied” message being written to the Application Event Log. A quick look at the first section of code inside the method itself (using Reflector) makes this point pretty clearly:
if (!SPAdministrationServiceUtilities.IsCurrentUserMachineAdmin())
{
thrownew SecurityException(SPResource.GetString("AccessDenied", newobject[0]));
}
This revelation told me that the ApplyApplicationContentToLocalServer method simply wasn’t going to cut the mustard for my purposes unless I wanted to either (a) require that the Timer Service account be added to the Local Administrators group on each server in the farm, or (b) require that an administrator manually execute an STSADM command or custom command line application to carry out the method call. Neither of these were acceptable to me.
Method Deconstruction
Since I couldn’t use the ApplyApplicationContentToLocalServer method directly, I wanted to dissect it to the extent that I could in order to build my own process in a manner that replicated the method’s actions as closely as possible. Performing the dissection (again via Reflector), I discovered that the method was basically iterating through each SPIisWebSite in each SPWebApplication within the SPWebService object being targeted. As implied by its type name, each SPIisWebSite represents a web site within IIS – so each SPIisWebSite maps to a physical web site folder within the file system at C:\Inetpub\wwwroot\wss\VirtualDirectories (by default if IIS folders haven’t been redirected).
Once each of the web site folder paths is known, it isn’t hard to drill down a bit further to each layouts.sitemap file within the _app_bin folder for a given IIS web site. With the fully qualified path to each layouts.sitemap file computed, it’s possible to carry out a programmatic XML merge with the new sitemap data from a layouts.sitemap.*.xml file that is deployed with a custom Feature or solution. The ApplyApplicationContentToLocalServer method carries out such a merge through the private (and obfuscated) MergeAspSiteMapFiles method of the SPAspSiteMapFile internal type, but only after it has created a backup copy of the current layouts.sitemap file using the SPAspSiteMapFile.Copy method.
The Solution
With an understanding of the process that is carried out within the ApplyApplicationContentToLocalServer method, I proceeded to create my own class that effectively executed the same set of steps. The result was the UpdateLayoutsSitemapTimerJob custom timer job definition that is part of my BlobCacheFarmFlush solution. This class mimics the enumeration of SPWebApplication and SPIisWebSite objects, the backup of affected layouts.sitemap files, and the subsequent XML sitemap merge of the ApplyApplicationContentToLocalServer method. The class is without external dependencies (beyond the SharePoint object model), and it is reusable in its current form. Simply drop the class into a SharePoint project and call its DeployUpdateTimerJobs static method with the proper parameters – typically from the FeatureActivated method of a custom SPFeatureReceiver. The class then takes care of provisioning a timer job instance that will update the layouts.sitemap navigational hierarchy for affected sites on each of the servers within the farm.
As an aside: while putting together the UpdateLayoutsSitemapTimerJob, there were times when I thought I had to be missing something. On a handful of occasions, I found myself thinking, “Certainly there had to be a multi-server friendly version of the ApplyApplicationContentToLocalServer method.” When I didn’t find one (after much searching), I had the good fortune of stumbling upon Vincent Rothwell’s “Configuring the breadcrumb for pages in _layouts” blog post. Vincent’s post predates my own by a hefty two and a half years, but in it he describes a process that is very similar to the one I eventually ended up implementing in my custom timer job. Seeing his post helped me realize I wasn’t losing my mind and that I was on the right track. Thank you, Vincent.
Conclusion
I can sum up the contents of this post pretty simply: when developing application pages that entail sitemap updates, avoid using the ApplyApplicationContentToLocalServer method unless you’re (a) certain that your Feature will be installed into single server environments only, or (b) willing to direct those doing the installation and activation to carry out some follow-up administration on each WFE in the SharePoint farm.
Why does the ApplyApplicationContentToLocalServer method exist? I did some thinking, and my guess is that it is leveraged primarily when service packs, hotfixes, and other additions are configured via the SharePoint Products and Technologies Configuration Wizard. Anytime a SharePoint farm is updated with a patch or hotfix, the wizard is run on each server by a local administrator.
An examination of the LAYOUTS folder on one of my farm members provided some indirect support for this notion. In my LAYOUTS folder, I found the layouts.sitemap.search.xml file, and it was dated 3/25/2008. I believe (I’m not positive) that this file was deployed with the SharePoint Infrastructure Updates in the middle of 2008, and those updates introduced a number of new search admin pages for MOSS. Since the contents of the layouts.sitemap.search.xml file include quite a few new search-related navigation nodes, my guess is that the ApplyApplicationContentToLocalServer method was leveraged to merge the navigation nodes for the new search pages when the configuration wizard was run.
In the meantime, if you happen to find a way to use this method in a multi-server deployment scenario that doesn’t involve the configuration wizard, I’d love to hear about it! The caveat, of course, is that it has to be a best-practices approach – no security changes, no extra manual work/steps for farm administrators, etc.
Here’s another blog post to file in the “I’ll probably never need it, but you never know” bucket of things you’ve seen from me.
