It’s a fact of life when dealing with many caching systems: for all the benefits they provide, they occasionally become corrupt or require some form of intervention to ensure healthy ongoing operation. The MOSS Binary Large Object (BLOB) cache, or disk-based cache, is no different.
Is BLOB Cache Corruption a Common Problem?
In my experience, the answer is “no.” The MOSS BLOB cache generally requires little maintenance and attention beyond ensuring that it has enough disk space to properly store the objects it fetches from the lists within the content databases housing your publishing site collections.
How Should a Flush Be Carried Out?
When corruption does occur or a cache flush is desired for any reason, the built-in “Disk Based Cache Reset” option is typically adequate for flushing the BLOB cache on a single server and single web application zone. This option (circled in red on the page shown to the right) is exposed through the Site collection object cache menu item on a publishing site’s Site Collection Administration menu. Executing a flush is as simple as checking the supplied checkbox and clicking the OK button at the bottom of the page. When a flush is executed in this fashion, it affects only the server to which the postback occurs and only the web application through which the request is directed. If a site collection is extended to multiple web applications, only one web application’s BLOB cache is affected by this operation.
Alternatively, my MOSS 2007 Farm-Wide BLOB Cache Flushing Solution (screenshot shown on the right) can be used to clear the BLOB cache folders associated with a target site collection across all servers in a farm and across all web applications (zones) serving up the site collection. This solution utilizes a different mechanism for flushing, but the net effect produced is the same as for the out-of-the-box (OOTB) mechanism: all BLOB-cached files for the associated site collection are deleted from the file system, and the three BLOB cache tracking files for each affected web application (IIS site) are reset.
For more information on the internals of the BLOB Cache, the flush process, and the files I just mentioned, see my previous post entitled We Drift Deeper Into the Sound … as the (BLOB Cache) Flush Comes.
Okay, I Tried a Flush and it Failed. Now What?
If the aforementioned flush mechanisms simply aren’t working for you, you’re probably staring down the barrel of a manual BLOB cache flush. Just delete all of the files in the target BLOB cache folder (as specified in the web.config) and you should be good to go, right?
Wrong.
Jumping in and simply deleting files without stopping requests to the affected site collection (or rather, the web application/applications servicing the site collection) risks sending you down the road to (further) cache corruption. This risk may be small for sites that see little traffic or are relatively small, but the risk grows with increasing request volume and site collection size. Allow me to illustrate with an example.
The Context
Let’s say that you decided to manually clear the BLOB cache for a sizable publishing site collection that is heavily trafficked. You go into the file system, find your BLOB cache folder (by default, C:\blobCache), open it up, select all files and sub-folders contained within, and press the <Delete> key on your keyboard. Deletion of the BLOB cache files and sub-folders commences.
Deleting the sub-folders and files isn’t an instantaneous operation, though. It takes some time. While the deletion is taking place, let’s say that your MOSS publishing site collections are still up and servicing requests. The web applications for which BLOB caching is enabled are still attempting to use the very folders and files currently being deleted.
The Race Condition
For the duration of the deletion, a race condition is in effect that can yield some fairly unpredictable results. Consider the following possible execution sequence. Note: this example is hypothetical, but I’ve seen results on multiple occasions that infer this execution sequence (or something similar to it).
- The deletion operation deletes one or more of the .bin files at the root of a web application’s BLOB cache folder. These files are used by MOSS to track the contents of the BLOB cache, the number of times it was flushed, etc.
- A request for a resource that would normally be present in the BLOB cache arrives at the web server. An attempted lookup for the resource in the BLOB cache folder fails because the .bin files are gone as a result of the actions taken in the last step.
- The absence of the .bin files kicks off some housekeeping. Ultimately, a “fresh” set of .bin files written out.
- The requested resource is fetched into the BLOB cache (sub-)folder structure and the .bin files are updated so that subsequent requests for the resource are served from the file system instead of the content database.
- The deletion operation, which has been running the whole time, deletes the file and/or folder containing the resource that was just fetched.
Once the deletion operation has concluded, a resource that was fetched in step #4 is tracked in the BLOB cache’s dump.bin file, but as a result of step #5, the resource no longer actually exist in the BLOB cache file system. Net effect: requests for these resources return HTTP 404 errors.
Since image files are the most common BLOB-cached resources, broken link images (for example, that nasty red “X” in place of an image in Internet Explorer) are shown for these tracked-but-missing resources. No amount of browser refreshing brings the image back from the server; only an update to the image in the content database (which triggers a re-fetch of the affected resource into the BLOB cache) or another flush operation fixes the issue as long as BLOB caching remains enabled.
Proper Manual Clearing
The key to avoiding the type of corruption scenario I just described is to ensure that requests aren’t serviced by the web application or applications that are tied to the BLOB cache. Luckily, this is accomplished in a relatively straightforward fashion.
Before attempting either of the approaches I’m about to share, though, you need to know where (in the server file system) your BLOB cache root folder is located. By default, the BLOB cache root folder is located at C:\blobCache; however, most conscientious administrators change this path to point to a data drive or non-system partition.
If you are unsure of the location of the BLOB cache root folder containing resources for your site collection, it’s easy enough to determine it by inspecting the web.config file for the web application housing the site collection. As shown in the sample web.config file on the right, the location attribute of the <BlobCache> element identifies the BLOB cache root folder in which each web application’s specific subfolder will be created.
Be aware that saving any changes to the web.config file will result in an application pool recycle, so it’s generally a good idea to review a copy of the web.config file when inspecting it rather than directly opening the web.config file itself.
The Quick and Dirty Approach
When you just want to “get it done” as quickly as possible using the least number of steps, this is the process:
Stop the World Wide Web Publishing Service on the target server. This can be accomplished from the command line (via net stop w3svc) or the Services MMC snap-in (via Start –> Administrative Tools –> Services) as shown on the right.- Once the World Wide Web Publishing Service stops, simply delete the BLOB cache root folder. Ensure that the deletion operation completes before moving on to the next step.
- Restart the World Wide Web Publishing service (via Services or net start w3svc).
Though this approach is quick with regard to time and effort invested, it’s certainly “dirty,” coarse, and not without disadvantages. Using this approach prevents the web server from servicing *any* web requests for the duration of the operation. This includes not only SharePoint requests, but requests for any other web site that may be served from the server.
Second, the “quick and dirty” approach wipes out the entire BLOB cache – not just the cached content associated with the web application housing your site collection (unless, of course, you have a single web application that hasn’t been extended). This is the functional equivalent of trying to drive a nail with a sledgehammer, and it’s typically overkill in most production scenarios.
The Controlled (Granular) Approach
There is a less invasive alternative to the “Quick and Dirty” technique I just described, and it is the procedure I recommend for production environments and other scenarios where actions must be targeted and impact minimized. The screenshots that follow are specific to IIS7 (Windows Server 2008), but the fundamental activities covered in each step are the same for IIS6 even if execution is somewhat different.
- Determine the IIS ID of the web application servicing the site collection for which the flush is being performed. This is easily accomplished using the Internet Information Services (IIS) Manager (accessible through the Administrative Tools menu) as shown to the right. If I’m interested in clearing the BLOB cache of a site collection that is hosted within the InternalHomeWeb (Default) web application, for example, the IIS site ID of interest is 1043653284.
- Determine the name of application pool that is servicing the web application. In IIS7, this is accomplished by selecting the web application (InternalHomeWeb (Default)) in the list of sites and clicking the Basic Settings… link under Edit Site in the Site Actions menu on the right-hand side of the window. The dialog box that pops up clearly indicates the name of the associated application pool (as shown on the right, circled in red). Note the name of the application pool for the next step.
- Stop the application pool that was located in the previous step. This will shutdown the web application and prevent MOSS from serving up requests for the site collections housed within the web application, thus avoiding the sort of race condition described earlier. If multiple application pools are used to partition web applications within different worker processes, then shutting down the application pool is “less invasive” than stopping the entire World Wide Web Publishing Service as described in “The Quick and Dirty Approach.” If all (or most) web applications are serviced by a single application pool, though, then there may be little functional benefit to stopping the application pool. In such a case, it may simply be easier to stop the World Wide Web Publishing Service as described in “The Quick and Dirty Approach.”
- Open Windows Explorer and navigate to the BLOB cache root folder. For the purposes of this example, we’ll assume that the BLOB cache root folder is located at E:\MOSS\BLOB Cache. Within the root folder should be a sub-folder with a name that matches the IIS site ID determined in step #1 (1043653284). Either delete the entire sub-folder (E:\MOSS\BLOB Cache\1043653284), or select the files within the sub-folder and delete them (as shown above).
- Once the deletion has completed, restart the application pool that was shutdown in step #3. If the World Wide Web Publishing Service was shutdown instead, restart it.
Taking the approach just described affects the fewest number of cached resources necessary to ensure that the site collection in question (or rather, its associated web application/applications) starts with a “clean slate.” If web applications are partitioned across multiple application pools, then this approach also restricts the resultant service outage to only those site collections ultimately being served by the application being shutdown and restarted.
Some Common Questions and Concerns
Q: I have multiple servers or web front-ends. Do I need to take them all down and manually flush them as a group?
The BLOB cache on each MOSS server operates independently of other servers in the farm, so the answer is “no.” Servers can be addressed one at a time and in any order desired.
Q: I’ve successfully performed a manual flush and brought everything back up, but I’m *still* seeing an old image/script/etc. What am I doing wrong?
Interestingly enough, this type of scenario oftentimes has little to do with the actual server-side BLOB cache itself.
One of the attributes that can (and should) be configured when enabling the BLOB cache is the max-age attribute. The max-age attribute specifies the duration of time, in seconds, that client-side browsers should cache resources that are retrieved from the MOSS BLOB cache. Subsequent requests for these resources are then served directly out of the client-side cache and not made to the MOSS server until a duration of time (specified by the max-age attribute) is exceeded.
If a BLOB cache is flushed and it appears that old or incorrect resources (commonly images) are being returned when requested, it might be that the resources are simply cached on the local system and being returned from the cache instead of being fetched from the server. Flushing locally-cached items (or deleting “Temporary Internet files” in Internet Explorer’s terminology) is a quick way to ensure that requests are being passed to the SharePoint server.
Q: I’m running into problems with a manual deletion. Sometimes all files within the cache folder can’t be deleted, or sometimes I run into strange files that have a size of zero bytes. What’s going on?
I haven’t seen this happen too often, but when I have seen it, it’s been due to problems with (or corruption in) the underlying file system. If regular CHKDSK operations aren’t scheduled for the drive housing the BLOB cache, it’s probably time to set them up.
Additional Reading and References
- MSDN: Caching In Office SharePoint 2007
- CodePlex: MOSS 2007 Farm-Wide BLOB Cache Flushing Solution
- Blog Post: We Drift Deeper Into the Sound … as the (BLOB Cache) Flush Comes
Hello,
After installing Service pack2, the images in people search results are appearing to be elongated and when we refresh or if we clear cache in browser only then it becomes normal.. may i kno the reason and solution as we cant remove cache on each users machine if users are n numbers..
Please help..
Yash,
If a refresh of the page or clearing the client-side cache of images fixes the issue you’re encountering, then chances are that there is no server-side fix that is going to address the behavior you’re seeing. If the MOSS BLOB cache is involved (something you’ll have to verify), then I’m guessing that the web.config entry for the element on the web application in question specifies a max-age attribute value.
When the max-age attribute is specified, images and other BLOB resources that are served up to clients are passed back to the browser with client cacheability headers. In essence, the client will hold onto such images and resources until the amount of time specified by the max-age attribute has elapsed. Only after that amount of time has elapsed will the browser once again begin requesting the images or resources from the WFE. This is why a client-side solution, such as forcing a re-fetch or clearing the browser cache, is the only one that will actually get the browser to re-request the objects in question.
It’s been a couple of days since you submitted your comment. It’s common for the max-age attribute to be set to something like 12 hours. Your situation may even be fixed by now. If it’s not, and the max-age attribute is causing client-side caching, you should be able to use the value of the attribute to determine when things will clear up for you. The max-age attribute specifies a client-side caching duration in seconds, so divide the value by 3600 to determine how many hours objects will remain cached on client browsers.
Again, my hypothesis is entirely dependent on BLOB caching being enabled and a max-age attribute being set. If either of these aren’t in-play, then what I’m proposing isn’t what’s happening.
Good luck!
Wouldn’t it be better to simply change the blob cache location in the web config?
Changing the location of the BLOB cache in the web.config is certainly a viable approach, Fred. ASP.NET does maintain a file system watcher on web.config files, so changing the web.config will force a recycle of the AppDomain for the zone of the Web application associated with the web.config. You could then go in after the fact and clean-up the old BLOB cache directory in the file system.
Whether or not this is “better” sort of depends on your situation. If you switch the BLOB cache to a completely new directory, it means that every cacheable asset is going to have to be re-fetched from the content databases housing them to populate the new BLOB cache. If you aren’t caching a lot of content on the WFEs, then this probably isn’t a big deal.
If you’ve got gigabytes of data that are being BLOB cached, though, re-fetching all of that content is going to put a strain (albeit a brief one) on your infrastructure. In this scenario, a targeted clean-up of a single Web application zone in the existing BLOB cache will be less invasive — assuming you have more than just the target Web application zone’s content in your BLOB cache, of course.
You’re absolutely right, though, in that there is more than one way to skin this cat; the approach I presented is not the only way. More than anything, I simply wanted to share the possible adverse side effects of clearing the file system without taking precautions.
Thanks for the feedback!
This article saved my a$$! I accidentally tried to delete the blob cache (having decided not to research before I did) by clearing the file system folder. With blob caching still turned on, all images on my site were broken. I tried resetting the disk cache in the site collection setting but that did not work either. So I took the easy way out and just changed the location of the blob cache to a different directory.
So, having cleared the file system incorrectly and clearing the disk cache not working, what would you recommend as a solution to get this working?
Greg,
If you’re in a situation where you can shut down the app pool or W3CSVC servicing each of your WFEs, clear out the file system, and then restart the app pool or service, you should be fine. If you have multiple WFEs, each one can be cleared in this fashion one at a time; there are no depedencies between BLOB caches on WFEs.
If your environment can’t tolerate any downtime, I would strongly suggest checking out the farm-level BLOB cache flushing solution I put together (http://blobcachefarmflush.codeplex.com). I created it specifically to deal with high traffic, can’t-be-brought-down environments where the cache occasionally needed a flush due to unusual corruption situations. If you install that solution, you’ll be able to safely flush the BLOB cache for the Web application housing your site collection. The solution is really easy to use, and the flush will take place across all WFEs and extended zones.
The problem with the built-in flush mechanism (exposed through the site collection administration settings) is that the flush only happens for the server that receives the postback, and the flush only occurs for the zone (IIS site) you are working through. If you have a Web application that has been extended to more than one zone and/or more than one WFE, it’s difficult to get the built-in flushing mechanism to work for you.
If you want a bit more background on what’s happening with the flush, check out another post of mine: http://sharepointinterface.com/2009/06/18/we-drift-deeper-into-the-sound-as-the-flush-comes/
Good luck!
Hi Sean,
Thanks for this exciting post. I am in middle of an issue.
We have a MOSS 2007 website hosted on separate staging and production farms. In this site we have a document library holding .css and .js files.
Issue is that when a .css or .js file is updated (e.g. overwritten by an updated file) on staging and then deployed to production (using a tool based on content deployment APIs), on one of the production WFEs the file that is served remains the old one. While from other production WFEs, new file gets served as expected.
The problematic server starts serving new file only when Blob cache is manually flushed from _layouts/settings.aspx -> “site collection object cache” -> “Force this server to reset its disk based cache”.
So I am suspecting that the blob cache is not getting refreshed as it should on this server.
Web.config entry for blob caching on this server is as below:
Kindly help !!
Kaushik,
Thanks for the comment! Unfortunately, your web.config file entry didn’t come through; I’m sure WordPress stripped it out since content that is marked-up is generally blocked.
As a general rule of thumb, the background sweeper thread the monitors for file changes should pick up the fact that the specific files (.css and .js files) changed within the content database. Without further digging, I don’t really know why one of your WFEs wouldn’t see the changes when the other WFEs do.
Some additional references and potential help:
1. My farm-level flushing solution will give you the ability to do a clean flush across all servers in the farm, and it doesn’t resort to the manual process I described in this post. You can get the solution from CodePlex: http://blobcachefarmflush.codeplex.com/
2. I have another post that goes into the mechanics of the BLOB cache flush process (and BLOB cache operations in general) here: http://sharepointinterface.com/2009/06/18/we-drift-deeper-into-the-sound-as-the-flush-comes/ I’d check it out; it may give you some additional insight into what’s happening.
I hope that helps :-)
I had to delete the web application and forgot to turn off blob cache before deleting the web application. Now, I want the associate blob cache folder deleted from the system. Can I manually delete the folder in which the images/etc were cached from both WFEs (load balanced)?
SharePoint Admin: assuming the folder is no longer in use, then the answer is “yes” – you can delete it :-)
Thanks for your prompt response. The folder was automatically created for the web application that was deleted with the same name as the application pool ID (which no longer exist).
So I guess it can be deleted. Thanks!