WebPartAdderExtension – a better way of deploying web parts

Recently, that nice young man Wictor Wilén blogged about WebPartAdderExtension and web part gallery sources – this is a little-known (and currently undocumented) capability in SharePoint which allows you to deploy custom web parts in a different way, with huge benefits. In summary, SharePoint 2010 provides a means for you to specify web parts to appear in the web part ‘picker’, without requiring a .webpart file in the local web part gallery (which is the usual route). Code is required, but SharePoint 2010 provides a ‘hook’ for you to add this in the form of the WebPartGallerySourceBase class. To help visualize this, the web part highlighted in the image below is deployed using WebPartAdderExtension – it appears just like a custom web part which uses a .webpart file and the user is blissfully unaware of any differences:

I read Wictor’s two excellent articles, and on a recent project I decided to extend his sample into a model which I think works well for many scenarios. Now, I don’t want to be too dramatic, but I think WebPartAdderExtension in general is A Big Deal – I just can’t imagine going back to the old way of doing web parts at all. In this article we’ll first have a recap of what the problem is and why this new way is better, and I’ll then talk about my implementation and where you can download the files.

The problem

In a nutshell, web parts and the web part ‘gallery’ model in SharePoint are great, until you have to make a change to the definition of a web part that’s already out there. This usually takes the form of:

Client: “Hey can we change the title of ‘foo’ web part? Actually that name we came up with isn’t great and users aren’t finding it.”
Client: “Hey can we move ‘foo’ web part into a different category? That’s a quick change right?”

As a developer you say “Sure, I’ll get right on it!”. But then you realise that it’s deployed to everyone’s My Sites. And you have thousands of them. Or maybe it’s in team sites, but come to think of it you have quite a few of those too. This matters, because the web part is in each site collection’s web part gallery (in the .webpart file), and you’ll have to perform some operation to iterate every site and do some sort of processing on each one. Depending on how many sites you have to deal with, this could take could take significant time to run, and will place a certain amount of load on the infrastructure whilst it does. On my last project, it usually took around 7-8 hours to iterate the 25,000 My Sites we had, and we could only run it during a long maintenance window due to the load on both the server doing the processing and the database servers. Of course, 25,000 sites isn’t that big. In larger enterprises, I’ve even heard of folks writing custom multi-threaded applications to deal with such “iterate the sites” scenarios. I’m not saying you won’t need to occasionally perform this iteration for other reasons, but if we could take one common use case (i.e. changes to web part definitions) out of the equation, that’s got to be good in my book.

The solution

Consider that with WebPartAdderExtension, rolling out such a change to (say) 25,000 sites has the following characteristics:

• No iteration required, therefore no load on servers
• Takes immediate effect across all sites
• Depending on your implementation, may not even require an app pool recycle!

How it works: WebPartAdderExtension is a Feature element (XML), and this can be defined at farm/web app/site collection scope (which is interesting in itself!). This ‘points’ to a class which derives from WebPartGallerySourceBase and in here you can do what you like – all you have to do is return a collection. Each item in the collection will then appear as a web part in the web part gallery. Your custom code is fired when a user browses for a web part (in page edit mode). In Wictor’s sample, he fetched items from a SharePoint list, meaning the pseudo web parts were actually list items rather than each having a traditional .webpart file.

 My implementation

Wictor’s sample conveyed the principle to me perfectly, but I ended up with a different model. Here’s what I came up with to meet our requirements:

• A series of XML files, which define web parts by web application (e.g. an XML file for ‘teams’ web parts, an XML file for ‘My Site’ web parts, and so on).
  • N.B. It could be the nature of our web parts, but by web app was definitely the most appropriate factoring for us.
• Code which reads the XML file for the ‘current’ web application
• A web application-scoped Feature, which uses WebPartAdderExtension
  • In fact, several variations of this Feature are created, one per web app. The ‘teams’ web parts Feature uses WebPartAdderExtension to point to the class which knows about ‘teams’ web parts – this in turn reads from the ‘teams’ web parts XML file.

So rather than Wictor’s list-based sample, I’m happy to use XML files on the filesystem – when we specify the web parts which can be picked, we must specify the assembly/type name etc., and since this is in developer territory I’m comfortable with the fact that a WSP deployment is required to update this XML (the files live under ‘_layouts’). Somewhat surprisingly, SharePoint doesn’t cache this data so even in my file-based implementation, an app pool recycle is not required – just update the WSP (and ensure ResetWebServer = false) and the changes take effect immediately. Even if it DID require a recycle, remember this is still several million times better than a long-running process to iterate site collections.

Here’s my XML (custom schema) for defining the set of ‘teams’ web parts (using dummy web parts here, in my COB namespace): 

   1: <?xml version="1.0" encoding="utf-8" ?>

   2: <WebParts>

   3:   <WebPart

   4:     Title="My web part for team sites - deployed via WebPartAdderExtension"

   5:     Category="COB"

   6:     Description="A dummy web part for team sites"

   7:     ID="TeamsWebPart1"

   8:     Assembly="COB.SharePoint.WebPartAdderExtension, Version=1.0.0.0, Culture=neutral, PublicKeyToken=23afbf06fd91fa64"

   9:     Type="COB.SharePoint.WebPartAdderExtension.TeamsWebPart1.TeamsWebPart1"

  10:     IconUrl="_layouts/images/webpart.gif"

  11:     RibbonCommand=""

  12:     OnClientAdd=""

  13:     />

  14:   <WebPart

  15:   Title="My second web part for team sites - deployed via WebPartAdderExtension"

  16:   Category="COB"

  17:   Description="Another dummy web part for team sites"

  18:   ID="TeamsWebPart2"

  19:   Assembly="COB.SharePoint.WebPartAdderExtension, Version=1.0.0.0, Culture=neutral, PublicKeyToken=23afbf06fd91fa64"

  20:   Type="COB.SharePoint.WebPartAdderExtension.TeamsWebPart2.TeamsWebPart2"

  21:   IconUrl="_layouts/images/webpart.gif"

  22:   RibbonCommand=""

  23:   OnClientAdd=""

  24:     />

  25: </WebParts>

We then need some code to consume this. As discussed earlier, this must be a class derived from WebPartGallerySourceBase – since my implementation is ‘XML file per web application’, I created a base class which understands my XML schema and then a derived class per web app. It’s this latter class which knows where to find the XML file:

   0: [Guid("98FF0ED8-6697-4E13-8BCC-161FEE3E4B0C")]
   1: public class GlobalWebPartGallerySource_Teams : GlobalWebPartGalleryXmlSource

   2: {

   3:     public GlobalWebPartGallerySource_Teams(Page page)

   4:         : base(page)

   5:     {

   6:     }

   7:  

   8:     public override string XmlPath

   9:     {

  10:         get

  11:         {

  12:             string path = SPUtility.GetGenericSetupPath(@"template\layouts\COB.SharePoint.WebPartAdderExtension\GlobalWebParts\GlobalWebParts_Teams.xml");

  13:             return path;

  14:         }

  15:     }

  16: }

..and the base class simply reads from the chosen XML file and returns the collection (of GlobalWebPartGalleryItem instances in my case – the important thing is that it derives from WebPartGalleryItemBase):

   1: public class GlobalWebPartGalleryXmlSource : WebPartGallerySourceBase

   2: {

   3: public GlobalWebPartGalleryXmlSource(Page page)

   4:     : base(page)

   5: {

   6: }

   7:  

   8: public virtual string XmlPath

   9: {

  10:     get

  11:     {

  12:         return string.Empty;

  13:     }

  14: }

  15:  

  16: protected override WebPartGalleryItemBase[] GetItemsCore()

  17: {

  18:     List<WebPartGalleryItemBase> items = new List<WebPartGalleryItemBase>();

  19:     string path = XmlPath;

  20:  

  21:     if (File.Exists(path))

  22:     {

  23:         XElement root = XElement.Load(path);

  24:         var webParts = from wpDef in root.Elements("WebPart")

  25:                         select wpDef;

  26:  

  27:         foreach(XElement wp in webParts)

  28:         {

  29:             string title = wp.Attribute("Title").Value;

  30:             string category = wp.Attribute("Category").Value;

  31:             string description = wp.Attribute("Description").Value;

  32:             string iconUrl = wp.Attribute("IconUrl").Value;

  33:             string id = wp.Attribute("ID").Value;

  34:             string ribbonCommand = wp.Attribute("RibbonCommand").Value;

  35:             string onClientAdd = wp.Attribute("OnClientAdd").Value;

  36:             string assembly = wp.Attribute("Assembly").Value;

  37:             string typeName = wp.Attribute("Type").Value;

  38:  

  39:             items.Add(new GlobalWebPartGalleryItem(this, base.Page, title, category, description, iconUrl, id, assembly, typeName, onClientAdd, ribbonCommand));

  40:         }

  41:     }

  42:  

  43:     return items.ToArray();

  44: }

The key bit that links my GlobalWebPartGallerySource_Teams class above with my ‘teams’ web app, is a web app-scoped Feature, having the following elements:

   1: <Elements xmlns="http://schemas.microsoft.com/sharepoint/"> 

   2:   <WebPartAdderExtension 

   3:       Assembly="$SharePoint.Project.AssemblyFullName$" 

   4:       Class="$SharePoint.Type.98ff0ed8-6697-4e13-8bcc-161fee3e4b0c.FullName$"/> 

   5: </Elements>

Note that I’m using the tokenization support to refer to my class using the GUID attribute it’s decorated with. NOTE – when using such tokens the GUID *must* be lowercase (I kid you not). This is just one of a couple of places in Feature/VS schema that I’m aware of which have this requirement.

Also, I’ll let you into a teensy secret, and say that my implementation has a couple of lines of reflection code in GlobalWebPartGalleryItem – this was necessary to carry the title through to the web part instance which was added to the page. Entirely optional, but since this is such an infrequent action in the grand scheme of web server traffic, it’s completely worth it IMHO for a consistent user experience.

In the end, the act of adding the web part to the page is just how the user expects. If you download my sample (link at the end of this article), here’s what it looks like here:

XML which is NOT needed (and must be deleted manually from the VS project)

It’s worth pointing here that you have to explicitly DELETE items from your project to use this approach. Remember that Visual Studio/CKS:Dev just know about the ‘regular’ way of developing web parts, and when you select ‘Add New Item> Visual Web Part’, items are added to your project which you no longer need – specifically this is the .webpart file and the elements.xml file which provisions this into the web part gallery. I’ve not tested, but I’d assume you get an error or duplicate if you forget to remove these. For clarity, it’s these:

Elements.xml (automatically added by Visual Studio) when web part added to project:

   1: <?xml version="1.0" encoding="utf-8"?>

   2: <!-- DO NOT USE - THIS XML IS NO LONGER NEEDED WHEN USING WEBPARTADDEREXTENSION -->

   3: <Elements xmlns="http://schemas.microsoft.com/sharepoint/" >

   4:   <Module Name="MySitesWebPart1" List="113" Url="_catalogs/wp">

   5:     <File Path="MySitesWebPart1\MySitesWebPart1.webpart" Url="MySitesWebPart1.webpart" Type="GhostableInLibrary" >

   6:       <Property Name="Group" Value="Custom" />

   7:     </File>

   8:   </Module>

   9: </Elements>

MySitesWebPart1.webpart:

   1: <?xml version="1.0" encoding="utf-8"?>

   2: <!-- DO NOT USE - THIS XML IS NO LONGER NEEDED WHEN USING WEBPARTADDEREXTENSION -->

   3: <webParts>

   4:   <webPart xmlns="http://schemas.microsoft.com/WebPart/v3">

   5:     <metaData>

   6:       <type name="COB.SharePoint.WebPartAdderExtension.WebParts.MySitesWebPart1.MySitesWebPart1, $SharePoint.Project.AssemblyFullName$" />

   7:       <importErrorMessage>$Resources:core,ImportErrorMessage;</importErrorMessage>

   8:     </metaData>

   9:     <data>

  10:       <properties>

  11:         <property name="Title" type="string">My web part title</property>

  12:         <property name="Description" type="string">My web part description</property>

  13:       </properties>

  14:     </data>

  15:   </webPart>

  16: </webParts>

And that’s it.

Summary

Although the old way of making custom web parts available has worked for years, that doesn’t mean it’s worked well! It turns out there’s an alternative which, in my opinion, has better characteristics – no more having to iterate each site collection if you ever need to make a change to the definition of a web part which is already out there. When I mentioned to our client that I’d used this new approach for our latest web part, the response was “Wow. I think we should go back and switch ALL our custom web parts over to this model.” I think Wictor gets the kudos for bringing it to folks attention though – I’m just providing some icing on the cake :)

Download link

Download my WebPartAdderExtension sample project

Avoiding bugs from cached JavaScript and CSS files in SharePoint

This article got quite detailed so here’s an executive summary – if you store custom JavaScript/CSS files etc. somewhere under SharePoint’s ‘layouts’ folder (or upload them manually to a library within the site), make sure the URL changes on each update. This is usually done with a querystring parameter in the URL e.g. /_layouts/MyProject/JS/MyScript.js?Rev=2011.11.20.1

Update 5th Dec 2011: Correction - if you exclusively use Microsoft's ScriptLink control to add JavaScript references to a page, the issue will NOT affect you if you store the .js files in “layouts”. It WILL, however, if you store them in a library and do not recycle IIS every time the file changes - something that should happen automatically for a WSP deployment to “layouts” of course. You may find though that it's not always possible to use ScriptLink exclusively - on my current project for example, we had many dependencies between JavaScript files, and ScriptLink does not allow you to specify the order (sequence) in which custom .js files should be added to the page (AFAIK). So, whilst your mileage may vary, I think the main thrust of the article still applies for many. In any case, many thanks to Mahmoud Hamed's comment below for spotting the flaw in my test results - I've updated the table below.

For most projects I work on, I prefer to store JavaScript/CSS etc. on the filesystem somewhere under the SharePoint root folder (e.g. “14”), usually under the “layouts” folder - for me, these files are often “site infrastructure” and are therefore critical for branding/functionality of the whole platform. That said, there are a couple of situations where I’d go the other route and store them in the site collection (and thus content database) – these would be:

• For CSS/JS files only applicable to one site (e.g. a team site)
• Sandboxed solutions

When I outlined these reasons in this answer to a forum question, the voting showed many agreed, and in general I see many folks using this approach. However, I’ve noticed recently that many are unaware of a crucial gotcha – and it’s so important that I’m constantly amazed it’s doesn’t get discussed more. The issue is end users not seeing updated JavaScript and CSS files following a recent code deployment. And OK, OK, I cheated a little bit with the article title – in fact the problem is not unique to SharePoint’s “layouts” folder (or even IIS), and when I say “JavaScript and CSS files”, I really mean JavaScript/CSS/images/audio/video and in fact most static files. But hey, sometimes framing a topic in a SharePointy way is the best way to remind folks that general behaviour of the web also applies to us ‘special’ SharePoint-types. It seems many folks may miss this stuff because they are focused on, say, some super-cool SharePoint ribbon development, and in terms of technical reading I think it’s fair to say that many focus almost exclusively SharePoint content.

Problem

As with many of life’s simplest pitfalls, this one is so easy to fall into. Remember that it applies only to SharePoint projects which do some degree of customization (e.g. branding/custom code) and store files on the filesystem – if those two things don’t apply to you, the problem is purely with Microsoft and they take care of it for their files, so there’s nothing to worry about. Otherwise, imagine the following:

1. Developer makes an update to some custom code – previously deployed files are updated. Let’s say we’re deploying some updated JavaScript and CSS.
2. The associated WSPs are deployed to SharePoint, and the files get updated across all web servers in the farm.
3. Users do not see the Javascript/CSS changes until they do a hard refresh (CTRL + F5). Worse still, they could experience script bugs or broken formatting which could make the application completely unusable.

This scenario typically results in the oft-heard question “Hmm, what happens if you do a CTRL + F5?”. Which is all fine, of course, for the test users who are in touch with the developers. In production, it’s certainly not fine for the rest of the organisation – strangely enough, asking 30,000 information workers to do a CTRL + F5 does NOT go down well in the enterprise! Things get even better in an internet/WCM scenario – would you really hear about JavaScript errors or CSS bugs from site users? Would that message actually get to the implementation team (who, let’s remember, are probably not seeing errors in event logs from this stuff)? Or would the users just go elsewhere?

Cause

The reason this happens is because the first time a file is requested, the web server (IIS here of course, but the same applies to most) serves it with a HTTP header which specifies that it can be cached for a long time. For the “layouts” folder, by default this is 31536000 seconds (1 year):

Browsers and proxies typically obey this, so on subsequent page requests, this file is pulled from ‘Temporary internet files’ on the user’s machine or a proxy between the user and the server. Without this, of course, web pages would load much more slowly. My testing showed that, generally, files served from content databases do not get served with the same headers and therefore updates show immediately. However, it turns out that SharePoint 2010 (and maybe earlier) treats files deployed using a Feature differently to files uploaded manually to the same library. My table below attempts to explain most of my testing and results:

Location	Blob cache enabled?	Result	Notes
Style library (publishing site)	No	OK
Style library (publishing site)	Yes	OK
Style library (publishing site)	Yes (now with max-age)	OK	Added by Feature
Style library (publishing site)	Yes (now with max-age)	Issue	Added manually to library
Layouts	Yes	OK	ScriptLink DID add querystring ?rev=q%2Fb304kubfwrNJVD%2BdYdxg%3D%3D but not clear when this gets updated – NOT when file changes!
Layouts	No	Issue	BLOB cache only relevant for files in content DB anyway
Custom library	Yes	Issue	Added manually to library
Custom library	Yes	OK	Added by Feature

Conclusions and notes:

• The issue applies to files in ‘layouts’ or files in a library which were not deployed using a Module tag in a Feature (or at least, which were uploaded through the browser)
  • N.B. I did actually go digging in Reflector to find the source of this behavior, but didn’t find it.
• For JavaScript files, it doesn’t matter how the file is added to the page (e.g. ScriptLink control or plain <script> tag)
• BLOB caching does not make any change
• Sandboxed or farm solution does not make any change
• Note – tested on SharePoint 2010 RTM (old) and SharePoint 2010 SP1 + August CU (fairly new at time of writing)

Solution

For the affected cases, to avoid browsers/proxies caching a file despite it being updated, wherever the file is linked simply add/change a querystring value on each deployment. So instead of something like:

• /_layouts/MyProject/JS/MyScript.js

..we need something like one of these:

• /_layouts/MyProject/JS/MyScript.js?Rev=1.0.0.0
• /_layouts/MyProject/JS/MyScript.js?Rev=2011.11.20.1

The parameter can have any name (though ‘rev’ or ‘revision’ is common), but the important thing is to change/increment it in some way on every modification. If you’re incrementing assembly versions (using AssemblyFileVersion rather than the main version number remember), then it may be appropriate to use same value – on my current project we do this in some places* (more on this later) so that we don’t have to remember to increment links like those above manually. Effectively we use code to read the current assembly file version, and append that as the ‘Rev’ querystring value (and this value will be the TFS automatically-generated build number for the current build). The trade-off, of course, is that you may be forcing users to re-download files the first time after an update which haven’t actually changed – but it’s safer than having bugs. As an aside, you may have noticed Microsoft do this with their links to CSS/JS files – their values will get updated in service packs/cumulative updates etc:

Attempts to solve the problem in a better way

So we can solve the problem by adding the querystring onto any URL which links to a CSS/JS file, great! In many cases though, this means remembering to do a find/replace in Visual Studio on all the locations which have links to these files before every deployment. This isn’t ideal – in our current project, this can be master pages, user controls, code-behind which uses ClientScriptManager and many other locations. What would be ideal is if all this could take care of itself on every release!

I haven’t spent too much time on this, but in our project we do use a solution for CSS files at least. All of our CSS files are emitted to the page using SharePoint’s CssRegistration and CssLink controls – this means all of the links are generated by one control, giving us a convenient ‘funnel’ which we can amend. So, we simply derive from the CssLink control, and in our custom class we rewrite the links to have the ‘Rev’ parameter. The value used is tied to the AssemblyFileVersion of our ‘core’ assembly (which also hosts this control), the idea being that the CI process updates this for every release. Of course, you could use whatever scheme you like:

   1: public class CacheSafeCssLinkControl : CssLink

   2: {

   3:     protected override void Render(HtmlTextWriter output)

   4:     {  

   5:         using (HtmlTextWriter tempWriter = new HtmlTextWriter(new StringWriter()))

   6:         {

   7:             base.Render(tempWriter);

   8:             string html = tempWriter.InnerWriter.ToString();

   9:             if (!string.IsNullOrEmpty(html))

  10:             {

  11:                 html = html.ToLower().Replace(".css\"", string.Format(".css?Rev={0}\"", Core.Common.Utilities.GetAssemblyFileVersion(Assembly.GetExecutingAssembly())));   

  12:             }

  13:             output.Write(html);

  14:         }

  15:     }

  16: }

So this works great for CSS files. But what about other files, especially JavaScript? Well, here’s where we hit the snag – unfortunately Microsoft.SharePoint.WebControls.ScriptLink (the equivalent control for JavaScript links) is sealed, so we can’t use the same approach. Interestingly, if you were happy to use reflection (perhaps with some caching), you could add the ‘Rev’ parameter using the same technique I use in Eliminating large JS files to optimize SharePoint 2010 internet sites – since the collection of links is stored in HttpContext.Current.Items, there’s an opportunity to modify the links before they are written to the page with no need to swap out Microsoft’s control.

I’d probably be comfortable with that, but the real issue in our project at least, is that JavaScript tends to be added to pages through other routes too such as ClientScriptManager and <script> tags. We’d need to update the (large) codebase to exclusively use ScriptLink so that we have a more effective funnel. Still, it gives me an idea of how I might look to do things in the future - in the long term though, it would be nice if Microsoft’s own controls offered a means of adding a cache-busting querystring.

Summary

Whilst developers happily use CTRL + F5 in their development processes, it’s often overlooked that end users will face the same issue of not seeing updated CSS and JavaScript files by default. Clearly this can cause all sorts of issues, including JavaScript bugs if the page HTML structure has changed but the corresponding JavaScript is not being used. The issue is mainly seen where such files are stored in the ‘layouts’ folder, but interestingly appears where files are manually uploaded to a library too (in SharePoint 2010 at least). The key is ensuring that the URL changes every time the file is updated, and since a querystring will suffice there’s no need to actually rename the file. Ultimately it doesn’t matter how you change the querystring value, whether it’s a simple manual update or a more automated process tied into CI.