SharePoint WCM in the finance sector

You might have been wondering where I've been for the last few weeks, so I thought I'd show rather than just tell. I rarely talk about specific 'daytime' projects I work on, but I guess I'm quite proud of this one and you might be interested in what we've done. I've been working with LBi (Europe's biggest design/technology agency) recently, leading a team of 5-7 developers on a WCM (Web Content Management) project for one of the global investment banks. You can see some aspects of what we've done with SharePoint in this post, and I'll follow up with some developer-to-developer info about techniques we used in the next one. Additionally, for UK-based folks I think Riaz Ahmed (my current boss and Mr SharePoint at LBi) is hoping to do a real show and tell at the UK SharePoint user group in a couple of months, where there'll be the opportunity for Q&A and to pick up far more info/lessons learnt (both technical and non-technical) than I can provide here.

The site is targeted at a closed audience over the internet (our client's clients), so unfortunately you can't see it for yourself - the site provides analysts with high-value information on financial markets, primarily in the form of market reports and newsflashes, though there are other content types. The user experience we were tasked with implementing (created by LBi's design team) means the user gets to have some pretty impressive functionality around all this (IMHO) which differentiates the service from others.

The centrepiece is the market report data, which is held for many countries and topics:

This is imported nightly from a master datasource (thus lending itself nicely to caching), and our client can also add comments to the imported data via SharePoint WCM page editing. Perhaps the slickest piece of functionality is the facility for the user to create personalized 'smart reports' with specific markets and topics which interest him/her. These are saved against the user's profile for quick access:

At runtime we build the personalized page from the same cache layer used by the standard pages in the first shot.

Since financial markets change quickly, our client's authoring teams (distributed around the world), can quickly create newsflashes via SharePoint publishing functionality for display on the site - this is controlled via SharePoint workflow. Users can search and filter by country, date, high priority flag and can also store in a 'My newsflashes' bucket for quick retrieval:

Users also have the option of receiving e-mail alerts when newsflashes are added to the site (immediate for high priority newsflashes, nightly batch for others), and can tailor their e-mail preferences by country...

..and topic:

The site also has a ton of other good stuff, such as:

• single sign-on with a sister site
• integration with a higher-level portal which can authenticate users itself (meaning users do not log-in separately on our site)
• WCAG 'A' -level accessibility - accessibility wasn't particularly a focus of the client for this site, but (happily) is just how things are done here anyhow. We use a lot of JavaScript to enhance the user experience but it always provides equivalent functionality when disabled - I have to say the LBi front-end developers use some incredibly innovative techniques to achieve this, I've rarely seen anything like it (see below for some examples).
• integration with Endeca for search (including the newsflash landing page you see above - this is powered by search)
• custom admin functionality for user creation/management
• many many smaller but 'interesting' requirements such as newsflashes having user-selected 'related items', attachments etc etc!

Rich, accessible controls

In addition to all the personalization, one of the things that I love about the user experience is the funky controls we have. So why have a dull dropdown like this:

...when you can have one like this:

 

This allows us to present only countries the user has told us they are interested in in the dropdown (thus avoiding clutter), whilst allowing them to easily see data for other countries if they have the occasional need to. To do this, clicking the 'More...' link expands the dropdown to show a second section containing the other countries (or more specifically, the other countries they are permitted to see):

Similarly for multiple selections, why have this:

...when you can have this:

 

Again, clicking the 'More...' link allows the user to select a country not in their 'preferred' countries:

Personally I think this makes for a great user experience, and it's all accessibility-compliant.

Project plan : caffeine

The development time for everything you see here (and everything you don't) was around 7-8 weeks, though this excludes some of the front-end development (HTML, CSS, JS). We're not quite over the finishing line yet, but are in the final stages of user acceptance testing and security testing. To say it's been tough would be something of an understatement, but I'm incredibly proud of my guys and it's been a great team effort - fortunately for me they're very talented :-)

Next time I'll detail some of the technical decisions we took and go through some developer bits which we felt worked well when building/deploying the site.

SPDataSource - every developer's friend (part 2)

SPDataSource - a refresher

So last time in part 1 of this 2-part series, we saw how SPDataSource is a great option for fetching data from lists since not only does it do the actual work of fetching the items for you, it also does this without any C#/VB.Net code. We can retrieve all the items from the list, or optionally supply a CAML query along with the list details if we wish to filter/sort the items. So as I showed in part 1, we can easily display a dropdown containing the items from a list with the following markup:

<SPWebControls:SPDataSource runat="server" ID="dsPersonTitles" DataSourceMode="List" 
  SelectCommand="<Query><OrderBy><FieldRef Name='SortOrder' Ascending='true' /></OrderBy></Query>"
  <SelectParameters>
      <asp:Parameter Name="WebUrl" DefaultValue="/configuration/" />
      <asp:Parameter Name="ListName" DefaultValue="PersonTitles" />
  </SelectParameters>
</SPWebControls:SPDataSource> 

<asp:DropDownList runat="server" ID="ddlPersonTitles" CssClass="title" DataSourceID="dsPersonTitles" DataTextField="Title" DataValueField="ID">
</asp:DropDownList>

We also saw how SPDataSource offers more than just retrieving items from a list, with the following other 'modes' (see part 1 for more details on these):

• CrossList - similar to doing a query with SPSiteDataQuery across all lists in a site collection
• ListItem - show field values from a single list item


• Webs - lists all webs in a site collection


• ListOfLists - lists all lists in a web

What I want to focus on this time is how to use parameters with SPDataSource, since when using the control for real you'll often want to do this.

Parameters with SPDataSource

SPDataSource works like other .Net data source controls, so the key to passing parameters is using one of the .Net 2.0 parameter classes, either declaratively (in markup) or in code. The example above shows using the basic Parameter class by setting the DefaultValue property to a known string, but the following parameter types can also be used (though note I haven't tried them all and the SPD Team Blog article I mentioned says somewhat vaguely that "most" of them can be used!):

• ControlParameter


• CookieParameter


• FormParameter


• ProfileParameter


• QueryStringParameter


• SessionParameter

Effectively using one of these saves you writing code to read the value from the respective location and passing it to SPDataSource yourself. I think that ControlParameter and QueryStringParameter are possibly of the most value, though all could be interesting depending on your requirements. As an example, to get the value a user selected earlier from a dropdown containing a list of lists and pass it to a SPDataSource control I would need:

<SelectParameters>
    <asp:Parameter Name="WebUrl" DefaultValue="/configuration/" />
    <asp:ControlParameter Name="ListName" ControlID="ddlLists"
      PropertyName="SelectedValue"/>
</SelectParameters>

I seemed to have some control execution lifecycle fun with ControlParameter but it did work in the end. To pull the value from a 'userID' querystring parameter I would need:

<SelectParameters>
    <asp:QueryStringParameter Name="userId" QueryStringField="userId" />
</SelectParameters>

And that's not all - I can also drop a parameter value into a string such as a CAML query used in the earlier example of 'List' mode:

<SPWebControls:SPDataSource runat="server" ID="dsPersonTitles" DataSourceMode="List"
  SelectCommand="<Query><OrderBy><FieldRef Name='{SortField}' Ascending='{SortAsc}' /></OrderBy></Query>">
  <SelectParameters>
      <asp:Parameter Name="WebUrl" DefaultValue="/configuration/" />
      <asp:Parameter Name="ListName" DefaultValue="PersonTitles" />
      <asp:QueryStringParameter Name="SortField" QueryStringField="SortField" DefaultValue="SortOrder" />
      <asp:QueryStringParameter Name="SortAsc" QueryStringField="SortAsc" DefaultValue="true" />
  </SelectParameters>
</SPWebControls:SPDataSource> 

Notice here I'm dynamically specifying the sort field and direction from querystring parameters. Cool stuff.

Even cooler is the idea of building your own parameter control - one that strikes me is one for retrieving values from a user's SharePoint profile (since the ProfileParameter listed above refers to ASP.Net profiles). Scott Mitchell has a great guide to this at Accessing and Updating Data in ASP.NET 2.0: Creating Custom Parameter Controls.

Summary

So we've covered a lot of ground there, so here's a recap of why the SPDataSource is your friend:

• Can bind to any control which can do data-binding - DropdownList, ListBox, CheckBoxList are some obvious candidates, but consider also Repeater, DataGrid and SharePoint DataView/SPGridView controls


• Can easily get parameters from other controls, querystring, session, form values etc. (or write some code to fetch from another location)


• Can use a variety of modes to make different queries e.g. items in a list, properties of a list item, webs in a site etc.

If you want to see more I recommend reading SPDataSource and Rollups with the Data View - this emphasises using SPDataSource with SharePoint DataViews but also has some good examples I haven't covered, such as using the 'CrossList' mode.

SPDataSource - every SharePoint developer's friend (part 1)

Sooner or later, nearly every SharePoint developer needs to write code to retrieve all the items from a list and display them - either on the page, or often in a control like a dropdown list control. An example could be retrieving a list of countries, or perhaps person titles for a form:

Clearly there are benefits from storing such values in a list, since we (or the client) can then add/edit/delete items easily. To implement this, the developer's first thought might be to write code like this:

private void bindPersonTitles()
{
    // the name of the DropDownList control we are populating is ddlPersonTitles..
    using (SPWeb configWeb = SPContext.Current.Site.AllWebs["configuration"])
    {
        SPList titlesList = configWeb.Lists["PersonTitles"];

        foreach (SPListItem titleItem in titlesList.Items)
        {
            ddlPersonTitles.Items.Add(new ListItem(titleItem.Title, titleItem.ID.ToString()));
        }
    }
}

Which is fine. Except if we want to filter or sort the list items, we really should use a CAML query instead of iterating through all the items - the code below shows this (sorting on a column called 'SortOrder'), and also has a slight twist on the previous example in that I'm using data-binding rather than looping through the items 'manually':

private void bindPersonTitlesByCamlQuery()
{
    using (SPWeb configWeb = SPContext.Current.Site.AllWebs["configuration"])
    {
        SPList titlesList = configWeb.Lists["PersonTitles"];
        SPQuery query = new SPQuery();
        query.Query = "<OrderBy><FieldRef Name=\"SortOrder\" /></OrderBy>";
        SPListItemCollection titlesItems = titlesList.GetItems(query);

        // showing alternative of using data-binding rather iterating through items..
        ddlPersonTitles.DataSource = titlesItems;
        ddlPersonTitles.DataTextField = "Title";
        ddlPersonTitles.DataValueField = "ID";
        ddlPersonTitles.DataBind();
    }
}

Fine again. Except I can't help thinking "all this just to take the items from a list and put them in a dropdown??" Surely there must be a better way. And what did Microsoft do every time they needed to do this, surely they didn't repeat the above code in every place in SharePoint? The answer is no, they used this:

SPDataSource

In short, SPDataSource is a web control which implements IDataSource and saves you writing code like the above. The great thing is that it is extremely flexible, and as we'll see, can be used for more than you might think. The best thing though, is that being a control it can be used declaratively, so I can bind my dropdown to the list without writing a single line of C# or VB.Net code - all I have to do is set properties correctly. We'll go through the detail in a second, but the markup to replace the last code sample would look like:

<SPWebControls:SPDataSource runat="server" ID="dsPersonTitles" DataSourceMode="List" 
  SelectCommand="<Query><OrderBy><FieldRef Name='SortOrder' Ascending='true' /></OrderBy></Query>"
  <SelectParameters>
      <asp:Parameter Name="WebUrl" DefaultValue="/configuration/" />
      <asp:Parameter Name="ListName" DefaultValue="PersonTitles" />
  </SelectParameters>
</SPWebControls:SPDataSource> 

<asp:DropDownList runat="server" ID="ddlPersonTitles" CssClass="title" DataSourceID="dsPersonTitles" DataTextField="Title" DataValueField="ID">
</asp:DropDownList>

So we're doing the following:

• setting the 'DataSourceMode' for the SPDataSource is set to 'List' - we'll examine other possible values shortly


• setting the 'SelectCommand' to the CAML query we want to use - here we're just sorting on the 'SortOrder' field once more


• telling the SPDataSource which list we want to use by supplying ASP.Net 2.0 parameter objects named 'WebUrl' and 'ListName' - we'll dive more into this later


• finally we bind the dropdown to the data by specifying the DataSourceID to the ID we gave our SPDataSource, and also say which fields in the resultset we want to use for the 'Text' and 'Value' of the dropdown. Incidentally I recommend using the ID for the value and the Title for the text (as shown above) so that it's easy to create an SPLookupValue to update a list item - more on this in part 2

We're only just starting to see the power of SPDataSource, but I love this approach for a couple of reasons:

• Details of my query aren't specified in compiled code, so if anything about the list changes (e.g. we restructure our site) I don't have to recompile and redeploy assemblies


• Less custom code, less bugs!


• Setting properties is arguably simpler than writing code

So let's dive deeper into what we can do - we'll cover 'modes' of SPDataSource in this article and how to dynamically pass parameters to control the query in part 2.

The different 'modes' of SPDataSource

SPDataSource isn't just limited to fetching the items from a list (DataSourceMode = 'List'). Other possibilities are:

• CrossList - similar to doing a query with SPSiteDataQuery across all lists in a site collection (for a sample of this see the SharePoint Designer Team blog post linked at the end of this article)


• ListItem - show field values from a single list item


• Webs - lists all webs in a site collection


• ListOfLists - lists all lists in a web

For the last 2 modes I was able to bind but had some difficulty working out values to use for the 'DataTextField'/'DataValueField' of my control. I was trying to simply get the web or list name, but none of the obvious values such as 'Title', 'ListName' etc. were in the resultset. I was unable to find any other information on this but I'm sure some more trial and error would solve it. The 'ListItem' mode can be interesting - in the following sample I'm binding a single list item to a DataGrid to show selected fields from the item, which itself is selected by using the 'ListItemID' property:

<SPWebControls:SPDataSource runat="server" ID="dsPeople" DataSourceMode="ListItem" UseInternalName="true"> 
  <SelectParameters>
      <asp:Parameter Name="WebUrl" DefaultValue="/configuration/" />
      <asp:Parameter Name="ListID" DefaultValue="34F91B0C-FCF2-455A-ABBA-67724FB4024A" />
      <asp:Parameter Name="ListItemID" DefaultValue="1" />
  </SelectParameters>
</SPWebControls:SPDataSource> 

<asp:GridView ID="grdPeople" runat="server" DataSourceID="dsPeople" 
    AutoGenerateColumns="False">
    <Columns>
        <asp:BoundField DataField="FullName" HeaderText="Blogger name" />
        <asp:BoundField DataField="WorkCity" HeaderText="City" />
        <asp:BoundField DataField="Blog_x0020_URL" HeaderText="Blog URL" />
    </Columns>
</asp:GridView>

This would give something like this, based on an underlying list item which has these fields (no formatting yet applied):

I found I had to specify UseInternalName = "true" and use the ListID rather than ListName parameter in this mode.

Next time - passing parameters to SPDataSource

So far we've looked at supplying parameters by using a standard ASP.Net parameter control and specifying the value in the DefaultValue' property. In fact, ASP.Net has a whole range of parameter controls which can do the work of retrieving a parameter from somewhere and passing it to the SPDataSource, so that's what we'll look at next time. Additionally, since we're often using SPDataSource to bind data to form controls, we'll look at the common scenario of getting the selected item out of the form control to save back to a lookup field in SharePoint.

<updated>That link to the SPD blog article I mentioned but forgot to link to is http://blogs.msdn.com/sharepointdesigner/archive/2007/04/24/spdatasource-and-rollups-with-the-data-view.aspx - the focus here is mainly on using SPDataSource with a DataView, but there's some great info and samples.</updated>

Received MVP award for SharePoint!

Had notification from Microsoft today that I'm now a SharePoint MVP :-)

This is great news, I'm really thrilled to be recognized for the community stuff I've done over the past year, and personally I know I've been impressed by the work of so many existing SharePoint MVPs, so it's a big honor to be part of that group. I'm particularly grateful to the people who nominated me, thanks guys. It was weird having to look back to fill in the spreadsheet MS ask you to complete to be considered for MVP - I guess some of the key highlights for me were:

• building the SharePoint Content Deployment Wizard
• other Codeplex tools such as the SharePoint Config Store and my solution to deploying lookup fields as a Feature
• blogging in general - not sure how scientific the process was, but nice to appear in the top 25 (just!) of Joel Oleson's top 100 SharePoint blogs list
• the speaking gigs e.g. my VS workflow deep-dive

I guess I'll enjoy putting the MVP image on my blog etc., but the biggest reward is without doubt when someone leaves a positive comment or mentions in person that they've found the articles or tools useful - I'm never sure if people realize how much this is listened to. Anyway, I'm busy on the next version of the Wizard and some other articles, so you'll hear a lot more from me in the future.

Still a long way to go until I'm Andrew Connell or any number of the others though ;-)

P.S. Great to hear my mate Vince received it as well, definitely well deserved..

 

Content Deployment Wizard updated - Release 1.0

I've released a new version of the Content Deployment Wizard today, labelled as 'Release 1.0'. This release adds only minor changes to beta 2, and is mainly intended to pave the way for a future beta release which will contain new functionality. However I also want to communicate that the current codebase is now stable, since I heard feedback from some folks saying the 'beta' label was putting some people off (e.g. non-technical clients of SharePoint consultants).

That said, the following minor updates have been incorporated:

• ability to modify which lists are filtered out of the treeview:
  
  Certain 'system'-type lists are not shown in the treeview as they are generally not relevant to import/export operations, but on occasion you may want to 'unhide' some lists e.g. 'Variation Labels' and 'Relationships List' lists if you are using variations. See the readme.txt file for instructions on how to do this
  
  
• minor bug fixes e.g. .cmp file now correctly saved with extension if a period (.) is present in your chosen filename, validation to prevent child objects being added from treeview when parent site collection already added

The new version can be downloaded from www.codeplex.com/SPDeploymentWizard.

Future development

Work is underway on the next release of the Wizard, with the main emphasis being on:

• allowing import/export operations to be scripted
• allowing settings used in an operation to be saved to a file for easy reuse
• support for incremental deployments (hopefully!)

If you have other suggestions, I can't guarantee I can include them in the next release but I'd certainly be interested to hear.

Disposing SharePoint objects - what they don't tell you

If you're a SharePoint developer and find yourself regularly writing code to work with SPListItem objects, here's some info which might help you factor your code in a slightly neater way in some scenarios.  Interestingly this doesn't get mentioned in any of the key papers/articles - I can think of a good reason for this (which we'll look at), but in my mind this is a safe technique. If you have other ideas however, I'd love to hear from you.

Most developers are now fairly well-versed in the techniques needed to write efficient SharePoint code (key resources are listed at the end of the article), specifically in terms of disposing of any SPSite/SPWeb objects your code creates. The thing is, having to always be mindful of disposing these objects can sometimes restrict us in the way we want to write our code - consider the following simplified example:

public void DoSomething()
{
    SPList list = getList();

    // do something with list..
    foreach (SPListItem item in list.Items)
    {
        processItem(item);
    }

    // ** PROBLEM - how do we now dispose of the SPSite/SPWeb objects we created earlier? **
}

private SPList getList()
{
    // can't dispose of these objects here if we're returning a list - we'll be attempting to use 
    // objects which have already been disposed of..
    SPSite site = new SPSite("http://cob.blog.dev");
    SPWeb web = site.OpenWeb("/MyWeb");
    SPList list = web.Lists["MyList"];

    return list;
}

The problem here is we created our SPSite/SPWeb objects in a different method to where we need to dispose them - this can happen a lot when writing library code. To extend this example, here's a pattern I regularly use to ensure I'm getting my objects the most efficient way - using SPContext if it's present, but instantiating them myself if it isn't (i.e. the code is called from an event receiver, console app etc.):

public void DoSomething()
{
    bool bDisposalsRequired = false;

    // get list from SPContext if we have one..
    SPList list = getListFromContext();
    if (list == null)
    {
        // otherwise get list from objects we create..
        list = getInstantiatedList();
        bDisposalsRequired = true;
    }
    
    // do something with list..
    foreach (SPListItem item in list.Items)
    {
        processItem(item);
    }

    if (bDisposalsRequired)
    {
        // ** PROBLEM - how do we now dispose of the SPSite/SPWeb objects we created earlier? **
    }
}

private SPList getInstantiatedList()
{
    // can't dispose of these objects here if we're returning a list - we'll be attempting to use 
    // objects which have already been disposed of..
    SPSite site = new SPSite("http://cob.blog.dev");
    SPWeb web = site.OpenWeb("/MyWeb");
    SPList list = web.Lists["MyList"];

    return list;
}

private SPList getListFromContext()
{
    SPContext currentContext = SPContext.Current;
    SPList list = null;

    if (currentContext != null)
    {
        list = currentContext.Site.AllWebs["MyWeb"].Lists["MyList"];
    }

    return list;
}

When the code starts to become more complex like this, we start to really want a simple way of disposing the objects we created elsewhere. I'd probably caution against this, but what happens if you created the SPSite/SPWeb objects not only in another method, but another class? All of a sudden disposing of objects correctly is a lot more complex than the examples given in the white papers. 

Now, the savvy SharePoint developer might think "Hold on, the SPList has a ParentWeb object (and SPListItem has a ParentList object!), what happens if I go up the hierarchy and dispose that way?" - code to this would look like:

if (bDisposalsRequired)
{
    list.ParentWeb.Dispose();
    list.ParentWeb.Site.Dispose();
}

Now how would we know the correct objects have been disposed? Are these objects the same objects we were using earlier? Or has SharePoint populated these properties with references to different objects, meaning we've left our original objects alive and could run into scalability problems?

Well, the good news is these are the same objects, so disposing in this way will dispose of the objects you created. We can tell this by comparing the objects using System.Object's GetHashCode() method:

private void compareObjects()
{
    SPSite site = new SPSite("http://cob.blog.dev");
    SPWeb web = site.OpenWeb("/MyWeb");
    SPList list = web.Lists["MyList"];
    
    SPWeb parentWeb = list.ParentWeb;
    


    // no message displayed..
    Debug.Assert(parentWeb.GetHashCode() != web.GetHashCode(), "Objects are not same!");
    

    // create a *new* SPWeb object and compare it to our other reference..
    web = site.OpenWeb("/MyWeb");

    // message IS displayed..
    Debug.Assert(parentWeb.GetHashCode() != web.GetHashCode(), "Objects are not same!");
}

What we see from this is that the SPWeb object stored in the list.ParentWeb property has the same hash code as the original SPWeb object we created, but if we create a new instance of the same web, it has a different hash code. This indicates the objects in the first comparison are the same - so disposing of them through the ParentWeb reference will indeed dispose of the objects we created. So happy days - we've found we can factor our code in a way which may be more logical and readable.

The caveat

So why might this not be mentioned in any of the published guidance? Well, before using this technique throughout your code you should consider that in writing code in this way we are effectively relying on an internal implementation detail of the current SharePoint API. If Microsoft were to change this in a service pack (or the next version of SharePoint), our code may no longer dispose of objects correctly. I'd recommend giving this your own consideration (not least because you might conclude differently to me), but I'd suggest this change would be fairly unlikely. The reason for this is the SharePoint team themselves need to eliminate any unnecessary SPSite/SPWeb objects, so of course it makes sense that the the same SPWeb instance used to obtain the SPList is used to populate the ParentWeb variable. Creating an extra instance would make the API less efficient. On that basis, I personally am happy to write code in this way. If you think I've got this wrong (I can't guarantee I haven't), of course I'd love to hear from you.

In an upcoming article, I'll also discuss the idea of going beyond the MSDN examples, and trying to write SharePoint data access code in a more 'enterprise' or 'patterns' kind of way. Stay tuned for more on this.

Disposing SharePoint objects - required reading

• Best Practices: Using Disposable Windows SharePoint Services Objects


• Best Practices: Common Coding Issues When Using the Shareoint Object Model

Code/Feature samples for common SharePoint development tasks

There are lots of tools available to help developers, but I always think the single most useful thing is actually seeing a good example of the thing I'm trying to develop. This is why SharePoint blogs are so popular right? As well as checking my favorite blogs or past articles, I'm always going back to previous work - perhaps to see what values I used in a certain SharePoint file or to grab a chunk of code I wrote which I can amend to fit my current requirement. So today I want to point you in the direction of some useful samples which could be good reference material if you're a SharePoint developer - in actual fact I'm not releasing anything new here, but highlighting that my recent 'Config Store' solution provides a great "one-stop shop" of downloadable examples of several common SharePoint development tasks. Examples of the following dev tasks can all be found in the solution:

• Deploy site columns, content types as Feature
• Deploy a list (with default list items) as a Feature
• Using Feature receivers
• Using SPWebConfigModification to make web.config changes in code/as part of a Feature
• Using SPQuery to efficiently find items in a list
• Build a Solution package (.wsp) without WSP Builder
• Write a Solution deployment script to retract Solution, deactivate Feature, redeploy Solution, activate Feature etc.
• Using event receivers
• As a .Net bonus, how to use the HttpRuntime class to work with ASP.Net caching outside of a web context

The entire set of code/Feature files can be downloaded from Codeplex, making it a great reference project containing samples of all the tasks listed above. If this is useful to you, the list below details the file to open to find the code sample, and some notes which might help you to get to grips with the task:

Deploy site columns, content types as Feature:

File	Notes
ConfigStoreElements.xml	This is accomplished by use of the Field/ContentType Feature elements. Note these should be before any ListTemplate/ListInstance items in the same elements file - if using both sets of elements in different files, ensure the field/content type file is listed first in manifest.xml.

Deploy a list (with default list items) as a Feature:

File	Notes
ConfigStoreElements.xml	This is accomplished by use of the ListTemplate/ListInstance Feature elements.

Feature receivers:

File	Notes
ConfigStoreFeatureReceiver.cs	In my case, the Feature receiver makes modifications to web.config and programmatically adds list event receivers to a list.

web.config modifications

File	Notes
ConfigStoreFeatureReceiver.cs	This is accomplished by use of the SPWebConfigModification class. I add three entries to the web.config to support my solution.

 

Using SPQuery to find items in a list:

File	Notes
ConfigStore.cs	I have two methods in the ConfigStore class which perform queries in this way (also known as a CAML query).

 

Build a Solution package (.wsp) without WSPBuilder:

File	Notes
makecab.ddf/                                   manifest.xml	Whilst WSPBuilder offers a great automated way to build Solutions, occasionally there is a need to step outside the tool to build the package 'manually'. In my case, I had a problem where WSPBuilder was adding elements in the 'wrong' order to the generated manifest.xml file, and this was causing the Solution deployment to fail.

 

Solution deployment script to retract Solution, deactivate Feature etc:

File	Notes
COB.SharePoint.Utilities.ConfigStore_Install.bat	This is the script I've often used to deploy my Solutions. It has error checking at each stage so I can easily tell at what point the install fails for simpler problem diagnosis.

 

Using event receivers:

File	Notes
ConfigStoreListEventReceiver.cs	In the Config Store solution I use a list event receiver to add items to the cache when the list contents change. This code shows using the ItemAdded and ItemUpdated events on the list.

 

Using the HttpRuntime class to work with ASP.Net caching outside of ASP.Net:

File

Notes

ConfigStoreListEventReceiver.cs

Something to remember with list event receivers is that we actually don't have access to HttpContext.Current or SPContext.Current - both are null. If it wasn't for the HttpRuntime class, this would be a problem for my Config Store code since I wanted to add/remove items from the cache in the list event receiver (i.e. manage the cache as soon as list items are changed). Fortunately, HttpRuntime allows us access to the ASP.Net framework from any code location (e.g. console app).

 

The link to download all this stuff is at http://www.codeplex.com/SPConfigStore. Hope you find the samples useful, feel free to leave a comment if anything doesn't make sense.

Config Store FAQs

So last week I introduced my Config Store 'framework' on Codeplex, which provides a complete solution (including source code!) for storing config values in a SharePoint list, but skipped over some of the details, including the optimizations which I think make it so much better than a simple implementation of this concept. This post aims to shed a bit more light on what the Config Store is about, and hopefully provides some key information to help you work out whether it's something which would be useful in your projects.

What kind of configuration values can I put in the Config Store?
In short, any value you wish to avoid hardcoding into your SharePoint application's C#/VB.Net code. Values are stored in the list as strings, but if they are actually integers, booleans, DateTime values etc., you would just cast them as appropriate after retrieval. Note also that we can also store complex data as XML, and either deserialize into a class or otherwise process with standard XML methods.

How do I retrieve config values in code?
For a single value it's as simple as:

string sAdminEmail = ConfigStore.GetValue("MyApplication", "AdminEmail");

To take advantage of the optimization (described later) for retrieving multiple values in a single control/page/web part/whatever, use the ConfigStore.GetMultipleValues() method - see the Config Store introduction post or Codeplex site for details.

What are the main components?
The main pieces are:

• Infrastructure - the list, site columns, and content type


• Event receivers - responsible for managing the config cache


• Feature and Feature receiver - responsible for provisioning the infrastructure (Feature) and also adding web.config entries and hooking up event receivers (Feature receiver)


• Code - the main class is the ConfigStore class, this has the GetValue() and GetMultipleValues() methods.

How do I install it?
Simply run the install script which will do the work of installing the Solution (.wsp) to your site. Once that's complete, there are a couple of things to check got installed correctly - a checklist is included in the readme.txt in the download.

Performance is important on our site - how is the Config Store optimized?
There are several levels of optimization - taking these one at a time:

• all values are cached in memory, so in general retrieval is lightning fast and doesn't require any kind of database lookup.


• even the first ever request for a particular value will be lightning fast, since the value will already be in the cache. This is possible because of the event receiver on the list - when the admin added the config value to the list, we executed some code to proactively add the item to the memory cache at this point. The caveat here of course, is that IISResets/app pool recycles will still clear down the cache (since it is the ASP.Net memory cache), so this only holds for values added since the last reset/recycle. All that said, you could certainly add code to the global.asax file to cache all values automatically after a reset/recycle if you want ultimate performance.


• if a query is required, the retrieval code uses the current SPContext if one is available - this avoids creating an SPSite/SPWeb object unless absolutely necessary. Needless to say, any objects which are created are also disposed.


• the query is performed using a CAML query (SPQuery) as opposed to iterating through the whole config list.


• the GetMultipleValues() method allows the developer to do just that whilst ensuring there is only one underlying query, rather than one for each value retrieved. 

Developers familiar with caching will recognise that this alone provides a huge boost to performance.

Where can I retrieve values from in code?
It doesn't have to be a SharePoint page, control or web part. The framework is designed to work in other scenarios even when there is no SPContext available - examples are event receivers, InfoPath managed code or even outside the SharePoint web app (e.g. console app, custom STSADM command).

I work in a large SharePoint deployment and my admin doesn't want to deploy assemblies to the GAC or run under Full Trust. Can I still use the Config Store?
By default the Config Store Solution installs the assembly to the GAC. This provides access to the functionality from outside the SharePoint web app as described above. However, it is possible to run the assembly from your site's bin directory, though remember you will then need to define CAS policy in your web.config code as appropriate.

I've installed the Config Store using the supplied .wsp and install script, how can I quickly tell it's working?
The quickest way is to add the supplied 'ConfigTestControl' to one of your pages in SharePoint Designer. You'll need the Register directive to link the assembly, and also the control declaration:

<%@ Register Tagprefix="COB" Namespace="COB.SharePoint.Utilities" Assembly="COB.SharePoint.Utilities.ConfigStore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=23afbf06fd91fa64" %>
. <!-- normal page markup here -->
.
.
<COB:ConfigTestControl runat="server"></COB:ConfigTestControl>

How does the Config Store work in terms of permissions? Do I need to apply any permissions to the list to make it work?
No specific permissions need to be configured, unless you choose to lock down write access to the config list beyond the default - this is for the list to inherit the permissions of the root web of your site. This means  anyone with 'Contribute' permission in this web (for example) will also be able to work with items in the config list. If you do wish to restrict who can modify config values, simply add whatever security you require to the config list - the Config Store framework will still continue to work since the code retrieves values under an admin context, via use of SPSecurity.RunWithElevatedPrivileges().

Our site is an internet site, will the Config Store work with anonymous users?
Yes. Again, since values are actually retrieved under an admin context, it doesn't matter that your anonymous users do not have direct permissions to read from the list.

Anything else?
It's worth remembering that we can also take advantage of services offered by the list infrastructure when using the Config Store. Some ideas could be auditing ("we need to log any changes to our app's config"), alerts ("if something changes, I want to know about it!"), version history ("let's see what the setting was when the site was having problems") and workflow ("I want all changes to go through me for change management").

The Config Store framework can be downloaded from www.codeplex.com/SPConfigStore.

Introducing the SharePoint Config Store for developers

Today I want to introduce something I've been working on recently which could be of use to you if you're a SharePoint developer. Often when developing SharePoint solutions which require coding, the developer faces a decision about what to do with values he/she doesn't want to 'hardcode' into the C# or VB.Net code. Example values for a SharePoint site/application/control could be:

'AdministratorEmail' - 'bob@somewhere.com'
'SendWorkflowEmails' - 'true'

Generally we avoid hardcoding such values, since if the value needs to be changed we have to recompile the code, test and redeploy. So, alternatives to hardcoding which folks might consider could be:

• store values in appSettings section of web.config
• store values in different places, e.g. Feature properties, event handler registration data, etc.
• store values in a custom SQL table
• store values in a SharePoint list

Personally, although I like the facility to store complex custom config sections in web.config, I'm not a big fan of appSettings. If I need to change a value, I have to connect to the server using Remote Desktop and open and modify the file - if I'm in a server farm with multiple front-ends, I need to repeat this for each, and I'm also causing the next request to be slow because the app domain will unload to refresh the config. Going through the other options, the second isn't great because we'd need to be reactivating Features/event receiver registrations every time (even more hassle), though the third (using SQL) is probably fine, unless I want a front-end to make modifying the values easier, in which case we'd have some work to do.

So storing config values in a SharePoint list could be nice, and is the basis for my solution. Now I'd bet lots of people are doing this already - it's pretty quick to write some simple code to fetch the value, and this means we avoid the hardcoding problem. However, the Config Store 'framework' goes further than this - for one thing it's highly-optimized so you can be sure there is no negative performance impact, but there are also some bonuses in terms of where it can used from and the ease of deployment. So what I hope to show is that the Config Store takes things quite a bit further than just a simple implementation of 'retrieving config values from a list'.

Details (reproduced from the Codeplex site)

The list used to store config items looks like this (N.B. the items shown are my examples, you'd add your own):


There is a special content type associated with the list, so adding a new item is easy:



..and the content type is defined so that all the required fields are mandatory:



Retrieving values

Once a value has been added to the Config Store, it can be retrieved in code as follows (you'll also need to add a reference to the Config Store assembly and 'using' statement of course):

string sAdminEmail = ConfigStore.GetValue("MyApplication", "AdminEmail");

Note that there is also a method to retrieve multiple values with a single query. This avoids the need to perform multiple queries, so should be used for performance reasons if your control/page will retrieve many items from the Config Store - think of it as a best practise. The code is slightly more involved, but should make sense when you think it through. We create a generic List of 'ConfigIdentifiers' (a ConfigIdentifier specifies the category and name of the item e.g. 'MyApplication', 'AdminEmail') and pass it to the 'GetMultipleItems()' method:

List<ConfigIdentifier> configIds = new List<ConfigIdentifier>();

ConfigIdentifier adminEmail = new ConfigIdentifier("MyApplication", "AdminEmail");
ConfigIdentifier sendMails = new ConfigIdentifier("MyApplication", "SendWorkflowEmails");
configIds.Add(adminEmail);
configIds.Add(sendMails);

Dictionary<ConfigIdentifier, string> configItems = ConfigStore.GetMultipleValues(configIds);

string sAdminEmail = configItems[adminEmail];
string sSendMails = configItems[sendMails];

..the method returns a generic Dictionary containing the values, and we retrieve each one by passing the respective ConfigIdentifier we created earlier to the indexer.

Other notes

• All items are wrapped up in a Solution/Feature so there is no need to manually create site columns/content types/the Config Store list etc. There is also an install script for you to easily install the Solution.


• Config items are cached in memory, so where possible there won't even be a database lookup!


• The Config Store is also designed to operate where no SPContext is present e.g. a list event receiver. In this scenario, it will look for values in your SharePoint web application's web.config file to establish the URL for the site containing the Config Store (N.B. these web.config keys get automatically added when the Config Store is installed to your site). This also means it can be used outside your SharePoint application, e.g. a console app.


• The Config Store can be moved from it's default location of the root web for your site. For example sites I create usually have a hidden 'config' web, so I put the Config Store in here, along with other items. (To do this, create a new list (in whatever child web you want) from the 'Configuration Store list' template (added during the install), and modify the 'ConfigWebName'/'ConfigListName' keys which were added to your web.config to point to the new location. As an alternative if you already added 100 items which you don't want to recreate, you could use my other tool, the SharePoint Content Deployment Wizard at http://www.codeplex.com/SPDeploymentWizard to move the list.)


• All source code and Solution/Feature files are included, so if you want to change anything, you can


• Installation instructions are in the readme.txt in the download

Summary

Hopefully you might agree that the Config Store goes beyond a simple implementation of 'storing config values in a list'. For me, the key aspects are the caching and the fact that the entire solution is 'joined up', so it's easy to deploy quickly and reliably as a piece of functionality. Many organizations are probably at the stage where their SharePoint codebase is becoming mature and perhaps based on a foundation of a 'core API' - the Config Store could provide a useful piece in such a toolbox.

You can download the Config Store framework and all the source code from www.codeplex.com/SPConfigStore.