Tuesday 20 January 2009

A better Config Store for SharePoint sites

I've recently made some enhancements to my Config Store framework on Codeplex which I'm now ready to share. Many of these enhancements are a result of adapting the solution to a large project where we built a platform for 80-100 internet sites - hence it's now become a bit more 'enterprise'. With such things I generally make a deal with my employer where I do the work (or most of it) in my spare time and then get to share the code publicly, so here we go. Before I delve into the details, let's have a reminder of what the Config Store is all about:

Recap - the SharePoint Config Store in a nutshell

Regular readers may remember this is a solution which allows use of a SharePoint list to store configuration values used by your SharePoint application - the idea is that your webparts/server controls/page layouts etc. store any strings/data they need for configuration in here as a more flexible alternative to web.config or similar. Since our values are now in a SharePoint list, configuration can be updated across the farm through the browser to administrators who have the appropriate permissions. We can also optionally take advantage of all the other things lists give us such as auditing, item-level security, alerts and version history etc. Finally, a caching layer is used to avoid round trips when your code fetches configuration values.

On the last couple of projects where my team has used it, we've finished up with 100+ config items in the list - storing all sorts of things from URLs, strings and 'application behaviour' switches:

ConfigStore

If you need a more complete overview, see:

Enhancements in the new release

  1. Optional "hierarchical" configuration model similar to web.config

    In the first release, since all the config values are stored in one list this means your configuration is stored in one 'nominated' site collection. This is fine for WCM sites which may only use one site collection, but may not easily map to certain enterprise requirements. What's new in this release is that the framework can now use a 'hierarchical' model, where a Config Store list exists in whatever site collections should have one, but a 'master' site collection is nominated which contains the 'master' config values. What happens is that if the config item you request is in the 'local' Config Store you'll get that value, and if not you'll get the value from the master list. This allows local overriding of the parent values if required - in practice we found 95% of the config items would be stored in the master list only, but having the facility to override the other 5% was critical to supporting some of the functionality we developed.

    Needless to say, this is the implementation which best suited our requirements. It could be it doesn't really suit yours, but developers could consider starting with my source code and modifying, since other aspects such as the caching layer, Feature files, event handlers etc. might not need to be changed much.

    If you want to continue to just use one Config Store list (even if you consume the config values in multiple site collections), the new code will continue to work just fine for this model too.

  2. Easier to use in ASPX markup

    Similar to my recent Language Store framework, the Config Store now supports easily dropping values into ASPX markup by implementing an expression builder. So if all you want to do is set the Text property of a control, you can now do with this without cluttering up the code-behind. Or, as an alternative example, here we're retrieving an URL from the Config Store (stored in Category 'PageUrls' and key 'MyAccountPage') to assign to a hyperlink:
    <asp:HyperLink id="hyperlink1" NavigateUrl="<%$ SPConfigStore:PageUrls|MyAccountPage %>"
    Text="My account" ImageUrl="images/pict.jpg" runat="server"/>


  3. Fixed caching bug for farm environments

    Yes, something I have to hold my hands up to here - the caching layer in the initial release didn't adequately deal with multiple servers. The effect was that it required an app pool recycle to pick up changes to config values, rather than them taking effect immediately. Not the end of the world, but certainly inconvenient and taking away some of the benefits of using a SharePoint list for configuration. So in this release, the implementation correctly relies on a back-end resource (using a CacheDependency) to invalidate the cache across all servers in the farm. The implementation I chose was to add the items to the cache with a CacheDependency on a text file - this needs to be located on a path which all servers can access - and the event handler now updates a timestamp in the file to invalidate the item in the cache across all servers.

    I went with using a file as the dependency item as I thought it was more lightweight than using a SqlCacheDependency - I didn't really want to impose the SQL configuration etc. to be able to use the Config Store. However, with the file cache dependency I've chosen, be aware that some production configurations may have internal firewalls which prevent all WFEs from accessing a shared file in this way - check before you deploy. Of course the code is there for you to modify should you wish to make changes in this area.

  4. Amendment to Feature to prevent web.config modifications being made on Feature activation

    Another thing that got in the way when I tried to implement the Config Store on our enterprise project was the Feature event receiver which adds the required appSettings keys to web.config. The idea here is that, to help simplify installation and initial setup, the required web.config entries are added with empty values so all the developer has to do is plug in the appropriate values for his/her site. Sounds great - and it is for single site collection sites. But for multiple site collections, when we come to activate the Feature in the 2nd, 3rd, nth site collection - of course the receiver runs and adds the empty entries to web.config again, despite the fact that we'd already inserted the real values on the first activation. And since the new values come later in the file, guess which ones are used?

    So in this release there's a Feature property which determines if web.config modifications are made - it's set to 'False' by default but it's there if you prefer to change it.

  5. 'Config value' column is now bigger

    Previously this column was a 'single line of text' but it's now a 'note'. This means you can happily store HTML/XML fragments or other larger values, which is very useful in some scenarios. However, note that if you're already using the Config Store on your site and want to 'upgrade', this schema change is significant - I discuss it in the readme.txt file.

So that's it. You can download the updates from www.codeplex.com/SPConfigStore - hope you find it as useful as we have.

15 comments:

Anonymous said...

You might be interested in Tobias Lekman's implementation of an SPListItemCacheDependency:
http://www.lekman.com/2008/12/cache-dependencies-in-sharepoint.html

Chris O'Brien said...

Hi Keith,

Yes I did consider writing the exact same thing as Tobias - mainly from the work that Vince did that Tobias references. However, I'm somewhat uncomfortable with having a timer job polling every 5 or 10 seconds which has to instantiate an SPSite, then fetch some SharePoint item to check if it's changed. Also, I felt that one requirement with the Config Store was that changes take effect immediately - even 10 seconds would be pushing it, as in my experience there will always be admins/developers who are unaware/forget that there is a delay to the caching implementation.

So using .Net's CacheDependency feels slightly lower level and less likely to incur a performance hit.

Interesting conversation though. Would be interested if anybody has done some proper tests.

Cheers,

Chris.

Anonymous said...

Hey Chris,
Great Job!!!
I had a question, we have used the earlier version of Config Store and have referenced it in couple of our custom components, does this need to change? i.e. do we need to re-compile the custom components/code with the new Version of Config store OR just that we upgrade the Config store WSP and rest all should work fine?

Appreciate your help..

Thanks
MossBuddy

Anonymous said...

Hi Chris, love this project.
Quick question. Would you recommend, or have you used your Config Store to retrieve database connection strings?

Chris O'Brien said...

@MossBuddy,

Good question. There are a couple of options to upgrade which I outline in the readme.txt - suggest downloading and taking a proper read. In brief you can either:

- use the new assembly only e.g. your suggestion of recompiling your projects with the new version
- use the new assembly AND the new list schema (larger Config Value field). This is more involved since it's not really possible to upgrade a field 'in-place' - the readme.txt contains more details of the process you'd need to go through

Any problems, please drop me a line in the discussions area on the Codeplex site and I'll respond.

HTH,

Chris.

Chris O'Brien said...

@Anonymous,

Security is obviously the key consideration here. Some points:

- is your site a public-facing site or an intranet only visible within your network perimeter?
- are your connection strings using integrated security or do they specify a SQL username/password?
- what additional permissions have you applied to the SharePoint list?

Like many security-related things, it does depend on some of the other factors. It might be completely acceptable to store an encrypted (e.g. one-way) version of the string, but remember that the data is inherently 'more available' if it can be accessed through a browser by users with appropriate permissions rather than requiring remote access to the actual machine.

HTH,

Chris.

Anonymous said...

Thanks for that Chris.

Alternative to using your list is web.config the preferred option for storing connection strings?

Anonymous said...

Hi Chris,
Thanks for this cool project.


The list template for Config store appears when we want to create a new list, can we hide it.

Cheers
Ajay

Anonymous said...

Hey Chris in the latest readme.txt it says to add the web.config settings ConfigStoreSiteUrl, ConfigStoreWebName, ConfigStoreListName. Should they be ConfigSiteUrl, ConfigWebName, ConfigListName or are both needed?

Chris O'Brien said...

Yes, if you need real security the best idea is to:

- use integrated authentication so that your connection strings don't have any usernames/passwords in the first place
- encrypt the appropriate sections of your web.config - here's the documentation for how to encrypt with RSA

HTH,

Chris.

Chris O'Brien said...

@Ajay,

To do this, I think you'd need to set the 'Hidden' attribute on the ListTemplate element in elements.xml to 'True' and reactivate the Feature.

HTH,

Chris.

Chris O'Brien said...

@djeeg,

Well spotted. You're right, it's ConfigSiteUrl, ConfigWebName, ConfigListName so it was a typo in the readme. Now fixed on Codeplex.

Apologies for the confusion!

Thanks,

Chris.

George Durzi said...

hey Chris,
After installing the solution and browsing to the Confog Store list, the add/edit view of any of the ListItems only shows the Config name field, not all the other fields as shown in the screenshots.

I verified that all the site columns, etc., were deployed.

Any tips on how I can troubleshoot this?

Chris O'Brien said...

Hi George,

Really sorry about this - this is a 'bug' (somewhere in the Feature XML I think) which another user has reported to me just today. The fields are there and can store data, it's just some corruption of the content type which means the form doesn't show all the fields.

I've seen it myself, but my team and I figured it was because we migrated from the version 1.0.0.0 in the middle of our project and our schema wasn't quite clean. (The cause is something to do with the change from 'Single line of text' to 'Note' for the Config Value field).

Unfortunately we didn't solve the root problem, but we wrote a few lines of code which 'fixes up' the issue and makes everything work again. However, we also saw it come back if you use Content Deployment (e.g. my Wizard tool) to deploy the list - when this happened we just ran our method on that environment to 'fix up' again - no data is lost when the command is run, it's just inconvenient to have to do it.

If you leave me another comment with your e-mail (which I won't publish), I'll forward the code to you. Same goes for anyone else affected.

Really sorry for the inconvenience. As soon as I've got my conference talk out of the way next week, my priority is fixing this. If anyone finds the issue in the meantime, I owe you beer!

Let me know if you want the fix-up code.

Cheers,

Chris.

Chris O'Brien said...

All,

The issue mentioned in the previous couple of comments was resolved in Release 2.1.0.0 now available on the Codeplex site.

Cheers,

Chris.