Thursday, 23 April 2009

Fix to my Config Store framework and list provisioning tips

Had a couple of reports recently of an issue with my Config Store solution, which provides a framework for using a SharePoint list to store configuration values. If you're using the Config Store this article will definitely be of interest to you, but I've also picked up a couple of general tips on list provisioning which I want to pass on. I have to thank Richard Browne (no blog) of my old company cScape, as the fix and several of the tips have come from him - as well as alerting me to the problem, he also managed to fix it before I did, so many thanks and much kudos mate :-)

Config Store problem

Under some circumstances, fields in the Config Store list were not editable because they no longer appeared on the list edit form (EditForm.aspx). So instead of having 4 editable fields, only the 'Config name' field shows in the form:

ConfigStoreMissingFields

I've not fully worked out the pattern, but I think the problem may only appear if you provision the list on a server which has the October or December Cumulative Update installed - either that or it's a difference between Windows 2003 and Windows 2008 environments (which would be even more bizarre). Either way, it seems something changed in the way the provisioning XML was handled somewhere. This is why the problem was undetected in the earlier releases.

I had seen this problem before - but only when the list was moved using Content Deployment (e.g. using the Content Deployment Wizard) - the original 'source' list was always fine. We managed to work around this by writing some code which 're-added' the fields to the list from the content type, since they were always actually present on the content type and the data was still corrected stored. Having to run this code every time we deployed the list was an irritation rather than critical, but something I wanted to get to the bottom of - however, on finding some folks were running into this in 'normal' use meant that it became a bigger issue.

The cause

I always knew the problem would be down to a mistake in the provisioning XML, but since I'd looked for it on previous occasions I knew it was something I was seeing but not seeing. In my case, Richard spotted that I was using the wrong value in my FieldRef elements under the ContentType element - I was mistakenly thinking that the 'Name' attribute needed to match up with the ''StaticName' attribute given to the field; the documentation says this attribute contains the internal name of the field. So my FieldRefs looked like this:

<ContentType ID="0x0100E3438B2389F84cc3965600BC16BF32E7" Name="Config item" 
Group="Config Store content types" Description="Represents an item in the config store." Version="0">
<FieldRefs>
<FieldRef ID="{33F5C8B4-A6BB-41a4-AB24-69F2152974C5}" Name="ConfigCategory" Required="TRUE" />
<FieldRef ID="{BD413479-48AB-41f5-8040-918F32EBBCC5}" Name="ConfigValue" Required="TRUE" />
<FieldRef ID="{84D42C64-D0BD-4c76-8ED3-0A9E0D261111}" Name="ConfigItemDescription" />
</FieldRefs>
</ContentType>

..to match up with fields which looked like this:

<Field ID="{33F5C8B4-A6BB-41a4-AB24-69F2152974C5}"
Name="Config category"
DisplayName="Config category"

StaticName="ConfigCategory"
....
....
/>

The CORRECTED version looks like this (note the change in value for the Name attribute of FieldRefs):


<ContentType ID="0x0100E3438B2389F84cc3965600BC16BF32E7" Name="Config item"
Group="Config Store content types" Description="Represents an item in the config store." Version="0">
<FieldRefs>
<FieldRef ID="{33F5C8B4-A6BB-41a4-AB24-69F2152974C5}" Name="Config category" Required="TRUE" />
<FieldRef ID="{BD413479-48AB-41f5-8040-918F32EBBCC5}" Name="Config value" Required="TRUE" />
<FieldRef ID="{84D42C64-D0BD-4c76-8ED3-0A9E0D261111}" Name="Config item description" />
</FieldRefs>
</ContentType>

So, the main learning I got from this is to remember that the 'Name' of the FieldRef attribute needs to match the 'Name' of the Field attribute - that simple. Why did it work before? No idea unfortunately.

However, I also picked up a few more things I didn't know about, partly from Richard (this guy needs a blog!) and partly from some other reading/experimenting..

Some handy things to know about list provisioning

  • To make a field mandatory on a list, the 'Required' attribute must be 'TRUE'. Not 'True' or 'true' - this is one of the cases where the provisioning framework is pernickety about that 6-choice boolean ;-)
  • FieldRefs need an ID and Name as a minimum (which must match the values in the 'Field' declaration), but you can override certain other things here like the DisplayName - this mirrors what is possible in the UI.
  • You don't have to include the list .aspx files (DispForm.aspx, EditForm.aspx and NewForm.aspx) in your Feature if you use the 'SetupPath' attribute in the 'Form' element in schema.xml (assuming you don't need to associate custom list forms).
  • You can use the 'ContentTypeRef' element to associate your content type with the list (specify just content type ID), rather than using the 'ContentType' element which needs to redeclare all the FieldRefs.
  • It's safe to remove all the default 'system' fields from the 'Fields' section of schema.xml

Going further than these tips, the best thing I found on this is Oskar Austegard's MOSS: The dreaded schema.xml which shows how you can strip a ton of stuff out of schema.xml. I've not tried it yet, but I'm sure that will be my starting point for the next list I provision declaratively. If you're interested in the nuts and bolts of list provisioning, I highly recommend you read it.

Happy XML'ing..

Tuesday, 14 April 2009

Slide deck from my deployment talk at Best Practices Conference

Had a great time presenting at the European SharePoint Best Practices Conference last week. I've been trying to put my finger on what made it such a good conference and I'm actually not sure, but I notice that other speakers and attendees have also been full of praise, so it's not just me. The event itself was extremely well-organized with excellent content, and Steve Smith and his team did a great job of looking after us speakers.

Highlights for me on the dev track were sessions from AC, Todd Bleeker, Eric (or "Uncle Eric" as I like to think of him, with his wise words on high-performance coding :-)) and Andrew Woody, but whenever I did stray from developer content I seemed to run into a great session like Mike Watson's on SQL Server in relation to SharePoint. Similarly I heard good things about speakers like Dan McPherson doing innovative sessions on the Information Worker track which I was disappointed to miss. [UPDATE: Here's a gratuitous shot of me in my session:]

COB_BestPracticesTalk_2

Another highlight was being on the two dev panel sessions we did, and having an interesting debate in one of them with Todd on approaches for provisioning - declarative (Features) vs. programmatic (code/PowerShell etc.). This was probably a good lead-in to my talk the next day, and some folks came up to say they really liked this conversation and that we covered it from angles they hadn't considered, which was good to hear. [UPDATE: Photo below of the second session, chaired by AC and with (from left to right) Todd Bleeker, Stacy Draper, Maurice Prather, Andrew Woodward, Ben Robb, Brett Lonsdale, me (with the mic) and Eric Shupps:]

DevPanel2

So all in all, a top conference, and fantastic to catch up with so many friends. Here's the link for my deck:

Slide deck - Approaches and best practices for deploying SharePoint sites through multiple environments (dev, QA, UAT, production)

SBP

Thursday, 26 March 2009

Command-line support for Content Deployment Wizard now available

I'm pleased to announce I've now completed initial development on the next version of the Content Deployment Wizard - this is a beta release for the next few weeks so if you need it "just work", you should continue to use the previous version (1.1), but I'm hoping some people out there are happy to test this beta. The tool has become fairly popular as a 'handy tool to have in the SharePoint toolbox', and hopefully this release extends it's usefulness significantly for some scenarios. If you're not familiar with the tool, it provides a way to import/export site collections, webs, lists, and files or list items, either between farms or between different sites in the same farm - the Codeplex site has more details. As previously mentioned, the key new additional functionality in this release is:

  • Command-line support
  • Support for saving of import/export settings to a file (in the Windows Forms app) for later re-use
  • An installer

Having command-line support for the Wizard means that it can now be used in an automated way. Some key scenarios I think this might be useful in are:

  • Continuous integration/automated builds - if your site relies on SharePoint content, you can now move 'real' data as part of a build process, copying selected content from 'dev' to 'build' or 'test' for example. I often see static data (perhaps from an XML file or Excel spreadsheet) used in this way in nAnt/CruiseControl/MSBuild scripts, but for frequently changing data (config values, lookup lists etc.), this doesn't work so well as there is always a static file to maintain separately. 
  • Deployment scripts - if you have deployment scripts to 'bootstrap' a website on developer machines, again pulling real data from a central 'repository site' can help here.
  • As part of a production 'Content Deployment strategy' - since out-of-the-box Content Deployment is restricted to deploying a web as the smallest item, the Wizard could be used to deploy selected lists/list items/files

Obviously you might have your own ideas about where it could slot into your processes too.

How it works

  1. First, we select the content to move as we would normally using the Wizard..

    SelectExportItems
  2. ..and select the options we want to use for this export..

    SelectExportSettings 

  3. On the final screen, the new 'Save settings..' button should be used to save your selections to an XML file: 

    SaveSettingsButton  
    This will then give you an XML file which looks like this:
  4. <ExportSettings SiteUrl="http://cob.publish.dev" ExcludeDependencies="False" ExportMethod="ExportAll" 
                    IncludeVersions="LastMajor" IncludeSecurity="None" FileLocation="C:\Exports" 
                    BaseFileName="BlogSubwebAndPageTemplates.cmp">
      <ExportObjects>
        <DeploymentObject Id="b0fd667b-5b5e-41ba-827e-5d78b9a150ac" Title="Blog" Url="http://cob.publish.dev/Blog" Type="Web" IncludeDescendants="All" />
        <DeploymentObject Id="cfcc048e-c516-43b2-b5bf-3fb37cd561be" Title="http://cob.publish.dev/_catalogs/masterpage/COB.master" Url="_catalogs/masterpage/COB.master" Type="File" IncludeDescendants="None" />
        <DeploymentObject Id="670c1fb3-12f3-418b-b854-751ba80da917" Title="http://cob.publish.dev/_catalogs/masterpage/COBLayoutSimple.aspx" Url="_catalogs/masterpage/COBLayoutSimple.aspx" Type="File" IncludeDescendants="None" />
      </ExportObjects>
    </ExportSettings>

  5. So we now have an XML 'Wizard deployment settings file' which has the IDs of the objects we selected and the export options. We'll go ahead and show how this can be used at the command-line, but note also these settings can also be loaded into the Wizard UI on future deployments to save having to make the selections again - the key is the 'Load settings..' button on the first page (which we didn't show earlier):

    LoadSettingsButton 

  6. For command-line use of the Wizard a custom STSADM command is used. We pass the settings file in using the -settingsFile switch. To run the export operation we showed above, our command would look like:
    stsadm -o RunWizardExport -settingsFile "C:\DeploymentSettings\ExportBlogSubwebAndTemplates.xml" -quiet
    The -quiet parameter is optional, and suppresses some of the progress messages which are returned during the operation.

  7. For an import operation, we follow the same process - go through the Wizard and select the settings for the import operation, then click 'Save settings..' at the end to get the file (N.B. note the 'Import settings' screen has been simplified slightly from previous versions):

    SelectImportSettings
  8. The command to import looks like this:
    stsadm -o RunWizardImport -settingsFile "C:\DeploymentSettings\ImportBlogSubwebAndTemplates.xml" -quiet
    So that's both sides of it.

Using it for real

In real use of course, you may be deploying from one SharePoint farm to another. In this case, you also need to deal with copying the .cmp file from the source environment to the target if you're going across farms - if you have network access between farms (e.g. you're using it internally for automated builds/CI), a simple XCOPY in your scripts is the recommended way to do this. For production Content Deployment scenarios with no network connectivity, what I'm providing here will need to be supplemented with something else which will deal with the file transport. Clearly something web service based could be the answer.

Summary

Using the Wizard at the command-line may prove extremely useful if you need to move any SharePoint content regularly in an automated way. In contrast with other ways you might approach this, the XML definition file allows you to choose any number of webs/lists/list items/files to move in one operation, which may suit your needs better than shipping items around separately.

This is very much a beta release, but as a sidenote I'm expecting the initial issues to mainly be around the installer rather than core code - hence I'm providing a 'manual' install procedure which will get you past any such issues (see the readme). Needless to say, all the source code is also there for you on Codeplex if you're a developer happy to get your hands dirty. As I say, I'm hoping a couple of friendly testers will try it out and help me iron out the wrinkles - please submit any issues to the Codeplex site linked to below.

You can download the 2.0 beta release of the Wizard (and source code) from:

Monday, 9 March 2009

Update on next version of Content Deployment Wizard

Generally I only ever talk about SharePoint tools I'm working on once they're 100% complete and ready for use, but recently I had a conversation with someone at a user group which made me think about a policy change. Regular readers will know the main tool I'm associated with is the SharePoint Content Deployment Wizard which has become fairly popular (over 7000 downloads) - occasionally I've mentioned that one goal was to implement a command-line version, since this opens up all sorts of deployment possibilities. However I've not talked about this for a while, and just recently I've spoken to a couple of people who assumed I dropped this/didn't have the time to look at it, so here I am to tell you this is not the case!

For anybody that cares, the good news is I've actually been working on this since December interspersed with blogging, and am nearly done. The yicky refactoring work is complete, and I got chance to write the custom STSADM command on the front of it on the flight to the MVP summit last week. I need to do more testing first, but I'm hoping to release a beta to Codeplex over the next couple of weeks - if you're interested in the idea of scripted deployment of specific sites/webs/lists/list items between sites or farms (remember MOSS Content Deployment only does sites/webs and requires HTTP(S) connectivity), I'm hoping some friendly beta testers will help me screw the last bits down. The key aspects of this release are:

  • Command-line support
  • Support for saving of import/export settings to a file (in the Windows Forms app) for later re-use

Shortly after this release, I'm hoping to add support for incremental deployments (so only the content which has actually changed in the sites/webs/lists/you select will be deployed), but that's not going to make into this next cut unfortunately.

Keep tuned for further updates :-)

Other stuff

Whilst I'm at it, other things in the pipeline from me include:

Needless to say, there are plenty of other blog articles on my 'ideas list' too.

Sidenote - reflecting on 2 years of SharePoint blogging

Bizarrely, I'm into my 3rd year of SharePoint blogging now. I've no idea how this happened. Having done some interesting work with SharePoint's Feature framework, the initial idea was to write 4 or 5 articles I had material for - as a record for myself more than anything - and be done with it. Since then, although I do write the odd 'easy' post (like this one), generally my articles seem to take a long time to get completed, but I know they could be better. Occasionally I get reminded of this! So there's a long way to go for me to become a better blogger, but I'm fully hoping to still be at it in another 2 years time - and I'll have plenty more to say when the next version of SharePoint approaches :-)

Tuesday, 24 February 2009

UK user group meeting in London this Thursday, with Q & A panel

Just a quick note to remind UK-based folks within reach that there is a UK SharePoint user group meeting in London this Thursday. There are two sessions, one of which is an open Q & A for you to bring your trickiest SharePoint questions - I'll be amongst those on the panel representing the developer side of the house, but the line-up will cover all the bases. Needless to say, if you don't get chance to ask your question during the main session, there'll probably be ample opportunity in the pub afterwards. Michael Noel's session also looks extremely interesting, with a whole host of architecture/infrastructure knowledge condensed into one easily-digestible chunk.

Details from the suguk.org - to sign-up, use the link at the bottom of this post:

Session 1 - Building the Perfect SharePoint Farm: A Walkthrough of Best Practices from the Field - Michael Noel (see books written by Michael)

SharePoint 2007 has proven to be a technology that is remarkably easy to get running out of the box. On the flipside, however, some of the advanced configuration options with SharePoint are notoriously difficult to setup and configure, and a great deal of confusion exists regarding SharePoint best practice design, deployment, disaster recovery, and maintenance. This session covers best practices encompassing the most commonly asked questions regarding SharePoint infrastructure and design, and includes a broad range of critical but often overlooked items to consider when architecting a SharePoint environment. In short, all of the specifics required to build the 'perfect' SharePoint farm are presented through discussion of real-world SharePoint designs of all sizes.
• Learn from previous real world deployments and avoid common mistakes.
• Plan a checklist for architecture of SharePoint environments of any size.
• Build the 'perfect' SharePoint farm for your organization.

Session 2 - SharePoint Q & A Session

Following the session from last year we thought it would be a good idea to have a session where you can bring your SharePoint problems and hassles to and we can debate them as a group. We'll have a whiteboard, a laptop, and lots of clever people to discuss your questions and issues - so bring along your best and toughest!

The meeting is hosted at Microsoft in Victoria - arrive 6pm for a 6:30pm start:

Microsoft London (Cardinal Place)
100 Victoria Street
London SW1E 5JL
Tel: 0870 60 10 100

To register, simply reply to this thread leaving your full name - http://suguk.org/forums/thread/16904.aspx

Look forward to your questions :-)

Tuesday, 10 February 2009

Extending the web part framework - part 2

In part 1, I showed how we implemented a 'toolbox' of page templates and functionality modules wrapped up in a governance framework, to fulfil our client's requirement of a flexible WCM platform for building 80-100 internet sites with varying requirements. In this post, I want to detail some of the issues we ran into and the resolutions we found, focusing primarily on the 'module framework' we developed which is heavily-oriented around SharePoint web parts. 

Quick recap

The client is a large multi-national enterprise, and the idea is that content authoring teams in 80-100 countries will take what we've delivered on MOSS to create their country's internet presence e.g. .com, .co.uk, .fr, .es etc., replacing the existing mish-mash of sites on different technologies with inconsistent branding/look and feel.

In terms of the module framework, the cornerstones of our implementation were (see part 1 for more complete details on these):

  1. Module matrix - rules for which module can be used where, to guide authors away from building a user experience which doesn't  'make sense'
  2. SmartPart-like approach, but with web part properties - web parts wrapping user controls but also supporting web part properties exposed in custom tool parts
  3. Base web part/base tool part class - responsible for 'framework' behaviour such as checking if the current web part can be added (according to the module matrix)
  4. Combine interface of publishing field controls with web part storage - since publishing field controls (e.g. RichHtmlField) must be added in a 'static' manner at design-time but our authors can add controls dynamically at run-time, we developed custom controls which combine the rich functionality of the publishing HTML editor with web part storage
  5. Control adapter for WebPartZone for accessibility compliance - to get round the problem of all the HTML tables generated by SharePoint's web part framework, which will prevent a site validating for AA
  6. Present only our web parts in the web part picker - since standard SharePoint web parts are not used anywhere in these sites
  7. Remove unnecessary options when editing web part properties (tool parts) - to avoid confusing the authors

Issues and resolutions

I think that many of the challenges we faced are worth sharing as they came about through general web part development, rather than anything specific to what we did. Before I detail the actual gotchas, take note of some key development characteristics of our project:

  • Solutions and features used to deploy artifacts such as page layouts, content types etc.
  • Kivati Studio used for some other deployment aspects
  • Main functionality implemented in user controls - web parts were effectively thin wrappers around the .ascx files using LoadControl()
  • Web parts which are 'mandatory' are added to pages using the AllUsersWebPart element in a feature (though as the points probably illustrate, we looked at numerous ways of dealing with this)

Finding #1 - web parts outside of web part zones cannot be edited

The reason we wanted to have web parts outside of zones (perfectly possible by dragging a web part directly into page layout markup in SharePoint Designer) is for 'fixed' page modules which could not be removed by the content author. When we placed web parts outside of web part zones, we found the web parts would run fine in presentation mode but unfortunately cannot be edited (e.g. to edit web part properties) - the edit menu for the web part simply does not appear. I speculate this is because it is web part zones which are linked to web part storage, and thus web part properties cannot be persisted without a zone (the values in the markup will always be used). Hence, if you want editable web parts, you need web part zones.

Resolution - ensure all web parts (even ones which cannot be removed) live in a web part zone.

Finding #2 - embedding web parts into user control markup appears to be problematic

We tested various permutations of using web parts in/out of web part zones, and also with the HTML markup directly in the page layout .aspx or in a child .ascx file. After establishing that web part zones were required, we also found that whether the markup was in the .aspx or .ascx appeared to make a difference. This was unexpected, but the net effect seems to be that if you insert the web part markup into a web part zone which is in a user control rather than directly in the page layout .aspx (i.e. by refactoring the HTML markup for the web part zone and it's contents into a user control), again the edit menu will not display. I'm not sure why this is, but it could be related to the page execution lifecycle.

Resolution - accept that if web part zones will have web parts added to them at design-time by markup, the web part zone declaration cannot be in a user control.

Finding #3 - when using AllUsersWebPart element, duplicate web parts appear if the feature containing your page layouts is reactivated

Having decided our 'fixed' web parts would be added to pages using the AllUsersWebPart feature element (N.B. using this approach, 'default' web parts are associated with page layouts in the feature which deploys them. Web part zones are left empty on the page layout, and SharePoint provisions the web part into the zone at the time of creating a page from the layout). The issue we had with this is that all the web parts in all the zones in existing pages would be duplicated if the page layout feature was reactivated - this is because this XML is used both when the feature is activated (in the same way as say, provisioning for content types happens on activation) but also when new pages are created from a page layout.

Resolution - write a script (a Kivati task in our case) to remove duplicate web parts across all sites

[UPDATE - Waldek has an elegant solution to this problem in 'Preventing provisioning duplicate Web Part instances on Feature reactivation', as well as sample code similar to what we wrote for our script. DOH!]

Finding #4 - duplicate web parts can also appear when the page layout is customized (ghosted)

I'm not exactly clear on the reasons why customized files would ever cause duplicate web parts to appear, but that's certainly what we seemed to find. What happened is that we would deploy our master pages/page layouts using a feature to our QA environment, but immediately these files would be provisioned in that site as customized (i.e. the content in the content database), instead of being uncustomized and referenced on the filesystem. After further investigation, we traced the cause of this unexpected behaviour to the use of these attributes SPD adds to page layouts:

meta:progid=”SharePoint.WebPartPage.Document” meta:webpartpageexpansion=”full”

Resolution - ensure the version of the file does not contain these attributes. We actually switched to running uncustomized master page/page layouts even in our development farm. This means that we deployed the files using a feature and thereafter never opened them in SPD (editing only the source-controlled feature file instead).

Finding #5 - avoid setting default properties in the web part definition file (.webpart)

A final lesson we learnt is that, when working with web parts it's often better to avoid using the .webpart definition file extensively for setting default property values. There's nothing wrong with the mechanism - effectively these values are read whenever the web part is provisioned on a page, and your instance will set it's properties to these values. The problem, of course, is when you realize a property value you defined in the .webpart file needs to be updated because something changed. What happens to all the existing instances on pages around your site? As you might guess, the answer is nothing - unless you take steps to update those also, which generally means writing some kind of script to use SPLimitedWebPartManager. This can be pretty inconvenient when all you wanted to do was quickly change a default value.

Resolution - consider ensuring .webpart files are stripped to the bare minimum (assembly name etc.) and configuration comes from somewhere else. We typically rolled these config items into our use of the Config Store.


Summary

We ran into a few unexpected gotchas when building on the web part framework, but steps can be taken to minimise their impact. Hope you find these useful if you do web part development. Special thanks to Karoly Szalkary for helping to refresh my memory on some of these!


P.S. After 2 years writing about it, I've decided I no longer need to capitalize the 'f' in 'feature' - I think we're all on the same page on that one now ;-)

Saturday, 31 January 2009

Extending the web part framework - part 1

Today I want to show some of the interesting things we've been doing with web parts for one of our clients. There's quite a lot to talk about so it will be over two articles:

  • Part 1 - background and implementation
  • Part 2 - issues and resolutions

There are a couple of things in particular which I think are quite cool, as we've effectively combined classic WCM (publishing) site functionality with a customized implementation of the web part framework. The context is a fairly large roll-out to an enterprise client, but what we're rolling out is a centralized platform for 80-100 internet sites. The idea is that content authoring teams in 80-100 countries will take what we've delivered on MOSS to create their own sites - replacing the existing mish-mash of sites on different technologies with inconsistent branding/look and feel.

Clearly a key challenge here is satisfying the diverse needs of so many stakeholders. So a cornerstone of the platform is that sites can be tailored somewhat, so each country has some flexibility to communicate with their audience in the way they think is best. We effectively give the authors a set of page templates and building blocks, and a system which governs how the blocks can fit together so that the user experience will still 'make sense'. Needless to say, a lot of analysis and consideration has gone into this - both in terms of what functionality was needed but also user journeys and navigation through the site, and the experience architects on our side (LBi) played a vital role here. There are many aspects to the project I could zone in on, but since I want to focus on the implementation details here, I'll briefly list some of these building block requirements before showing how we did it.

Key requirements/challenges:

In order to create the different page types, we needed around 15 page layouts, including these:

  • Home page
  • Channel hub/Alternative hub/Sub home - these are different template options for '2nd and 3rd level' pages 
  • Content page
  • Product page
  • Media release
  • List - provides links to a series of related pages
  • Etc.

And whilst some aspects of page functionality was 'fixed' on the template, there were many other items which were optional - these were to be added to pages by the authors, either in a 'web part' kind of way or perhaps something else. Some examples of these optional 'page modules' were:

  • 'Hero' feature - used to highlight something on prominent pages with an image/flash/text
  • Right-hand promo
  • Content editor module - allows an author to enter arbitrary content, but for reasons which will become clearer we developed an interesting custom control which is kind of a cross between a publishing HtmlField and a Content Editor web part (covered later)
  • Generic content module - rolls-up formatted content/links to a selected page
  • List/tabbed list - provides links to a series of related pages
  • Dynamic share price - displays latest stock price based on web service call
  • Product selector - using AJAX cascading dropdowns to filter products
  • Etc.

Although there were lots of other challenges (such as multi-lingual content, packaging/documenting every deployment aspect so the hosting company could deploy etc.!), I felt that building the 'framework' could be more challenging than individual functionality bits. To help frame what you'll read next, some initial questions we had for the implementation were:

  • How do these optional bits of functionality get added to the page? As web parts, or something else?
  • How do we get accessibility-compliance if web parts?
  • How do we provide configuration if not web parts?
  • How do we restrict which modules can be used where (as per the specification)?
  • Since we're in a 'flexible' publishing site, how do we determine which fields are needed on the content types? Does each content type need to have all the possible fields the author might choose to add?
  • If we are working with publishing controls, how would we bind the dynamically added control to the 'back-end' publishing field on the content type?

The implementation

As well as the optional page modules, most of the templates had a classic set of publishing fields. After looking at custom approaches, we concluded the web part framework had a lot going for it for the optional stuff - clearly we could avoid building a user interface to pick the module from a list/add to page/allow configuration of properties specific to the module, and also get drag and drop (amongst other things) as an added bonus. The concept of web part zones - as a container where one or more modules could be placed - was also important to our page structure.

Another challenge for the optional modules was where to store the data. If they were publishing fields, we would need every possible module to have a corresponding field on every possible content type, and this was pretty impractical when looking at the spec. Web parts, of course, use a different model and the framework takes care of data storage regardless of how many controls are on the page.

On the downside, a key thing to remember with web parts in publishing pages is that web part data is (by definition) not stored in publishing fields, and therefore isn't versioned in the same way. After discussing with the client, in our case this proved to not have as big an impact as we initially thought, due to the split and nature of what content would be stored in publishing fields vs. what would be web parts. So, having the client's acceptance of this trade-off, we went with web parts and came up with these solution elements:

  1. Module matrix

    This comprised two SharePoint lists which contained the 'rules matrix', to enforce the design team's specification of what functionality could be used on which page type. Effectively the data provides the mapping of modules and page layouts. Being list data, it meant that it could be easily updated by the central team if a policy change was required. This data was consumed by our base web part (point 4).

  2. SmartPart-like approach, but with web part properties

    We wanted the actual functionality of our web parts to be implemented in user controls, for the typical reason of avoiding building HTML in C# code (wrong on so many levels!). This is obviously what the SmartPart does using LoadControl(), but we had the additional requirement of needing to pass web part property values to our user controls - this meant we could use the familiar 'tool part' interface (i.e. setting web part properties in the right-hand pane) for control configuration. 

    In our model, each user control has a corresponding 'wrapper' web part/tool part which understands which properties are required and how to build the properties UI. In the web part's OnInit() method, values are passed from the web part properties to the user control so that the latter is initialized ready to do it's processing.

  3. Base web part/base tool part

    All our web parts/tool parts were derived from our custom classes which abstracted some responsibilities. Since we couldn't easily change the web part picker screen to only display appropriate web parts for the zone the author had selected, we built the check into the base web part - if an 'invalid' web part was added, the web part renders nothing in presentation mode but in edit mode we display a message to the author like this:

    ModuleNotValidMessage 

    Adding too many web parts to a zone (count determined in the module matrix data) would have a similar effect.

  4. Combine interface of publishing field controls with web part storage

    Having decided to use web parts for our control architecture, we had one requirement for something similar to the standard Content Editor web part (CEWP). However, this control is pretty lame compared to the MOSS publishing HtmlField, and we quickly established our client needed more than the basic CEWP. So we combined the bits we wanted from both - the front end control used by the publishing field type (the RichHtmlField control), but the backing store of web part storage rather than a publishing field. This meant authors could add multiple instances of this optional module to their page (and get the nice editing experience), but because it's a web part we didn't need to worry about having a corresponding set of fields on each possible content type. In code/integration terms it's the same approach, but in the end we actually swapped the standard MOSS control for the control which fronts Telerik's RADEditor field since the client wanted to move to this: 

    CustomContentEditorWebPart  

    Also note use of another control typically used with publishing fields here, the AssetUrlSelector - this provides the 'Browse...' button shown above, and can be used to provide a friendly way for an author to browse to a file.

  5. Control adapter for WebPartZone for accessibility compliance

    Since web parts normally render with a stack of nested HTML tables which won't validate against AA, action needs to be taken to remedy this if accessibility is a design goal. However this isn't necessarily a big deal - the approach is that you 'correct' the HTML for the WebPartZone control in presentation mode only, thus leaving the tables intact in edit mode for all the web part editing framework stuff which needs to happen. You do lose the client-side Web Part Services Component (WPSC) API doing this, but we had no requirement for it anyway (I rarely see it used). I initially assumed I'd have to write a control adapter to do this, but I found that David Schneider has already done the job - this works fine. It's also possible the latest version of the AKS has one, can't remember if I checked.

  6. Present only our web parts in the web part picker

    Since this is a highly bespoke WCM platform rather than a standard collaboration environment, we don't want to see any of the standard web parts in the picker for these sites. Two steps to this one:

    - delete all the .webpart files from the web part galleries in the sites (N.B. we used Kivati for rolling out such changes across all the site collections - more on this in the future). However, doing this will still leave you with ListView web parts for all the lists/libraries in your site, so you also need to..
    - ensure all your WebPartZone declarations have the little documented 'QuickAdd-ShowListsAndLibraries' property set to false:
    <WebPartPages:WebPartZone id="g_AB07678E486C46bc962DFC8446A6CD13" runat="server" title="Zone 1" QuickAdd-ShowListsAndLibraries="false" />


    Authors are then not confused by any standard web parts which aren't appropriate for our scenario:

    StrippedWebPartPicker


  7. Remove unnecessary options when editing web part properties (tool parts)

    Finally, we do a bit of work with the accompanying tool parts (for properties editing) for our web parts to avoid confusing our authors with options which won't take effect. As an example, for a web part which looks like this in presentation mode:

    ProductSelectorModule 

  8. The tool part looks like this:

    ProductSelectorToolPart

    In case you're wondering what to look at, it's that we've removed the standard options SharePoint would normally provide for every web part (such as chrome style etc.), since we want to control these to ensure proper formatting. Normally we'd have these sections at the bottom of the tool part:

    RemovedToolPartOptions

Summary

There are many ways SharePoint's web part framework can be extended, and here I'm only showing the path we followed. For a requirement such as our client's, web parts provided a great starting point, perhaps showing there can sometimes be a place for web parts in an accessible publishing site so long as the trade-offs are understood and accepted.

In part 2 of this series we'll look at issues encountered and their resolutions.

Tuesday, 20 January 2009

A better Config Store for SharePoint sites

I've recently made some enhancements to my Config Store framework on Codeplex which I'm now ready to share. Many of these enhancements are a result of adapting the solution to a large project where we built a platform for 80-100 internet sites - hence it's now become a bit more 'enterprise'. With such things I generally make a deal with my employer where I do the work (or most of it) in my spare time and then get to share the code publicly, so here we go. Before I delve into the details, let's have a reminder of what the Config Store is all about:

Recap - the SharePoint Config Store in a nutshell

Regular readers may remember this is a solution which allows use of a SharePoint list to store configuration values used by your SharePoint application - the idea is that your webparts/server controls/page layouts etc. store any strings/data they need for configuration in here as a more flexible alternative to web.config or similar. Since our values are now in a SharePoint list, configuration can be updated across the farm through the browser to administrators who have the appropriate permissions. We can also optionally take advantage of all the other things lists give us such as auditing, item-level security, alerts and version history etc. Finally, a caching layer is used to avoid round trips when your code fetches configuration values.

On the last couple of projects where my team has used it, we've finished up with 100+ config items in the list - storing all sorts of things from URLs, strings and 'application behaviour' switches:

ConfigStore

If you need a more complete overview, see:

Enhancements in the new release

  1. Optional "hierarchical" configuration model similar to web.config

    In the first release, since all the config values are stored in one list this means your configuration is stored in one 'nominated' site collection. This is fine for WCM sites which may only use one site collection, but may not easily map to certain enterprise requirements. What's new in this release is that the framework can now use a 'hierarchical' model, where a Config Store list exists in whatever site collections should have one, but a 'master' site collection is nominated which contains the 'master' config values. What happens is that if the config item you request is in the 'local' Config Store you'll get that value, and if not you'll get the value from the master list. This allows local overriding of the parent values if required - in practice we found 95% of the config items would be stored in the master list only, but having the facility to override the other 5% was critical to supporting some of the functionality we developed.

    Needless to say, this is the implementation which best suited our requirements. It could be it doesn't really suit yours, but developers could consider starting with my source code and modifying, since other aspects such as the caching layer, Feature files, event handlers etc. might not need to be changed much.

    If you want to continue to just use one Config Store list (even if you consume the config values in multiple site collections), the new code will continue to work just fine for this model too.

  2. Easier to use in ASPX markup

    Similar to my recent Language Store framework, the Config Store now supports easily dropping values into ASPX markup by implementing an expression builder. So if all you want to do is set the Text property of a control, you can now do with this without cluttering up the code-behind. Or, as an alternative example, here we're retrieving an URL from the Config Store (stored in Category 'PageUrls' and key 'MyAccountPage') to assign to a hyperlink:
    <asp:HyperLink id="hyperlink1" NavigateUrl="<%$ SPConfigStore:PageUrls|MyAccountPage %>"
    Text="My account" ImageUrl="images/pict.jpg" runat="server"/>


  3. Fixed caching bug for farm environments

    Yes, something I have to hold my hands up to here - the caching layer in the initial release didn't adequately deal with multiple servers. The effect was that it required an app pool recycle to pick up changes to config values, rather than them taking effect immediately. Not the end of the world, but certainly inconvenient and taking away some of the benefits of using a SharePoint list for configuration. So in this release, the implementation correctly relies on a back-end resource (using a CacheDependency) to invalidate the cache across all servers in the farm. The implementation I chose was to add the items to the cache with a CacheDependency on a text file - this needs to be located on a path which all servers can access - and the event handler now updates a timestamp in the file to invalidate the item in the cache across all servers.

    I went with using a file as the dependency item as I thought it was more lightweight than using a SqlCacheDependency - I didn't really want to impose the SQL configuration etc. to be able to use the Config Store. However, with the file cache dependency I've chosen, be aware that some production configurations may have internal firewalls which prevent all WFEs from accessing a shared file in this way - check before you deploy. Of course the code is there for you to modify should you wish to make changes in this area.

  4. Amendment to Feature to prevent web.config modifications being made on Feature activation

    Another thing that got in the way when I tried to implement the Config Store on our enterprise project was the Feature event receiver which adds the required appSettings keys to web.config. The idea here is that, to help simplify installation and initial setup, the required web.config entries are added with empty values so all the developer has to do is plug in the appropriate values for his/her site. Sounds great - and it is for single site collection sites. But for multiple site collections, when we come to activate the Feature in the 2nd, 3rd, nth site collection - of course the receiver runs and adds the empty entries to web.config again, despite the fact that we'd already inserted the real values on the first activation. And since the new values come later in the file, guess which ones are used?

    So in this release there's a Feature property which determines if web.config modifications are made - it's set to 'False' by default but it's there if you prefer to change it.

  5. 'Config value' column is now bigger

    Previously this column was a 'single line of text' but it's now a 'note'. This means you can happily store HTML/XML fragments or other larger values, which is very useful in some scenarios. However, note that if you're already using the Config Store on your site and want to 'upgrade', this schema change is significant - I discuss it in the readme.txt file.

So that's it. You can download the updates from www.codeplex.com/SPConfigStore - hope you find it as useful as we have.

Tuesday, 6 January 2009

Speaking at the SharePoint Best Practices conference

By now you've probably seen a few posts on the Best Practices™ SharePoint Conferences which are coming up - I'll be speaking at the European 'edition' which is being held 6th - 8th April in London, UK. I submitted a couple of sessions, but here's the one which got confirmed:

Customizing SharePoint the supported way: from end-user to admin interfaces

One of the key concepts SharePoint developers must remember is that modifications to core shipped files are typically unsupported. This session looks at ways in which you can implement the kind of customizations your users may ask for without doing anything naughty. Aspects covered include custom site definitions, modifying the site administration/central administration areas, and changing the user controls SharePoint uses with the DelegateControl architecture. The session is packed with demos, and also contains tips on the best way to make other common customizations not shown in the detailed examples.

I'll probably be off his Christmas card list for saying this, but when Steve Smith (fellow UK MVP and conference organiser) mentioned a while back that he wanted to put something like this on and I should think of sessions, I kind of assumed he meant something with the usual UK suspects, a bit like a user group meeting on steroids. It was only when the site went up I saw how wrong I was! Many of the world's biggest SharePoint names are on the speaker list - the developer track has Andrew Connell, Todd Bleeker, Eric Shupps, Maurice Prather (ex SharePoint product group) and fellow UK SharePoint MVPs Andrew Woodward and Ben Robb. And that's not all - there are 4 tracks in total and I've already noticed some non-dev sessions I'd like to get to from experts such as Spence, Mike Watson, Ben Curry and others. The keynote speaker is Joel Oleson. Regardless of your level with SharePoint, for my money there is huge potential for pushing your SharePoint capability forward at this conference.

As an aside, the other session I submitted (which I really wanted to do!) was this:

Approaches and best practices for deploying SharePoint sites through multiple environments (dev, QA, UAT, production)

Solutions and Features, STSADM export, the Content Deployment API – all can be used to move SharePoint site artifacts from one environment to another, and all have different capabilities and limitations. Failing to plan your deployment processes properly can add significant risk to a SharePoint development project, and in some scenarios the problems may only become apparent when initial development is complete and updates need to be released. In this session we’ll explore the nuts and bolts of each development/deployment approach, and you’ll emerge on a firmer footing in this complex area of SharePoint. We’ll also touch on 3rd-party tools which can help with the deployment puzzle, from Codeplex utilities such as the Content Deployment Wizard to emerging tools such as Kivati Studio.

If you're thinking of attending and would be interested in this topic, leave me a comment and if there's a good response I'll try and persuade the organisers to shoehorn me in somewhere (perhaps in one of the lunchtime sessions which haven't yet been announced).

European Best Practices™ SharePoint Conference

Look forward to seeing you there!

Tuesday, 16 December 2008

Using .Net Expression Builders to set control properties

In my last post I introduced my Language Store solution for multi-lingual SharePoint sites, and showed the two ways it can be used:

In standard .Net procedural code:

string sButtonText = LanguageStore.GetValue("Search", "SearchGoButtonText");

Declaratively in HTML:


<asp:Button runat="server" id="btnSearch" Text="<%$ SPLang:Search|SearchGoButtonText %>" />


This declarative syntax is very useful as it means the developer doesn't have to clutter up code-behind files just to call the method to retrieve a value, then assign it to the 'Text' property of various controls. I've also retrofitted this to my Config Store solution (along with some other enhancements) and this will be available on Codeplex soon. You might notice it's the same syntax as the SPUrl token which can be used in master pages/page layouts to get a relative path to an image or CSS file, and that's because I'm using the same .Net technique. Since I had to do some digging to work out how this was done, I'm guessing (could be wrong here!) many other developers haven't come across this either, so here's how it's done.

Implementing an expression builder class

An expression builder is essentially a class which derives from System.Web.Compilation.ExpressionBuilder and contains logic to evaluate an expression at page parse time. The 'secret' is that the ASP.Net page parsing engine understands that it needs to call the class's method whenever it encounters an expression in the appropriate form. These are the things that join this mini-framework together:

  • Class derived from ExpressionBuilder which overrides the EvaluateExpression() and GetCodeExpression() methods

  • Declaration in web.config which associates your prefix ('SPLang' in my case) with your expression builder class

  • Optional use of ExpressionPrefix attribute on class for designer support

  • An expression in declarative HTML (as per the example above)

Taking things step-by-step, here's what my class looks like:


[ExpressionPrefix("SPLangStore")]
public class LangStoreExpressionBuilder : ExpressionBuilder
{
private static TraceSwitch traceSwitch = new TraceSwitch("COB.SharePoint.Utilities.LanguageStore",
"Trace switch for Language Store");

private static LangStoreTraceHelper trace = new LangStoreTraceHelper("COB.SharePoint.Utilities.LangStoreExpressionBuilder");

public static object GetEvalData(string expression, Type target, string entry)
{
trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "GetEvalData(): Entered with expression '{0}'.",
expression);

string[] aExpressionParts = expression.Split('|');
string sCategory = aExpressionParts[0];
string sTitle = aExpressionParts[1];

if ((aExpressionParts.Length != 2) || (string.IsNullOrEmpty(sCategory) || string.IsNullOrEmpty(sTitle)))
{
trace.WriteLineIf(traceSwitch.TraceError, TraceLevel.Error, "GetEvalData(): Unable to parse expression '{0}' into " +
"format 'Category|Title' - throwing exception.",
expression);

throw new LanguageStoreConfigurationException("Token passed to Language Store expression builder was in the wrong format - " +
"expressions should be in form Language Store Category|Item Title e.g. Search|SearchGoButtonText");
}

string sValue = LanguageStore.GetValue(sCategory, sTitle);

trace.WriteLineIf(traceSwitch.TraceInfo, TraceLevel.Info, "GetEvalData(): Retrieved '{0}' from Language Store.",
sValue);

trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "GetEvalData(): Returning '{0}'.",
sValue);

return sValue;
}

public override object EvaluateExpression(object target, BoundPropertyEntry entry,
object parsedData, ExpressionBuilderContext context)
{
return GetEvalData(entry.Expression, target.GetType(), entry.Name);
}

public override CodeExpression GetCodeExpression(BoundPropertyEntry entry,
object parsedData, ExpressionBuilderContext context)
{
Type type1 = entry.DeclaringType;
PropertyDescriptor descriptor1 = TypeDescriptor.GetProperties(type1)[entry.PropertyInfo.Name];
CodeExpression[] expressionArray1 = new CodeExpression[3];
expressionArray1[0] = new CodePrimitiveExpression(entry.Expression.Trim());
expressionArray1[1] = new CodeTypeOfExpression(type1);
expressionArray1[2] = new CodePrimitiveExpression(entry.Name);
return new CodeCastExpression(descriptor1.PropertyType, new CodeMethodInvokeExpression(new
CodeTypeReferenceExpression(base.GetType()), "GetEvalData", expressionArray1));
}

public override bool SupportsEvaluate
{
get { return true; }
}
}


If you're wondering why two methods are required, it's because GetCodeExpression() is used where the page has been compiled and EvaluateExpression() is used when it is purely being parsed. My code follows the MSDN pattern which supports both modes and uses a third helper method (GetEvalData()) which both call into. It's this GetEvalData() method which does the work of parsing the passed expression and then using it to obtain the value - in my case the expression is the 'category' and 'title' of the item to fetch from the Language Store. Notice that effectively, the key line in all of that is one line in GetEvalData() which calls my existing LanguageStore.GetValue() method - so effectively my expression builder is just a wrapper for this method.

My web.config entry looks like this:


<add expressionPrefix="SPLang" type="COB.SharePoint.Utilities.LangStoreExpressionBuilder, COB.SharePoint.Utilities.LanguageStore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=23afbf06fd91fa64" />

And finally here's how the component parts of the expression get used:

ExpressionBuilderSyntax

For SharePoint solutions, assuming we're deploying our code as a Feature/Solution we'd generally want to add the web.config entry automatically by way of the SPWebConfigModification class. You can find the code to do this in on Codeplex in the source code for my Language Store solution (in the Feature receiver).

Finally, if you're building an expression builder and this information doesn't get you all the way, the MSDN documentation for the ExpressionBuilder class has some additional details.

Enhancing the design-time experience with a custom expression editor

This is something I haven't looked at yet, but looks extremely cool! If the standard expression builder stuff wasn't convenient enough for you, you can extend things further by providing a custom 'ExpressionEditor' for use in Visual Studio. If I understand the possibility correctly, this can provide a better experience in the VS properties grid in two ways:

  • a custom editor sheet (e.g. a dialog to enter the 'category' and 'title' of the Language Store item - let's say 'Search' and 'SearchGoButtonText' was entered respectively, this would 'build' the string in the correct delimited 'Search|SearchGoButtonText' form required)

  • a custom picker using the Expressions collection - this could (I think) be used to query the Language Store list and display all the items, so that selecting the item to display the translation of is as simple as a few clicks, no typing!

I'd absolutely love to implement this for the Language Store/Config Store - so I might return to this at a later date!

Conclusion

Expression builders provide a powerful, clean way to inject method calls into your markup. In most cases we're used to seeing them return strings as in my Language Store/Config Store implementations, and note that the following implementations are also present in the .Net framework:

However, one final thing to bear in mind is that the signature of the method returns an object - so theoretically it should be possible to do a whole host of other things, where the processsing returns a more complex object which gets assigned to the control property. An example could be data-binding scenarios where your method returns something which implements IEnumerable/IList - this could then be assigned to the DataSource property of your control declaratively. You might have other possibilities in mind, but hopefully that's food for thought ;-)