Thursday 1 November 2007

Master pages/page layouts deployed as Feature not updating

Since Deploying master pages and page layouts as a Feature has been the most heavily commented article on this blog, and several of the posters seem to have run into the same problem, I wanted to write a quick post with some more information from my experiences on this.

So this is something of a non-standard post, feel free to tune out if it doesn't affect you ;-)

Anyway, I decided to do some more testing to see if either I'd got something wrong or if perhaps I was doing something differently to the people having problems. My test was basically to knock up a publishing site with a master page and page layout (associated with a custom content type as it often would be), then go through the update process. This is what I found:

  • making updates to the files (outside of the 12 folder) and then XCOPYing these to overwrite the files in the 12\TEMPLATE\features\MyFeature\ directory successfully updated the site. No need to deactivate/activate the Feature.
  • using a Solution package to deploy the files (when using Features this is generally what I do since I'm in a farm environment) - again this updated the site correctly when I upgrade the Solution (stsadm -o upgradesolution). This is to be expected since underneath the exact same thing is happening as in the previous test. (However, I also noticed occasionally the directory would complete disappear even after the solution upgrade had completed, or was there but still locked by another process, meaning the files could not be accessed even in Windows Explorer - this is slightly irritating but running the Solution upgrade again always succeeded.)
  • after causing the file to be customized (e.g. modifying or even just checking out with SPD), any subsequent updates to the files via the Feature/Solution did not appear on the site (though further updates in SPD are fine).

In short, this is all what I expected. Assuming the file has not been customized, anything which updates the copy on the filesystem will cause an update to the site. If it has been customized, updates on the filesystem will not (since the file has now been added to the content database, and the filesystem version is no longer used). If you've not come across this before, Considerations when using Features to deploy SharePoint files - ghosting/unghosting may help.

So, I'm guessing that if your master pages etc. are not updating when you overwrite the Feature files, it's because the files have become customized somehow. Unfortunately it's not so easy to tell for publishing files - for other files, SharePoint Designer provides a handy blue dot next to the file in it's Explorer view if the file is customized, but alas this doesn't happen for master pages/page layouts. The blue dot can be seen next to the AllItems.aspx file below (click to enlarge):

Unfortunately this also means reverting to the file on the filesystem is not straightforward either (we can't right-click the file in SPD and select 'Reset to site definition' as we can with other SharePoint files). So this can be a pain if you do want to keep your page layouts referenced from the filesystem (e.g. because performance is critical), but you've ended up in this state. It is possible to revert the files using the API though. I've not needed to do this myself, but the property to check is SPFile.CustomizedPageStatus and if this returns SPCustomizedPageStatus.Customized, then the SPFile.RevertContentStream() method can be used - this should cause SharePoint to henceforth use the version on the filesystem (though note you may lose some updates which had been made after the file was unghosted (customized) - you will need to re-apply these to the filesystem file after the reversion. And remember, don't use SPD for this or you'll be back where you started!)

So far, so (reasonably) straightforward.

However, one poster (deelpunt) had an interesting question about updating page layouts with web parts. As far as I can see, updating all page instances to have web parts in web part zones by updating the layouts will not be possible. This is because if web part zones are used, the web part is associated with the page instance rather than the page layout. Indeed, this can be the power of the architecture. It is possible to either:

  • have default web parts added to a zone when a page instance is created from a page layout. This can be done by deploying the page layout using a Feature, and using the AllUsersWebPart tag. This would not affect pages already created however.
  • add web parts to all the pages by adding them directly to the page markup in SPD, rather than in a zone. Of course, this then means the settings for the web part can only be modified by the page designer in SPD, rather than site users.
  • use the API to iterate through all pages in the site to add/modify webparts using SPFile.GetLimitedWebPartManager(). Needless to say, this is the kind of operation which requires a lot of care and planning in production!

As I've mentioned before, because of these issues the web part zone architecture is often not the best choice for scenarios such as WCM site development, since here we want our changes to apply across all pages which use the layout.

Hopefully this has been of some use. As always, leave a comment if you've had different experiences to those detailed here, I'd definitely be interested to hear.

[I also wanted to say sincere apologies to the commenters on the original post (and any others I've been slow in replying to) that it took a couple of weeks for me to come back. Something to do with my project going live and moving house at the same time, hopefully normal service now resumed ;-) ]


Anonymous said...

Good info, in particular that part about the web part zones. I experienced that first hand and it was quite annoying. Fortunately, the original web application completely blew up on me, so I had to rebuild and that fixed that! ;)

One thing I have noticed is that the last update date in the Master Page Gallery does not update when upgrading, though the pages do get updated. Odd.

I've run into another issue related to the publishing preview images. I added them to my custom feature:

<Module Name="PreviewImages"
Url="_catalogs/masterpage/Preview Images"
<File Url="tp_one_col.png"
Type="GhostableInLibrary" />

Then I added the PagePreviewImage property to the page layout:

<File Url="tp-onecolumn.aspx"
<Property Name="ContentType"
<Property Name="Title"
Value="Trading Post One Column"/>
<Property Name="PublishingPreviewImage"
Value="~SiteCollection/_catalogs/masterpage/Preview Images/tp_one_col.png, ~SiteCollection/_catalogs/masterpage/Preview Images/tp_one_col.png"></Property>
<Property Name="PublishingAssociatedContentType"
Value=";#Trading Post;#0x010100C568DB52D9D0A14D9B2FDCC96666E9F2007948130EC3DB064584E219954237AF390016611AA92AAE4F4BA3A28A1CC4C75A3A;#" />

Deploying the solution seems to go fine. I'm able to active the feature on the site collection; however, the images do not appear in the Preview Images folder and the page layouts continue to use the default page layout image. If I try to manually upload the images to the Preview Images folder, I get an error that the file exists and needs to be checked out before updating, yet I can't see the darned things anywhere.

Any ideas?


Chris O'Brien said...

Hi Colin,

Yes, the last updated info doesn't get modified when upgrading files with a Feature. I think this is because the file on disk gets overwritten but the list item does not get touched.

With regards to your preview images problem, I think you have the module declaration for the images slightly wrong. From comparing with mine, I think to solve this you should:

- set the Url attribute of the module to "_catalogs/masterpage"
- set the Name attribute of the image file "Preview Images/tp_one_col.png"

I think this related to the way folders in lists are implemented. Everything else look OK though.

Let me know if that solves it.



Anonymous said...

You are brilliant! You've managed to get me half way there with one post!

The images, I am happy to say, are now appearing in the Preview Images folder in the site collection; however, I don't see them when I go to create a page and select one of our custom layouts. Do you think this might be an issue with my property declaration or something in the site configuration itself?

Thanks again!

By the way, if you're ever in downtown Minneapolis, let me buy you lunch!

Chris O'Brien said...

Happy to help dude :-)

Hmm, I would have thought that should work. However, note there's also a PreviewImages folder lower down the Master Page Gallery structure, under the folder for your culture (I'm assuming "en-us"). Other preview images are stored here.

Let's try targeting that folder instead. Having made the changes I suggested last time, also try this:

- on the Module element for the preview image, add the following attribute: IncludeFolders="??-??"
- change the 'Value' attribute of the 'PublishingPreviewImage' on the page layout to be:

"~SiteCollection/_catalogs/masterpage/en-us/Preview Images/tp_one_col.png, ~SiteCollection/_catalogs/masterpage/en-us/Preview Images/tp_one_col.png"

This should hopefully cause the image to end up in that lower down folder, and it to be properly referenced from there.

Keep me posted.


Anonymous said...

SharePoint just doesn't like to let go...

I finally got this working on a test site collection. I couldn't get the IncludeFolders="??-??" to do anything useful, but I was able to get it to work by changing the module URL element to "_catalogs/masterpage/en-us".

To get the pagelayouts to reference the proper images, though, I had to delete the layouts from the Masterpage Gallery, then reactivate the feature. ON our intranet site, I can't delete any of the layouts, since they are in use. And it isn't the layout page itself, but rather the meta info about the layout. I tested this by adding comments to all my layouts then redeployed the solution. The comments are all there for existing pages, but the preview image is not set in the properties for the layouts.

I can't update the properties, since this would ghost the layouts, right? So, what's the easiest way to add the image associations for these layouts? Can I do it directly in the database, or do I need to write an update script?


Chris O'Brien said...

Ah OK, so if you're trying to add preview images to page layouts which have already been provisioned, I could well believe that the Feature won't update the ListItem (i.e. metadata) for these files. I would have thought that the info I gave last time should be good for new page layouts though - a quick check in the 'PublishingLayouts' Feature added by MS confirms the same settings I used. [N.B. I was under the impression IncludeFolders="??-??" added the files to all child folders.]

In terms of whether editing the ListItem only will ghost the file, I'm afraid I'm not 100% sure. This is something I've been meaning to test but haven't got to yet. If you have a test environment, test by:-

- write code which tests the SPFile.CustomizedPageStatus propety for a file
- on your test file, check the value is not SPCustomizedPageStatus.Customized
- edit the metadata
- check if the value has changed



Rachel said...

I have created a feature that deploys two masterpages and some page layouts. The masterpages deploy fine with no problems, however the page layouts are causing an issue.

When I deploy the feature the page layouts (they are new layouts) appear in sharepoint designer, however they do not appear in the master page gallery on my site, nor within the page layout list when you add a new page.

On further investigation if I check out the new layout page from Sharepoint Designer its happy but when I try and check it in again I get an error “Cannot perform this operation. The file is no longer checked out or has been deleted”

The page layouts definitely aren't unghosted at the time of the deployment.

Any ideas?
Thanks! Rach

Chris O'Brien said...

Hi Rachel,

Hmm, even though they're appearing in SharePoint Designer it kinda sounds to me like something might not be quite right with your elements.xml file. Does the XML look like mine at Deploying master pages and page layouts as a Feature?

I'm happy to take a look if you post it here. You'll probably need to remove the angle brackets though.



Anonymous said...

Hi Chris,

Great blog you have and great articles you write.

I am the poster of the interesting topic. (deelpunt).The issue aroung updating page layouts with web-parts is still open.

1. the page layout using a Feature, and using the AllUsersWebPart tag. This would not affect pages already created however.

2. add web parts to all the pages by adding them directly to the page markup in SPD, rather than in a zone. Of course, this then means the settings for the web part can only be modified by the page designer in SPD, rather than site users.
3. use the API to iterate through all pages in the site to add/modify webparts using SPFile.GetLimitedWebPartManager(). Needless to say, this is the kind of operation which requires a lot of care and planning in production!
As I've mentioned before, because of these issues


the web part zone architecture is often not the best choice for scenarios such as WCM site development, since here we want our changes to apply across all pages which use the layout.

Remarks: option 1 I am using now is for a better deployment. But updating still needs to be done via adding webparts programmatically through code. And this is difficult in a deployment scenario.

But the main reason for using the combination between publishng controls and webparts is that you can use OOB listview webparts and filtering webparts. Some content is managed via lists. And filtering can be done easily.

So that comes to a second topic around this:

I can't get the webpartconnection in a page layout to work.

I have a serious problem getting the webpart connection to work within the pagelayout.

I am making a pagelayout with two webpart zones. The first webpart zone contains a DataForm webpart. The second contains a Sharepoint LisFilter webpart.

I used a DataForm webpart to view List-content from another web in the SiteCollection. I set the ConsumerID for the The DataForm webpart. Then I use the List Filter webpart for filtering the DataForm. Now I want to set the webpart connection in the page layout. So that the filtering occurs automatically when creating a page from the pagelayout. An content-editor only needs to set the filter-value.

I used the following snippet in my Page Layout for setting the webpartconnection:

WebPartPages:SPProxyWebPartManager runat="server" id="ProxyWebPartManager"

But the problem is that the webpartconnection is not set correctly. The filter webpart still says: This filter webpart is not connected. I think there is a problem with the ConsumerConnectionPointID and the ProviderConnectionPointID. Because I use out of the box webparts I don't know if the [ConnectionProvider] and [ConnectionConsumer] Attributes are set correctly. In these Attributes you define the ConsumerConnectionPointID or the ProviderConnectionPointID.

[ConnectionConsumer( "filter", "UniqueIDForConsumer", AllowsMultipleConnections = true)]


[ConnectionProvider( "Subject", "UniqueIDForRegionConnection", AllowsMultipleConnections = true)]

Because I don't write my own webpart and thus not set these Attributes.

Because you are an authority on WCM I hope you can clear these matter a bit for me. A lot of customers ask for Page Layouts where we must combine Publishing controls with webparts. This causes a lot of tricky scenario's.


1 Combination Publishing controls and webparts in webpart zones when filtering needs are there must be possible Or ??? Is there another scenario ?? (write user controls or something).

2. Page layouts: setting webpart connections can it been done or must I write custom code for connecting webparts to each other.

Thanx for any answers. All help is welcome

Bertrick Karsten
Consultant Capgemini

Chris O'Brien said...

Hi Bertrick,

Apologies for not responding sooner. I'm afraid I've not done anything with connected web parts in page layouts, so not sure how much help I can be. I would have thought that so long as the web part manager is present everything should work OK though.

If you're open to alternatives, one option could be to ditch the ListViewWebParts in favour of custom controls. These would issue CAML queries based on the parameter the user enters. You won't get the full "working with lists" experience provided by the ListViewWebPart, but if you just want to list items in a specific format it could be OK. If you need the dropdown context menu on each item (but not the list toolbar), you could look at the enhanced Content Query Web Part on Codeplex - you should be able to set the QueryOverride property programatically, in order to take the parameter entered by the user.

If you want to stick to the web parts, I would definitely recommend trying code to do the connection as you suggest.

Hope that's of some use..



Anonymous said...

Hello Cris.
Tnx for an enlightening article. I have though som piece of information about the ghosting/unghosting debate.

It all concerns publishing sites.
A little history.
while teaching sharepoint development - wich i do a lot, i had my class conduct a little experiment. We implemented a little microsecond counter to see what the difference between ghosted and unghosted pages actually were.
The servers were differen - what i could get my hands on - including several virtual mashines.
To unghost the page we used SharePoint Designer. We made ourselves 2 teamsites (wss 3.0) and unghosted one of the defaultpages doing nothing but opening and saving the file in sharePoint Designer. then we started hitting the pages and all measurable responsetime was logged.
This showed ud that the unghosted page performed between 16 and 34 times slower than the twinn ghosted page.
That should easily lead me th the conclusion that if the page is important - like the default page on an intranet - unghosting would be a serious mistake.
So that is what im teaching.

Later on i started running some tests on a publishing site( moss 2007), and the: SPCustomizedPageStatus returns Customizen (unghosted) for all publishing pages. Wich I due the the experiment above makes me shiver :-)
To me unghosting means - not beeing cashed by the .NET cache - wich again means that it will load intrepetted like the old asp line by line. Is it that the architecture of this if fucked up from the top , or doesnt it really mean that much ( that would be a question).
Another part of this question.
IF the layoutpage is unghosted you will not see changes to the layoutpage itself - as you mention in this article. That COULD lead the the conclusion that editing an GHOSTED layout page would imideately change all pages created from this layout. But Sadly it isnt so - if the users have altered the page in any way by inserting a webpart or whatever the page will become disconnected to the layoutpage In wich case the ghosted/unghosted issue becomes totally irellevant for this excact publishing page.

IS there any way that i can better the performance of the publishing pages?

Does it in anyway affect the performance if the pagelayout connected to a publishing page is ghosted or unghosted?

Using some kindda hardware cache or outputcachint ofcouse helps, but makes a lot of trouble for the personalization - like welcome peter even though you are hannah

Chris O'Brien said...


First off, sorry for the delay - your comment kind of slipped through the net temporarily.

You raise some interesting points - I definitely agree uncustomized pages will perform better than customized pages, though to be honest I'm not sure about the performance stats you quote. In terms of publishing page layouts, again uncustomized (i.e. deployed through a Feature) will perform better, so if page load times are critical consideration should be given to this. I'd suggest performance testing in your environment is a good idea to establish if it's absolutely required - often once all your custom code has been added I'm not sure the customized/uncustomized is as big a factor as you'd think.

On a final note, in terms of combining output caching with personalization, the key is using post-cache substitution to add in the personalized content (e.g. "Welcome Hannah") after the main page content has been fetched from the cache.



Unknown said...

can u pls tell me how to upgrade the already generated visual studio solution(using Sharepoint Solution Generator)with new features made in sharepoint site.

1) Do I need to delete any files while creating the new solution for the updated features?
2) After running the sysadm upgrade command & i try to open the site, it is giving me error as not found? Can you please help.

Chris O'Brien said...


Afraid I haven't done much with Solution Generator, but I'd expect an upgrade to perhaps not be as simple as you're thinking - in general SolGen will give you the site definition which you can create new sites from, but modifying sites which have already been created cannot be done by modifying the definition. You should do this by adding new Features/using the browser UI/using the object model/STSADM etc. to make your changes.



Randy Bouquet said...

Chris, once again, you have posted an interesting and helpful article. During my testing I found something which augments one of your statements, so I thought I would comment so everyone could benefit from testing efforts.

You mentioned merely checking out a file via SPD would customize a file. I went into the Master Page Gallery through the UI and did some testing. Checking out a file does not customize the page (as it does in SPD) however, the page does become customized when it is checked in. Discarding a checkout will also make sure the file does not become customized. Since this differs from the way you reported SPD working, I thought I would call it out.

Chris O'Brien said...


That's a useful clarification, appreciate it.