Sunday, 4 January 2015

Developing for Office 365 – thoughts on use of custom master pages and web templates etc.

As more and more organizations are choosing to deploy SharePoint workloads on Office 365, it’s interesting to see Microsoft change the guidance given to developers on how customizations should be implemented. If you’ve been following closely, you might be aware that the following messages have started to emerge in recent months:

  • Avoid custom master pages
  • Avoid using web templates
  • Avoid using declarative XML (i.e. Feature XML) in a WSP (even a sandbox WSP – and remember, we already should not be using server-side code here either!)

These messages are mainly relevant to the Office 365/SharePoint Online space – but arguably have some relevance for on-premises SharePoint too. At the time of writing (Christmas 2014), these messages aren’t really on MSDN yet but will very soon be published in Microsoft Virtual Academy training. I have a lot of respect for the guys in Microsoft behind this thinking. I’ve had some great conversations with Vesa Juvonen, Steve Walker, Bert Jansen and so on about these topics, and these guys ultimately have to battle slightly opposing forces - “making Office 365 a good development platform WITHOUT making the service too difficult/expensive to run” would be a summary of that. Hopefully this doesn’t get me crossed off their Christmas card list (!), but in this post I’ll explain why I’m not sure that I agree with the guidance in all cases.

After all, these are BIG changes!! No custom master pages? No web templates? Really?? Web templates were the answer to world hunger not that long ago – after all, if you’ve got several hundreds/thousands of team or project sites to create, and they all should have some tools/widgets above the out-of-the-box team site, web templates are the tool in SharePoint way that Microsoft provided to do this! And custom master pages have long been the way to implement a reasonable level of branding in SharePoint – and that’s important to many organizations. So what is behind this guidance? Do you need to follow it? For our projects, I’m taking the view that we SHOULD be following it for collaboration/team site-based stuff compared, but NOT necessarily for highly-branded publishing intranets which are more communications-oriented. So to summarize this decision process:

Custom master pages and web templates - collab vs publishing - small

I’ll explain my thinking later, but first let’s understand why Microsoft are giving this guidance.

Using custom master pages/web templates means you might not *automatically* get new Office 365 functionality

As we all know, Office 365 is an “evergreen” platform – it is regularly updated with new functionality for end-users (the “ripples, not waves” approach). To roll out these updates, Microsoft sometimes need to update out-of-the-box site templates and master pages. Some examples of this could be:

  • New functionality added to SharePoint Online team sites (e.g. new lists or settings used) = Microsoft update to team site template (STS#0)
  • New functionality added to SharePoint Online via new page controls = Microsoft update to OOTB master pages

Of course, if you’re using a custom web template and master page, you won’t receive these kind of changes.  That’s a pain because ultimately you WANT this stuff in many cases – it could be some high-value functionality that your end-users would really benefit from (and perhaps that you were thinking of building a customization for!). And indeed, you’re paying for this “service” in your Office 365 licenses; it seems a shame not to get the full benefits. As time goes by, it’s also true to say that your custom master page will become more and more out-of-sync with OOTB master pages, as depicted by this diagram from Microsoft training:

Custom master page vs OOTB

Of course, what you could do is manually copy over any master page updates to your custom master page(s) for example – but this comes with effort each time. Some level of analysis, code changes, testing, deployment through multiple environments and so on – all these tasks are needed for each Microsoft update. Microsoft refer to this as the “customization tax” that you must pay if you do use custom web templates/master pages in Office 365 – and I think that’s a good description.

The flip side – having control of updates

So, some extra effort is required if we do decide to use custom web templates or master pages. But on the flip side, it could give you more control over when and how Office 365 updates are applied to your environment, and this can be appealing. Certainly if you have 50,000 users on your Office 365 global intranet and a Microsoft update breaks the home page one day (and such “breaking” changes have occurred), that’s a lot of folks calling the helpdesk and a very unhappy business! That’s probably the worst case scenario here, and it’s true to say that Microsoft have improved communications around updates – but even less dramatic scenarios such as changes which don’t break anything but do change an intranet experience need to be managed properly. Compared to team/project sites, publishing intranets tend to have more “designed” user experiences, with custom branding/look and feel and potentially more customizations. And this means a greater chance of a Microsoft update causing an issue, perhaps some kind of conflict with a minor customization – usually you’d want the opportunity to test these things together before applying in production.

If nothing else, perhaps using a custom web template/master page is one layer of protection against uncontrolled updates for publishing intranets. It won’t cover all scenarios (depending on the type of update), but certainly will cover many. However, I do strongly feel that collaboration scenarios such as team sites and project sites generally should NOT use custom web templates/master pages. There’s way less benefit to heavy branding/look and feel customization in these areas (so less need for a custom master page), and I think you really want any updates which come from Microsoft here – it could be valuable new functionality that your business users will appreciate. Additionally, since you often have many collaboration site collections (but just one publishing intranet site collection), the idea of manually copying any updates from Microsoft to these areas has more complexity too. So, my starting position is a custom web template/master page for publishing scenarios but not for collaboration scenarios – obviously each implementation may then have further variables to consider, but that’s my default thinking.

Other layers of protection

As a slight aside, other things you could do around Microsoft updates to Office 365 include:

  • Close monitoring of Microsoft communications around updates – particularly:
  • Running other Office 365 tenancies for test purposes with “First Release” mode enabled
  • Signing up to the NDA Preview program if possible

And just how much of a burden is the “customization tax” anyway??

For many organizations rolling out Office 365 (especially large enterprises), I can’t help but think this “customization tax” is often a drop in the ocean in many ways. Most of the clients I work with already have a team (or teams) handling other roll-out or customization tasks, and they know that it takes much more than switching the thing on and sending a couple of e-mails to get the most of the platform. Having to occasionally have a developer make some updates is perfectly fine in the overall running of the intranet.

All these factors feed into my thinking about using a different approach for collab vs. publishing intranet scenarios.

Alternative approaches to custom web templates and master pages

But let’s say you don’t want to use a custom web template or master page or both) - perhaps because you’re doing team sites, or for other reasons.  How can you get the same effect (and what do Microsoft suggest instead)?

If you need…

“Classic” approach

Alternative approach

To apply custom branding/look and feel
  • Custom master page
  • Use a Composed Look/theme/alternate CSS file for look and feel
  • JavaScript injection to add custom page components/controls
To have a template to create many sites from
  • Custom web template
  • Site is based on an OOTB site type (e.g. team site) and..
  • ..use Remote provisioning code to configure content types, lists/libraries and so on after site creation. The two key approaches here are:

Obviously the alternative approaches need some thought during your implementation, and extra effort is usually involved – but you should find that they at least work well for non-publishing scenarios (e.g. team/collab sites), thus aligning somewhat with my thoughts of treating these implementations differently. You might find though, that NOT using a custom master page for a highly-bespoke intranet would be difficult anyway, particularly if aspects such as responsive design are involved. On the templating side, the steer from Microsoft is to use remote code approaches to apply customizations to out-of-the-box sites. This goes back to the other point in my opening bullet point list, which we haven’t discussed yet:

Avoid using declarative XML for provisioning (use imperative/code-based approaches instead)

So yes, whilst many of us have spent the last few years using Feature XML for provisioning, Microsoft would now prefer us NOT to use this approach in Office 365 even if it is still 100% supported. Certainly the “XML vs. code for provisioning” debate has been around for many years now, but Microsoft are now coming down on the code side heavily – and in SharePoint Online this means remote code (CSOM or JSOM). It’s interesting to pause and consider the types of things we’d commonly use XML for here:

  • Provisioning fields and content types
  • Provisioning lists and libraries
  • Provisioning files (either “site infrastructure” files, such as master pages/page layouts/CSS/JavaScript/images etc., or default content such as pages or documents)
  • Provisioning web parts
  • Declaring templates – list templates and web templates (and site definitions before that)
  • Adding a CustomAction (e.g. to link a JavaScript file, or to add a ribbon customization)
  • Custom field controls
  • Delegate controls
  • Declaring event receivers
  • Feature receivers

Only a couple of those (e.g. Feature receivers, custom field controls, delegate controls) can’t be used at all in the sandbox (and therefore SharePoint Online), but you can achieve the same thing in other ways.  The rest can, but Microsoft are starting to say we should use remote code where possible instead of XML. The reason for this is that inside SharePoint, the functionality becomes dependent on an XML file somewhere instead of purely being within “provisioned” items in the content database – and this could make some kinds of system upgrade more complex. Whilst this is an interesting SharePoint engineering “thing”, it’s somewhat hard to see this as our problem rather than the Office 365 engineering team’s – after all, these upgrades are mainly things that Microsoft will undertake as part of running SharePoint Online. The “content database upgrade” step as part of, say, upgrading Office 365 to use SharePoint 2015 (or whatever) would be an example. 

As someone who leads a group of 20 developers working in many SharePoint/Office 365 engagements at any one time, I currently have some reservations about the team switching entirely to imperative/c0de-based provisioning for all scenarios. I’ve mentioned this to Steve/Vesa etc. in the past – in the spirit of open discussion, here’s a slightly expanded list of my thoughts on the “don’t use declarative XML” message from that conversation:

Some quick thoughts from me are:

  • Some aspects of switching to imperative provisioning are pretty inconvenient - my team (and SharePoint devs generally) are very efficient at declarative provisioning (~7 years experience ;)), and lots of the resources on the internet are heavily-oriented towards this.
  • The declarative/XML approach has been pushed by Microsoft for 7 years and even now is still 100% supported in Office 365.
  • I completely get that if all customers used imperative provisioning rather than declarative, certain things/upgrades would be simpler for Office 365 engineering. But frankly, I'm not sure I'm on board yet that this is my problem too! I know that in some places it *can* be very relevant to me as the initial implementor (e.g. initial deployment can be done declaratively, but changes/updates require code), but even with this in mind, often we *still* prefer declarative for initial provisioning!
  • Imperative provisioning obviously doesn't cover some templating scenarios easily - list templates, site templates etc. Sure, I can implement some remote provisioning solution which does a similar thing, but even with the OfficeDev samples I’d say it's more effort than the declarative approach - AND, sometimes there's no easy way to have things appear in the expected place in the end-user UI (especially for list templates). Big gap here really.
  • Providing a production-grade, resilient “remote code hosting” solution can be difficult for some clients (e.g. those not comfortable with Azure for this). 

It will be interesting to see how this area plays out in time. These are just my opinions obviously, and it’s certainly true that using code rather than XML can bring advantages (such as better debugging and logging). Still, it seems a shame to no longer have both options. Ultimately this sub-decision might now only be a consideration for publishing intranets and the like, if you’re happy to go with the thinking that collaboration-oriented sites should indeed use OOTB site types with remote provisioning code (in other words, you’ve already decided to use code here). But I’m certainly reluctant to abandon the declarative approach across the board just yet.

Summary

There’s quite a lot to think about with the new guidance, and what’s right in one situation may not be right in another. It’s up to you to consider the factors (alongside what you need to achieve), but as usual understanding the landscape as much as possible helps you make the best decision. My thoughts are to consider publishing intranets and collaboration sites differently – but hopefully this post at least helps you with some detail around what to think about!

23 comments:

Stefan Bauer said...

Great thoughts on customisations of master pages and web templates.
I take a look into custom master pages and found something that might be worth to considered how Microsoft SharePoint Master Pages and changes to the user interface are might can be updated in the future. I documented this in the blog post Case study how to use same master page in Office 365 and on-premises
Kind regards
Stefan.

Unknown said...

Interesting reading. As someone who works with an on-prem SharePoint 2010 installation, I'm not sure I've ever thought to look for the kind of "ripple" changes in v4.master you describe here. How often do master pages change in Office 365?

Quatic said...

I always say, the closer the page is to the end users daily processes, the more you want to stick to the out-of-the-box SharePoint interface. Which is in line with you.

So I agree => corporate intranet => branded. Team site, project sites etc. => not branded (only use minor styling by e.g. themes)

Bruce said...

Nice write-up (as usual). I can appreciate your reluctance to move away from declarative provisioning of IA, considering how invested you are in that approach.

For those of us who have been using Excel to define our IA and using that to feed PowerShell deployment scripts for years now, we get off lightly (or rather, have inadvertently picked the winning side in advance).

The difficulty I have with Intranet = Branded vs Team Sites = not is that in my experience, clients want the same UX for both. Trying to sell them an uglier team sites experience is not going to be easy...

Unknown said...

"I completely get that if all customers used imperative provisioning rather than declarative, certain things/upgrades would be simpler for Office 365 engineering."

Annnd we've come full circle. Does Microsoft even properly understand the concept behind declarative? It's extremely frustrating from a consultant perspective to have things we've been told and thus have re-preached to all be thrown under the bus without any clear guidance in any direction. It's frustrating that other CMS solutions provide clear paths to the approach on branding and have opt-in versioning. Perhaps this the guidance that MS wants us to resell: go elsewhere.

Chris O'Brien said...

@Unknown (first one),

It's a good question (on how often master pages get updated in Office 365) - I don't have any numbers on this, so it's difficult to say exactly (unless someone who has been specifically monitoring this could chime in?). But several UX changes have been rolled out in the last year, from changes to the top navigation bar (and later, the introduction of the app launcher) through to changes to the mobile view of SharePoint sites. I guess another interesting point here is that, looking forward, I'd suggest even Microsoft couldn't tell you today how many changes they'll make in the next 12 months say - it will depend on the kinds of new features which get introduced, and this is now driven more by responding to analytics rather than long-range planning.

Cheers,

COB.

Chris O'Brien said...

@Bruce,

Yes, interesting thought, and we've definitely used similar approaches in places. However, I feel they don't always fit because they heavily rely on sites being created by I.T. using the "magic script" - absolutely fine for some scenarios, but not always (e.g. self-service site provisioning).

But as you say, this approach is indeed inadvertently (one form of) the winning side if moving away from declarative provisioning becomes mandatory/critical at some point :)

On your other comment, do you find your clients ask for heavy UX customisation of team sites? i.e. more than could be achieved with themes/Composed Looks?

Cheers,

COB.

Chris O'Brien said...

@Unknown (second one),

Yes, I definitely hear you that it can be frustrating when the direction of Microsoft travel changes - it often hurts the things I work on too. BUT, on the positive side, I believe the SharePoint/Office 365 engineering teams are now more sensitive to these changes than before. Also, as someone who perhaps sees a bit more behind the curtain, I do see the challenge Microsoft are often faced with - either improve things (but annoy people because changes are required along the way), or sacrifice the long-term improvements due to the disruption. Of course, sometimes the "new stuff" is simple and doesn't bring the need for changes to what's out there already, but with a platform that originated 10 years ago, clearly that's not always going to be the case.

Tricky one!

COB.

Fabian Williams said...

Great read Chris
I just started a new project where we will be designing and implementing a move to O365 and need to discuss a migration strategy with a branding component. This article has helped me in that decision making process. Cheers. see you in a few months in London mate.

Nigel said...

Hi Chris
Happy New Year !
I think this goes wider than you intimate. If you have a SP2010 farm and you are looking to move to SP2013 (online is not an option yet - no MS data centre in the UK yet !). How does one construct a SP2013 farm so that when it is time to move to SharePoint online how do we minimise the effort and the disruption. The things you have described should be taken into account now when constructing the SP2013 farm. Hopefully, you can use these techniques with locally hosted apps rather than Azure. If so then we are someway to reducing the effort and cost of "moving to the cloud" at a later date.

Nigel

Adam Toth said...

Using a custom master page isn't going to insulate/protect you from breaking changes from Microsoft.

Your custom master page is going to reference controls (like the ribbon) whose underlying DLLs can be changed by Microsoft at any time, potentially changing the markup emitted from those controls. If you have custom JavaScript or CSS that targets specific page DOM elements, the markup changes may make those customizations fail.

It's a really tough situation for customers to be in. Many companies I work with aren't willing to wait indefinitely for Microsoft to finally provide a responsive team site master page and create it themselves (and sign up for the customization tax in the process).

All symptoms of a technology still based on ASP.NET 2.0, that never had a good separation of logic/UI.

Chris O'Brien said...

@Nigel,

Yes, this is a good point - I'd say though, that this perhaps speaks to the wider aspects of "cloud-friendly" development. For a while now, many of the on-premises projects my team has worked on have used cloud-friendly approaches precisely to minimise any re-work if/when a move to Office 365 happens. This means developing exclusively using client-side techniques and the app model/remote code model, and avoiding all the things which cannot be used in SharePoint Online (e.g. server-side SharePoint code, timer jobs, files deployed to the _LAYOUTS directory and so on). More details on this stuff at Presentation deck: Modern SharePoint development - techniques for off-box code

Ultimately, this "roadmap to the cloud" discussion is something we *always* have with our clients, even if they are on-premises and it isn't particularly on their radar.

Cheers!

COB.

Chris O'Brien said...

@Adam,

Well, I definitely agree that using a custom master won't protect you from *all* breaking changes from Microsoft, and that some updates will come from web control DLLs for example. This is what I was trying to say with:

"If nothing else, perhaps using a custom web template/master page is one layer of protection against uncontrolled updates for publishing intranets. It won’t cover all scenarios (depending on the type of update), but certainly will cover many."

But, I do still think a custom master page will give you something - otherwise Microsoft wouldn't care if folks used one or not! The fact that they are saying "you shouldn't use a custom master page if you want all the updates" is clear to me that some changes will be deployed this way. And in the scenario of the intranet home page for 50,000 users - well, frankly it might be worth taking every layer of protection you can get?

Cheers,

COB.

Jeremy Thake said...

As always Chris, thanks for a great balanced write up.

Vesa Juvonen and Steve Walker recently did a live event on this topic which is now available on-demand (http://www.microsoftvirtualacademy.com/training-courses/transform-sharepoint-customizations-to-sharepoint-app-model )

I've also done some podcasts with Steve and Vesa on this too which can be found here http://blogs.office.com/2014/10/02/office-365-developer-podcast-episode-018-steve-walker-sharepoint-ux-developer-guidance/ and more shows here http://dev.office.com/podcasts .

We also very recently updated the Solution Pack for SharePoint Online with more guidance on this topic http://www.microsoft.com/en-us/download/details.aspx?id=42030

I will continue to monitor the comments around this as people chime in as its very useful to take back to engineering.

Sly Gryphon said...

It will be good to have this guidance from Microsoft, as I often tell customers they should try and stick to out of the box if possible.

One big area with remote provisioning is however the issue of tooling. The tool support in Visual Studio for declarative fields, content types, etc, has improved every version and to switch back to an entirely untooled approach (raw code) is a big productivity hit.

(The other big area, as mentioned in the article, is existing skill sets, but MS can't really solve that problem.)

If MS wants this to work, they really have to invest heavily in tool support for generated code (partial classes) for remote provisioning. (Similar to what is available in ASP.NET or Entity Framework).

With some decent tooling, including support for modularisation and easy apply/rollback, a remote provisioning solution will be as easy as existing declarative (with improved benefits such as debugging).

Chris O'Brien said...

@Jeremy,

Thanks for adding the links. There's obviously a lot to think about/talk about here, so thanks for your efforts on being Microsoft's eyes and ears as usual :)

Cheers,

COB.

Chris O'Brien said...

@Sly Gryphon,

Agree 100% with all of those points I think. Could be a while until we get to that ideal state though - especially on aspects like rollback (never straightforward given the underlying SharePoint architecture).

Good comments, cheers!

COB.

Bruce said...

@Chris, the biggest thing that clients ask for that can't be achieved through composed looks/themes is a fully responsive design (commonly done by integrating bootstrap in the master page and page layouts). Jeremy has said that MS is slowly moving towards making the OOTB master more responsive (some components like the suite header already are), but they aren't there yet.

nigelpwordpress said...

@Bruce There are patterns and practises expounded by Steve Walker and Vesa Juvonen of Microsoft here :- http://www.microsoftvirtualacademy.com/training-courses/transform-sharepoint-customizations-to-sharepoint-app-model

Allow you to inject the Bootstrapper for the Responsive UI into a master page. So you can do this now.

Gerald Ferry said...

We also recently updated our Guidance around branding and customization for SharePoint Online

Plan customizations, solutions, and apps for SharePoint Online

Bruce said...

@Nigel - I'm aware of that training (I got up early to attend the session live). I would question the feasibility of "injecting" boostrap markup at run-time. Even if you could get it working in this fashion, it would be incredibly brittle - the slightest change to the OOTB masterpage that it is manipulating could break it and then you're back where you started.

Richard Walsh said...

Reading this excellent summation and the comments that follow it, especially around the new direction on NOT using declarative XML provisioning techniques, I can't help but see this as yet another "solution to a non-problem", a la Furuknap.

I have nearly 10 years experience in the SharePoint space, and I am really starting to despair with the direction Microsoft are going with all of this. I get the reasons why they are doing it (in many cases anyway), but the way it is asking SP Devs to jettison skills they have perfected over many years is hard to take. As someone who is not a young Dev fresh out of university, I'm not sure I have the desire to re-skill at this stage, especially when the cynicism based on hard experience tells me Microsoft will likely change direction again on all of this at some point in the future.

omar said...

It's interesting that MS would introduce the Design Manager and discourage branding in the same release.