Friday, 27 March 2015

Key skills and topics for today’s SharePoint/Office 365 developer

Chris OBrien SharePoint Office 365 skills topics 2 As I prepare to talk at the upcoming Ignite and SharePoint Evolutions conferences (will post details soon), I was reflecting on the technology landscape I’m currently working in.  As I’m not the first to mention, for many SharePoint developers the world is a different place now to how it’s been in previous years. If you work with the cloud in some form (Office 365, Azure, Yammer or other platforms) then you’ll probably be nodding your head at that statement. If not, then maybe the wave hasn’t hit you yet but could do in the next 12 months. Either way, the level of change seems to have ramped-up from mere evolution to full-on revolution.

I often maintain a list of skills and topics which I think are relevant to me and the kind of projects I’m involved in. I do this partly for my development and that of my team around me at Content and Code, but for other reasons too. As someone involved heavily in recruitment of technical people, some of this powers my list of current interview questions and what I look for in candidates. As such, I always thought publishing it would probably be a bad idea, because:

  • The list is ever-changing – at the very least, it evolves every few months
  • Recruiters (and candidates) would take advantage of this info

Call me cynical, but I know that in the UK market at least, recruiters do seek to boost the chances of their guy getting an interview by ensuring CVs are targeted to what is being looked for. Nothing wrong with that if everything stays honest - but I’ve seen quite a lot of cases where the truth is stretched too far. Fortunately not all recruiters use such tactics of course, but it can make the job of identifying good developers somewhat harder. Still, I found myself writing my skills/topics list on a whiteboard in a meeting the other day, and the idea of jotting it down somewhere public came to mind again. I now figure “Ah, what the hell – if a candidate actually *does* know all this stuff, then however they gained the knowledge I probably want to work with him/her anyway!” After all, seeing the list isn’t the same as having genuinely deep knowledge in each area. I’m not sure what other technical hirers do but, like the Microsoft exams, I personally take the ‘adaptive’ questioning approach – if the candidate appears to know a certain topic, I’ll do my best to keep asking questions to see just how deep the knowledge goes. I certainly don’t claim to be the world’s best developer, interviewer or anything else for that matter (and not even on my “strongest” topics!), but I often find I *can* establish where the limits of the candidate are for a given area.

Key skills/topics

So, anyway here’s my list – it’s just a dump from my OneNote notes rather than anything particularly structured. And there’s no special sequence or anything:

  • Remote SharePoint/SharePoint Online development techniques
    • .NET CSOM and JSOM
    • Using CSOM in PowerShell
    • The REST API
    • The App Model
      • Apps for SharePoint/Apps for Office
        • Provider-hosted vs. SharePoint-hosted
        • App authentication – context tokens/access tokens, app-only policy etc.
        • Infrastructure requirements
      • Office 365 apps
        • App registration – and the App Launcher/My Apps page
        • App authentication – ADAL and adal.js, the Common Consent framework, options for app-only tokens via a certificate
        • Office 365 APIs and client libraries
      • Deciding between Office 365 apps and Apps for SharePoint
      • Deploying apps to Azure (both provider-hosted Apps for SharePoint/Office and Office 365 apps
        • Protecting the app with Azure AD authentication
    • OfficeDev
      • The Patterns and Practices core libraries
      • The Patterns and Practices samples
    • Office 365
      • Identity management options
      • AAD Sync
      • Hybrid
    • Office 365 and Azure
      • Azure AD
      • How the "special" restricted Azure subscription behind an Office 365 tenancy works (i.e. just AAD)
  • General SharePoint/SharePoint Online development
    • Key building blocks for out-of-the-box solutions
    • Search-based solutions - Content Search web part, search APIs etc.
    • Managed Properties/Crawled Properties - provisioning (including differences in SharePoint Online), auto-created properties, using in search refiners etc.
    • Display templates - provisioning templates, the end-to-end process of having values from a custom column displayed in a display template (via a custom Managed Property)
    • JSLink
    • Considerations around use of custom master pages, web templates etc.
    • Branding options – Office 365 theming, alternate CSS, custom master pages etc.
  • Azure
    • Azure Web Apps (was Azure Websites) – development considerations, scaling models, deployment slots etc.
    • Azure Web Jobs
    • Azure SQL Database
  • General web development
    • Responsive design - level
    • JavaScript
      • jQuery (DOM manipulation, AJAX methods etc.)
      • Patterns for OO JavaScript
      • Promises
      • Cross-domain issues and options
      • Knockout.js
      • Angular.js
    • ASP.NET
      • MVC
      • WebAPI - especially:
        • Calling from JavaScript
        • Dealing with incoming/outgoing JSON
        • Implementing a REST service
        • Implementing CORS etc.
    • Token-based authentication models
    • Developing for performance - bundling and minification, page weight issues, useful tools etc.
  • Yammer
    • Yammer concepts - networks, groups, topics etc.
    • Using Yammer Embed
    • User sync (i.e. Yammer DSync)

No doubt this isn’t comprehensive and there are lots of items that could be added to this. Feel free to leave a comment if you think I’m missing something big :)

Summary – reflecting on this list

What’s interesting about this to me, is that if I’d written a similar list a couple of years ago it would be full of things like “web parts, timer jobs, event receivers, Feature XML” and lots of other SharePoint-only constructs. Some of these things can still be very relevant depending on the type of work you’re doing of course - but in general there are so many new concepts for most SharePoint developers in the last couple of years. Key themes here include general web development (i.e. non-SharePoint specific), new platform aspects such as Office 365 and Yammer, and Azure and Azure Active Directory (AAD) deserve a special mention since they become extremely important in the new world. For example, for Office 365 I highly recommend that even developers get their hands dirty with some infrastructure/platform aspects such as implementing domain integration and perhaps AAD Sync. After all, the production environment you’ll be targeting will probably use this configuration! Doing this in my dev environment certainly helped solidify my understanding of a few things – for example, what controls the attributes which are sync’d between the on-premises AD and Azure AD:

AAD Sync chris obrien com

One thing is for sure – things have changed irreversably for SharePoint developers, and moving forward I think these types of changes will apply more and more to “on-premises only” SharePoint developers too. More details on SharePoint 2016 will be announced at Ignite very soon, and whilst we already know the full-trust/farm solutions model will live on, I think various aspects of cloud app development are likely to come into that world too. Exciting times my friends!

Thursday, 26 February 2015

Podcast interview: customizations and Office 365 – Microsoft Cloud Show

I was interviewed last week for the Microsoft Cloud Show by Andrew Connell and Chris Johnson. If you need something to listen to whilst you're working/travelling/exercising, I think it was a fun chat about lots of Office 365/SharePoint things! I’m a regular listener of the show (perfect for my morning runs!), so it was cool to be a guest and catch up with those guys.

The overall show lasts for just over an hour - like all their shows there's a segment at the beginning where AC and Chris talk about the week's news (something they do really well IMHO), so the interview itself starts after around 20 minutes. We covered things like:

  • The recent changes in Microsoft guidance for Office 365 development (e.g. avoid custom master pages, avoid WebTemplates, avoid use of Feature XML etc.) - this was the main topic of the interview
  • Microsoft now releasing SharePoint updates via Windows Update
  • Overall changes in Microsoft strategy (i.e. the new "openness" discussed in another Yammer thread recently)

The podcast episode can be listened/downloaded from:

Chris OBrien MS Cloud Show interview

Sunday, 4 January 2015

Developing for Office 365 – thoughts on use of custom master pages and web templates etc.

As more and more organizations are choosing to deploy SharePoint workloads on Office 365, it’s interesting to see Microsoft change the guidance given to developers on how customizations should be implemented. If you’ve been following closely, you might be aware that the following messages have started to emerge in recent months:

  • Avoid custom master pages
  • Avoid using web templates
  • Avoid using declarative XML (i.e. Feature XML) in a WSP (even a sandbox WSP – and remember, we already should not be using server-side code here either!)

These messages are mainly relevant to the Office 365/SharePoint Online space – but arguably have some relevance for on-premises SharePoint too. At the time of writing (Christmas 2014), these messages aren’t really on MSDN yet but will very soon be published in Microsoft Virtual Academy training. I have a lot of respect for the guys in Microsoft behind this thinking. I’ve had some great conversations with Vesa Juvonen, Steve Walker, Bert Jansen and so on about these topics, and these guys ultimately have to battle slightly opposing forces - “making Office 365 a good development platform WITHOUT making the service too difficult/expensive to run” would be a summary of that. Hopefully this doesn’t get me crossed off their Christmas card list (!), but in this post I’ll explain why I’m not sure that I agree with the guidance in all cases.

After all, these are BIG changes!! No custom master pages? No web templates? Really?? Web templates were the answer to world hunger not that long ago – after all, if you’ve got several hundreds/thousands of team or project sites to create, and they all should have some tools/widgets above the out-of-the-box team site, web templates are the tool in SharePoint way that Microsoft provided to do this! And custom master pages have long been the way to implement a reasonable level of branding in SharePoint – and that’s important to many organizations. So what is behind this guidance? Do you need to follow it? For our projects, I’m taking the view that we SHOULD be following it for collaboration/team site-based stuff compared, but NOT necessarily for highly-branded publishing intranets which are more communications-oriented. So to summarize this decision process:

Custom master pages and web templates - collab vs publishing - small

I’ll explain my thinking later, but first let’s understand why Microsoft are giving this guidance.

Using custom master pages/web templates means you might not *automatically* get new Office 365 functionality

As we all know, Office 365 is an “evergreen” platform – it is regularly updated with new functionality for end-users (the “ripples, not waves” approach). To roll out these updates, Microsoft sometimes need to update out-of-the-box site templates and master pages. Some examples of this could be:

  • New functionality added to SharePoint Online team sites (e.g. new lists or settings used) = Microsoft update to team site template (STS#0)
  • New functionality added to SharePoint Online via new page controls = Microsoft update to OOTB master pages

Of course, if you’re using a custom web template and master page, you won’t receive these kind of changes.  That’s a pain because ultimately you WANT this stuff in many cases – it could be some high-value functionality that your end-users would really benefit from (and perhaps that you were thinking of building a customization for!). And indeed, you’re paying for this “service” in your Office 365 licenses; it seems a shame not to get the full benefits. As time goes by, it’s also true to say that your custom master page will become more and more out-of-sync with OOTB master pages, as depicted by this diagram from Microsoft training:

Custom master page vs OOTB

Of course, what you could do is manually copy over any master page updates to your custom master page(s) for example – but this comes with effort each time. Some level of analysis, code changes, testing, deployment through multiple environments and so on – all these tasks are needed for each Microsoft update. Microsoft refer to this as the “customization tax” that you must pay if you do use custom web templates/master pages in Office 365 – and I think that’s a good description.

The flip side – having control of updates

So, some extra effort is required if we do decide to use custom web templates or master pages. But on the flip side, it could give you more control over when and how Office 365 updates are applied to your environment, and this can be appealing. Certainly if you have 50,000 users on your Office 365 global intranet and a Microsoft update breaks the home page one day (and such “breaking” changes have occurred), that’s a lot of folks calling the helpdesk and a very unhappy business! That’s probably the worst case scenario here, and it’s true to say that Microsoft have improved communications around updates – but even less dramatic scenarios such as changes which don’t break anything but do change an intranet experience need to be managed properly. Compared to team/project sites, publishing intranets tend to have more “designed” user experiences, with custom branding/look and feel and potentially more customizations. And this means a greater chance of a Microsoft update causing an issue, perhaps some kind of conflict with a minor customization – usually you’d want the opportunity to test these things together before applying in production.

If nothing else, perhaps using a custom web template/master page is one layer of protection against uncontrolled updates for publishing intranets. It won’t cover all scenarios (depending on the type of update), but certainly will cover many. However, I do strongly feel that collaboration scenarios such as team sites and project sites generally should NOT use custom web templates/master pages. There’s way less benefit to heavy branding/look and feel customization in these areas (so less need for a custom master page), and I think you really want any updates which come from Microsoft here – it could be valuable new functionality that your business users will appreciate. Additionally, since you often have many collaboration site collections (but just one publishing intranet site collection), the idea of manually copying any updates from Microsoft to these areas has more complexity too. So, my starting position is a custom web template/master page for publishing scenarios but not for collaboration scenarios – obviously each implementation may then have further variables to consider, but that’s my default thinking.

Other layers of protection

As a slight aside, other things you could do around Microsoft updates to Office 365 include:

  • Close monitoring of Microsoft communications around updates – particularly:
  • Running other Office 365 tenancies for test purposes with “First Release” mode enabled
  • Signing up to the NDA Preview program if possible

And just how much of a burden is the “customization tax” anyway??

For many organizations rolling out Office 365 (especially large enterprises), I can’t help but think this “customization tax” is often a drop in the ocean in many ways. Most of the clients I work with already have a team (or teams) handling other roll-out or customization tasks, and they know that it takes much more than switching the thing on and sending a couple of e-mails to get the most of the platform. Having to occasionally have a developer make some updates is perfectly fine in the overall running of the intranet.

All these factors feed into my thinking about using a different approach for collab vs. publishing intranet scenarios.

Alternative approaches to custom web templates and master pages

But let’s say you don’t want to use a custom web template or master page or both) - perhaps because you’re doing team sites, or for other reasons.  How can you get the same effect (and what do Microsoft suggest instead)?

If you need…

“Classic” approach

Alternative approach

To apply custom branding/look and feel
  • Custom master page
  • Use a Composed Look/theme/alternate CSS file for look and feel
  • JavaScript injection to add custom page components/controls
To have a template to create many sites from
  • Custom web template
  • Site is based on an OOTB site type (e.g. team site) and..
  • ..use Remote provisioning code to configure content types, lists/libraries and so on after site creation. The two key approaches here are:

Obviously the alternative approaches need some thought during your implementation, and extra effort is usually involved – but you should find that they at least work well for non-publishing scenarios (e.g. team/collab sites), thus aligning somewhat with my thoughts of treating these implementations differently. You might find though, that NOT using a custom master page for a highly-bespoke intranet would be difficult anyway, particularly if aspects such as responsive design are involved. On the templating side, the steer from Microsoft is to use remote code approaches to apply customizations to out-of-the-box sites. This goes back to the other point in my opening bullet point list, which we haven’t discussed yet:

Avoid using declarative XML for provisioning (use imperative/code-based approaches instead)

So yes, whilst many of us have spent the last few years using Feature XML for provisioning, Microsoft would now prefer us NOT to use this approach in Office 365 even if it is still 100% supported. Certainly the “XML vs. code for provisioning” debate has been around for many years now, but Microsoft are now coming down on the code side heavily – and in SharePoint Online this means remote code (CSOM or JSOM). It’s interesting to pause and consider the types of things we’d commonly use XML for here:

  • Provisioning fields and content types
  • Provisioning lists and libraries
  • Provisioning files (either “site infrastructure” files, such as master pages/page layouts/CSS/JavaScript/images etc., or default content such as pages or documents)
  • Provisioning web parts
  • Declaring templates – list templates and web templates (and site definitions before that)
  • Adding a CustomAction (e.g. to link a JavaScript file, or to add a ribbon customization)
  • Custom field controls
  • Delegate controls
  • Declaring event receivers
  • Feature receivers

Only a couple of those (e.g. Feature receivers, custom field controls, delegate controls) can’t be used at all in the sandbox (and therefore SharePoint Online), but you can achieve the same thing in other ways.  The rest can, but Microsoft are starting to say we should use remote code where possible instead of XML. The reason for this is that inside SharePoint, the functionality becomes dependent on an XML file somewhere instead of purely being within “provisioned” items in the content database – and this could make some kinds of system upgrade more complex. Whilst this is an interesting SharePoint engineering “thing”, it’s somewhat hard to see this as our problem rather than the Office 365 engineering team’s – after all, these upgrades are mainly things that Microsoft will undertake as part of running SharePoint Online. The “content database upgrade” step as part of, say, upgrading Office 365 to use SharePoint 2015 (or whatever) would be an example. 

As someone who leads a group of 20 developers working in many SharePoint/Office 365 engagements at any one time, I currently have some reservations about the team switching entirely to imperative/c0de-based provisioning for all scenarios. I’ve mentioned this to Steve/Vesa etc. in the past – in the spirit of open discussion, here’s a slightly expanded list of my thoughts on the “don’t use declarative XML” message from that conversation:

Some quick thoughts from me are:

  • Some aspects of switching to imperative provisioning are pretty inconvenient - my team (and SharePoint devs generally) are very efficient at declarative provisioning (~7 years experience ;)), and lots of the resources on the internet are heavily-oriented towards this.
  • The declarative/XML approach has been pushed by Microsoft for 7 years and even now is still 100% supported in Office 365.
  • I completely get that if all customers used imperative provisioning rather than declarative, certain things/upgrades would be simpler for Office 365 engineering. But frankly, I'm not sure I'm on board yet that this is my problem too! I know that in some places it *can* be very relevant to me as the initial implementor (e.g. initial deployment can be done declaratively, but changes/updates require code), but even with this in mind, often we *still* prefer declarative for initial provisioning!
  • Imperative provisioning obviously doesn't cover some templating scenarios easily - list templates, site templates etc. Sure, I can implement some remote provisioning solution which does a similar thing, but even with the OfficeDev samples I’d say it's more effort than the declarative approach - AND, sometimes there's no easy way to have things appear in the expected place in the end-user UI (especially for list templates). Big gap here really.
  • Providing a production-grade, resilient “remote code hosting” solution can be difficult for some clients (e.g. those not comfortable with Azure for this). 

It will be interesting to see how this area plays out in time. These are just my opinions obviously, and it’s certainly true that using code rather than XML can bring advantages (such as better debugging and logging). Still, it seems a shame to no longer have both options. Ultimately this sub-decision might now only be a consideration for publishing intranets and the like, if you’re happy to go with the thinking that collaboration-oriented sites should indeed use OOTB site types with remote provisioning code (in other words, you’ve already decided to use code here). But I’m certainly reluctant to abandon the declarative approach across the board just yet.


There’s quite a lot to think about with the new guidance, and what’s right in one situation may not be right in another. It’s up to you to consider the factors (alongside what you need to achieve), but as usual understanding the landscape as much as possible helps you make the best decision. My thoughts are to consider publishing intranets and collaboration sites differently – but hopefully this post at least helps you with some detail around what to think about!

Tuesday, 16 December 2014

Office 365 apps – deciding between the SharePoint app model and the Office 365 APIs

Just like me, many other SharePoint developers have spent the last year or two getting to grips with the SharePoint app model – this was introduced in SharePoint 2013 as a way of building customizations which do NOT run on the SharePoint server. Enhancements built this way can run on a remote machine (e.g. Azure, IIS, whatever) and therefore any stability/performance issues do not affect core SharePoint. If you’re using Office 365 of course, the app model is a big cornerstone of extending the platform, since Microsoft do not allow custom SharePoint code to be deployed to the Office 365 servers (server-side code at least). These were big changes in the SharePoint development world, with lots to learn around authentication, remote APIs and patterns for solving common needs.

However, there’s a new kid in town. Microsoft have introduced a series of new APIs for Office 365 which span SharePoint, Exchange and Lync – and rather than needing to know each platform as a developer, more general concepts are exposed such as:

  • Mail
  • Calendar events
  • Contacts
  • Files

Of course, those who know the territory will recognize that “mail”, “calendars” and “contacts” are related to Exchange and that “files” are related to SharePoint (N.B. right now this is primarily about files stored in the user’s OneDrive for Business area). But the point is that not everyone who wants to work with Office 365 data these days has that background. Initially this new direction can be confusing to the experienced developer – after all, can’t I access SharePoint files using the existing SharePoint CSOM or REST APIs? Also, why is authentication different? I just spent months learning about provider-hosted apps and the “new” authentication models with context tokens, access tokens and so on! Why did we need something new?  If this is a new way of building apps for Office 365, is the old model going away? These are all logical questions in my opinion, so I wanted to talk about them here, as well as discuss some things to look out for in your apps which suggest you SHOULD be using the new approach. But for sure, if you’re about to start building an app you might be faced with this decision (with a gratuitous graphic to illustrate):

COB - provider-hosted SP app vs Office 365 app

After all, there are many similarities between the two – in both cases, the core implementation could be ASP.NET MVC, on Azure or some IIS servers for example. Let’s start by understanding what an Office 365 external app is about.

What an Office 365 app looks like

Here are some key tenets of these apps:

  • The application is an “external” or “standalone” app of some kind, perhaps implemented in ASP.NET (most likely MVC flavor these days). And when I say website, remember that it could really be something else altogether, like an App for Office which surfaces in Word or Outlook (since these use web pages underneath). Equally it could also be a Windows Store/Phone app or one which targets other devices, such as an iOS/Android app
  • We want to somehow integrate the app with Office 365 (perhaps in terms of user experience/navigation, data storage or authentication)
  • The app is registered in Azure Active Directory – specifically the Azure AD behind your Office 365 tenant
  • Apps can implement the same authentication mechanism as Office 365. This means your users get single sign-on, and can pass between Office 365 and your app without re-entering their username/password
  • Users can navigate to the app via the Office 365 App Launcher or “My apps” page
  • In coding terms, the apps are likely to use the new Office 365 APIs to access data there, but can also revert to using lower-level APIs such as the SharePoint CSOM/REST APIs
    • N.B. To do this, you use the authentication token obtained from Office 365 as the “bearer” token in the HTTP headers for the request to the other API:

Developing with the Office 365 APIs

If I was to summarize the development model, I’d say there are two high-level approaches:

  • Using the Office 365 REST API (more operations, but more complex since you need to craft the HTTP requests yourself and deal with authentication etc.)
  • Using a flavor of the Office 365 client libraries (simpler to use, but don’t have 100% coverage of what’s possible). These flavors include:
    • .NET (for use in ASP.NET websites, Windows Store/Windows Phone apps, Xamarin apps etc.)
    • iOS
    • Android
    • Cordova apps (multi-device hybrid apps)

Key differences between Office 365 apps and Apps for SharePoint/Apps for Office

But what if your app is SharePoint/Office focused, and could therefore be implemented as EITHER an Office 365 app or an App for SharePoint/Office? Let’s consider that decision for a while - here are some of the differences as I see them (N.B. I’m also summarizing these differences in a table below):

Difference 1 - how the app is accessed (by end-users)

Office 365 app:

An Office 365 app can be accessed in a few ways:

  • From any link (e.g. on your intranet home page), since it’s a standalone website
  • From the Office 365 App Launcher (if the app is “pinned” there for quick access)
  • From the Office 365 “My apps” page

App Launcher experience:

COB app pinned to app launcher

“My apps” page experience:

COB My apps page

App for SharePoint:

An App for SharePoint can be accessed in the following ways:

  • From the Site Contents page of a SharePoint site where the app is installed
  • From a developer-provided SharePoint link, using AppRedirect.aspx?instance_id=[app instance GUID here]

In other words, these apps can only be accessed from a SharePoint/SharePoint Online site – assuming that is, that the app accesses some SharePoint data (as most would). This is because the authentication model relies on the context token being passed from SharePoint into the app – otherwise the app cannot talk to SharePoint in any way.

Site Contents page experience:

App deployed to target site

Difference 2 – app registration (for implementer)

Office 365 app:

Firstly, these apps must be registered in the Azure AD which relates to the Office 365 tenancy:

COB Azure AD COB Azure AD - applications

Note that Visual Studio (more specifically, the Office 365 dev tools) perform this step for you when you create an app for a single Office 365 tenancy. Azure AD administrators can also register apps, and this is what needs to happen for multi-tenant apps.

App for SharePoint:

These apps are registered in SharePoint/Office 365 using AppRegNew.aspx (or using the Office Seller Dashboard for Store apps):


Difference 3 - how authentication and authorization works e.g. for SharePoint data (for implementer)

Office 365 app:

Once the app is registered in Azure AD, you can write code using the Office 365 client libraries to authenticate to the platform (simpler), or alternatively go lower level and obtain tokens yourself through OWIN (for MVC) or ADAL for .NET (both more complex, but what is needed when you’re coding directly against the Office 365 REST APIs). In terms of the client libraries approach, here’s a snippet showing how auth works in the .NET Office 365 library (taken from the Office 365 Starter Project for ASP.NET MVC sample):

You can cache these access tokens as you see fit to avoid repeated auth prompts. But if a new token needs to be obtained, this code does the work of redirecting the user to sign-in to Office 365 (whether they are in a web browser or on a device):

COB Office 365 sign-in - web


App for SharePoint:

Authentication for an App for SharePoint works in the following ways:

  • SharePoint-hosted app: authentication happens implicitly, since web pages are hosted in SharePoint context (and only the SharePoint JSOM API can be used)
  • Provider-hosted app: app must be registered with AppRegNew.aspx (or the Office Store), and .NET code can authenticate using the SharePointContext and TokenHelper classes which are added to your Visual Studio web project on creation:

Difference 4 – app deployment (for implementer)

Office 365 app:

Whatever is needed to to copy the app to the hosting location e.g. publish to Azure (from Visual Studio – shown below) or FTP or similar:

Publish web app - validate settings  

App for SharePoint:

Similar to above in that app files must be published to the hosting location, but with two extra steps:

  1. The app package (.app) must also be deployed to an App Catalog:

    Publish app - upload to app catalog
  2. Once the app is in the App Catalog, it must be installed to any SharePoint site which should use the app (or, a tenant-scoped installation should be performed):

     App install - in site

Summary table of differences/commonalities

Since the above sections are a bit difficult to cross-reference, let me try to summarize in a table:

Office 365 app

App for SharePoint

How the app is accessed (for end-user)
The App Launcher/My Apps page

Some other link (e.g. from some other intranet site you have) or bookmark – it’s a standalone website
Typically from the Site Contents page of a SharePoint site
Authentication to Office 365/SharePoint (for implementer) App is registered in Azure AD, then authentication helper in Office 365 client libraries typically used (e.g. AuthenticationContext object in .NET Office 365 SDK) App authentication – context token is passed from site into the app, and access token obtained from there. Helper code supplied in form of SharePointContext and TokenHelper classes (added automatically to app projects)
App hosting (for implementer) Any web platform – Azure or other cloud-hosting, on-premises IIS, or other web servers Any web platform – Azure or other cloud-hosting, on-premises IIS, or other web servers
App registration (for implementer) Registered in Azure AD Registered in SharePoint/Office 365 using AppRegNew.aspx (or using the Office Seller Dashboard for Store apps)
App deployment (for implementer) Deployed once to the hosting platform (e.g. Azure/IIS) Deployed to a SharePoint/Office 365 App Catalog, and then installed by site owners to various SharePoint sites or tenant-scoped install.

Alternatively, submitted to the Office Store for sale there.

Other differences

There are some wider differences not covered above too. Don’t forget things like:

  • For Office 365 apps, I no longer need knowledge of Exchange and SharePoint APIs to pull together an app which (for example) stores some files in an area I can access (i.e. my OneDrive) and sends an e-mail on my behalf. In other words, I don’t need to be a “career developer” in Exchange or SharePoint to make use of this stuff
  • Office 365 apps are probably easier to develop for devices – having SDKs for iOS and Android means that some common dev tasks are pretty simple on those platforms. I’d say the fact that these apps don’t need to be accessed from a SharePoint site can help here too

Summary - reasons to consider the Office 365 approach

So, there are clearly some things to consider if you’re building an app which could work as either a provider-hosted SharePoint app OR an Office 365 app. Here’s my summary list of indicators that the Office 365 app could be a good fit:

  • Your app doesn’t need to work with on-premises SharePoint/Exchange
  • Your app doesn’t really “fit” in SharePoint:
    • It’s an external app, not associated with a “site”
    • It sits at the “Office 365 suite” level
  • Your requirements match files/messages/events etc.
  • You want to integrate with the Office 365 App Launcher
  • You don’t want your app to need to be installed into SharePoint sites by site owners (and a tenant-scoped SharePoint app doesn’t suit your requirements)
  • You’re developing for devices and want to take advantage of device-specific SDKs
  • It’s the future!

In short, there will always be cases where either an Office 365 app or an App for SharePoint is a better fit – both models are valid going forward. The key is understanding the differences, both to the end-user and the implementer.

Feel free to leave a comment if you think I’m missing anything, or have any thoughts related to this!

Sunday, 16 November 2014

More speaking – SharePoint Connect 2014 (Amsterdam) and SharePoint Saturday UK

I’ll be doing a couple more talks in the next few weeks – these will cover different topics, ranging from the new Office 365 APIs to techniques for cloud-friendly SharePoint development, and also Apps for Office. Hopefully see you at one of these events!

SharePoint Connect 2014 (Amsterdam)

  • Modern SharePoint development - techniques for moving code off servers (Tuesday November 18, 14:50)
    • Those implementing Office 365 know that custom code cannot run on the SharePoint servers, and remote code alternatives must be used. However, it can be valuable to adopt the same "cloud-friendly" development techniques even for on-premises implementations. In addition to increasing platform stability, this enables you to build in flexibility to your deployment - leaving the door open to a potential move to Office 365 in the future without major re-engineering. With several code-focused demos, this session examines approaches for switching to "off-box" code - including Remote Event Receivers, PowerShell and CSOM scripts and client-side JavaScript.
  • An intro to the new Office 365 APIs (Wednesday November 19, 11:15)
    • Microsoft have stated that they will move away from SharePoint, Exchange and Lync APIs towards generic Office 365 APIs. The preview version of the first APIs were announced in March 2014, and cover several core building blocks for user collaboration - including mail, OneDrive files, calendar entries and contacts. Developers who need awareness of Office 365/hybrid scenarios can expect more such APIs in the future.

      Get on the bus early - come learn about the APIs and the new permissions model ("the Common Consent framework") in this code-focused session.

SharePoint Saturday UK (Hinckley, UK)

  • Apps for Office - the hidden gem that's easier than you think (Saturday November 28, 12:15)
    • Most developers know that it's now possible to develop Apps for Office which are based on core web technologies - HTML, CSS and JavaScript. But the take-up appears to be fairly limited - why is that when development and deployment (even in the enterprise) are fairly easy? This session takes a renewed look at Apps for Office, now that the Word, Excel and Mail APIs have been enhanced and the app model is no longer the weird new kid on the block. Some cool demos are involved, including a Word task pane app which analyses the contents of a document to find you similar items across your SharePoint environment/tenancy whilst you're working with the document.

Sunday, 9 November 2014

Slides/video from my Apps for Office talk at TechEd 2014 now available

I presented at TechEd 2014 last week (Barcelona) on “Weave your enterprise solutions directly into Office products with the app model”, alongside Andrew Salamatov from the Exchange product group - in this post I want to provide the PowerPoint deck and link to the TechEd video, but I also to provide some thoughts on Apps for Office in general. First off, here is the slide deck:


Link to TechEd video

The TechEd video can be watched in the embedded video at the bottom of this post, or the link to the video over on Channel 9 is - 

General information

If you’ve read other posts of mine, you’ll know that I’ve spent a lot of time developing Apps for SharePoint and Office 365 recently and doing “cloud-friendly” SharePoint development in general – but (like many people I suspect) I hadn’t spent too much time around Apps for Office. Having spent the last couple of months preparing to talk on the subject at TechEd last week, I feel like my eyes have been opened a bit here – Apps for Office have HUGE potential, and I feel you could easily become a hero in your client or organization with the right idea and a bit of dev effort!

The big game-changer is the development model for how Office is extended these days:



Deployment to users

Some weird VSTO/VSTA stuff that I never did :) Write code in C# against slightly strange APIs specific to Word, Excel etc. Have to get a DLL deployed to each user’s machine – good luck with that
Apps for Office

Web page hosted in client app (e.g. Word) – developer writes HTML, CSS and JavaScript.

JavaScript API for interacting with document. Many methods work across all client apps (e.g. Word, Excel, PowerPoint etc.)

No deployment to user’s machine – it’s a web page

Additionally, an App for Office could optionally make use of SharePoint/Office 365 building blocks if useful. Some apps will just relate to Office (e.g. the Wikipedia app in the Office Store), but in the enterprise there are quite a few scenarios where it would be useful to bring in SharePoint/Office 365 – perhaps to store files (e.g. in a team site, or the user’s OneDrive For Business site), or access the user’s calendar, mail, contacts etc. When you need an Office app which uses these things, there are two high-level approaches:

  • Implement your app as an App for SharePoint (usually provider-hosted), which encompasses an App for SharePoint
    • This works for both on-premises SharePoint and Office 365, but has some trade-offs (e.g. App for SharePoint needs to be deployed to sites, or tenant-scoped install)
  • Implement your app as a standalone/external app, which consumes Office 365 services via the new Office 365 APIs and is registered in Azure AD
    • This only works for Office 365, but lots of advantages (e.g. app can be made available in the new App Launcher)

This is a decision that will come up quite frequently for Office 365 development from this point (autumn/fall 2014) onwards, and I’m sure I’ll be touching on it in quite a few future articles.

My “find similar documents” sample app

For the talk, I developed an app which surfaces in Word - specifically in the “task pane” which sits alongside the document. My app is nowhere near production-ready, but it tries to illustrate a few concepts, including use of SharePoint/Office 365 for surrounding functionality. Since I wanted it to work with both on-premises SharePoint and Office 365, I chose the provider-hosted App for SharePoint model – but my conclusion now is that the Azure AD/Office 365 APIs approach is preferable if the app only needs to work with Office 365 and you can go that way. Anyway, my app is based around the idea of “finding documents similar to the one you are working with” – perhaps to identify similar or duplicate content within your organization, find other people with expertise on the same topic, and so on. Here’s a screenshot to give you the idea (click on an image to enlarge):

Find similar docs app

Here’s what my app does:

  1. Analyzes the document content (or the user can choose to analyze just a specific part), and extracts the most relevant keywords/phrases:

    Find similar docs app - keywords

    It uses the top 3 keywords/phrases to search your SharePoint/Office 365 environment for similar content. I wanted something which worked purely on the content of the document, so that the user doesn’t need to tag the document in any way (to find related documents). By using the Office JavaScript API, I’m able to fetch the content of the document (or even just the content selected by the cursor) and then perform some analysis on it. Other, less important keywords/phrases are listed for information.

    1. As a bonus, it also analyzes the document for “sentiment” – and reports whether the document is mainly positive/negative:

      Find similar docs app - sentiment

      At this point, you might be wondering what kind of analysis I do to extract the keywords and determine the sentiment score. I’d love to tell you it’s some amazing algorithm that I spent weeks on and have registered for patent :) In reality of course, there’s actually a bunch of services on the internet that do this kind of thing – so (unfortunately) my smartness only extends to combining the Apps for Office model with one of these services! The one I chose is called AlchemyAPI, and it seems pretty cool. It’s a pay-for service to use in production, but you can obtain a free key for demos/evaluation etc.

  2. As shown in the image above, those top 3 keywords/phrases are used to power a SharePoint search query. The results are shown in the task pane – each result has a “Save” button (floppy disk icon!) next to it:

    Find similar docs app 
  3. The user can save any interesting references for follow-up – these get saved into a list in the user’s OneDrive For Business site (i.e. what we used to call their “My Site” in SharePoint). Here, the user can edit the list item and add notes if they choose – the main thing is they have the link and author details stored for future reference:

    Find similar docs app - saved references in OneDrive

    From here I thought it might be interesting if these “saved references” could be easily searched by other users – almost in a social bookmarking kinda way. So, I thought some enhancement of the search experience would be useful here:

  4. If any “saved references” appear in search results, they are highlighted with some special presentation (mainly just a grey background, some tweaks to the display so that the author’s notes and details of the “linked to” item are shown, along with a custom hover panel):

    Find similar docs app - saved references in search 

I also implemented a custom Result Source for saved references, so that a search only within these items can be done if desired.

Over the next few weeks, I’ll tidy up the implementation somewhat and publish the code somewhere.


  • Apps for Office have huge potential in putting useful business tools in front of users, so that they don’t need multiple windows/applications open. We can put the experience right there in the applications they would already be in – and this makes even more sense when the functionality somehow relates to documents or e-mail
  • The development model is WAY simpler than previous versions of extending Office – if you’re comfortable with a little JavaScript, you can interact with Office documents and implement apps which in Word, Excel, PowerPoint, Outlook and so on. The beauty is that your app will appear in the rich client, as well as web *and* mobile versions where appropriate – there are some caveats here, such as Word task pane apps only appearing in the rich client (for now), but for other permutations (e.g. mail apps and Excel apps) it’s a pretty good story already
  • Deployment and making your app available to end–users is also much simpler than before. There *are* some things to take care of on the IT Pro side (e.g.. deploying Office Trust Center settings through Group Policy), but these aren’t too tricky – I cover them towards the end of the slides

I’ll definitely be looking for opportunities to bring in Apps for Office to my client solutions moving forward – cool stuff indeed.

TechEd Video


Wednesday, 15 October 2014

Speaking at TechEd 2014 on Apps for Office

TechEd Europe is just round the corner (October 28-31, in Barcelona), and recently I’ve been heads-down preparing TEEU_2014_Im speaking_3to speak there on the Office 365 track. Most of the other speakers on this track are Microsoft folks from the Office 365/SharePoint/Exchange product groups (including Rob Howard, Brian Jones, Vesa Juvonen, Steve Walker, Jeremy Thake, Mauricio Ordonez and Sonya Koptyev) so there’ll be some *great* information being dispensed – my aim is to sprinkle in some field experience there too :) As an added bonus, I hear that TechEd will have an announcement or two that developers in this space will like.

My session is on Apps for Office, and I’m fortunate to be co-presenting with Andrew Salamatov from the Exchange team. Andrew is a Program Manager whose team builds the capabilities around mail apps, and he has some very cool stuff to show in this area. Apps for Office in general are becoming very interesting to me, as it kinda feels like there hasn’t been huge amounts of take-up of Apps for Office since the launch – maybe that’s confined to my corner of the Office 365/SharePoint-focused development world, but I think it’s surprising given the power and possibilities there. After all, many users LIVE in Office applications, and from the dev perspective it’s really just web development with some Office-specific JavaScript APIs these days – WAY less scary than previous models :)

Overall, Andrew and I have a couple of cool demos to show, including a Word task pane app which analyses the contents of a document to find you similar items across your SharePoint environment/tenancy whilst you're working with the document.

Here are the session details:

DEV-B307: Weave your enterprise solutions directly into Office products with the App Model (Friday, October 31 12:00 PM - 1:15 PM Room: Hall 8.0 Room D4)

Apps for SharePoint and Apps for Office allow you as an enterprise developer to weave your existing enterprise solutions throughout the Office products that your users live in day to day. In this session you will see compelling real world examples that light up existing Office products in context of key enterprise scenarios. The session will highlight tips and tricks on architecting this solution in Visual Studio along with Patterns & Practices guidance.

  • Understand what scenarios are available to surface your customizations in Office products
  • See real world enterprise scenarios that span multiple products stitching together an enterprise solution
  • Learn tips on how to architect enterprise solutions with the App Model

Hopefully see you there!

Monday, 8 September 2014

JavaScript-based provisioning in SharePoint/Office 365

In this post I want to specifically focus on the idea of JavaScript-based provisioning code – a potentially useful approach which has a place in every modern SharePoint developer’s toolkit. This is the idea of using SharePoint’s JSOM API (i.e. the JavaScript version of CSOM) to perform various one-time configuration steps for SharePoint sites - I was initially somewhat sceptical of this technique (JavaScript? For provisioning? Would that be reliable??), but having seen some team-mates use it with success on projects (kudos guys!), I think it’s worthy of consideration.


For SharePoint developers who are working with SharePoint Online/Office 365, or are working on-premises but choosing not to use full-trust WSPs simply to ensure their solution is “cloud-friendly”, the previous options for custom SharePoint code are no longer available. And some common scenarios still require code – especially when some form of custom site template is being developed, or we are working towards automated configuration of sites.

Some examples of provisioning steps that a developer might need to use code for could be:

  • Binding Managed Metadata/taxonomy fields
  • Setting a theme/Composed Look on a site
  • Setting the master page of a site
  • Modifying navigation settings
  • Branding OneDrive (personal) sites
  • [Optional] Provisioning fields and content types through code, rather than XML

This presents a challenge when server-side SharePoint code cannot be used. As we all know, Microsoft have deprecated the use of sandboxed SharePoint code (which we might previously have used for say, a Feature receiver), and this facility may be turned off in Office 365 in in the future. So, we have to use one of the remote APIs instead – and the code will either run from some other server (e.g. the .NET CSOM) or the user’s browser (JSOM).

Options for remote code

Broadly, the options for remote code include:

  1. Some use of the .NET CSOM API:
    1. Code deployed to a cloud service such as Azure, or running off your own server somewhere. This could be an “App for SharePoint”, or it could just be remote code which does not use app authentication.
    2. A “PowerShell + CSOM” script you wrote
  2. Some use of the JSOM API (i.e. the JavaScript version of CSOM)
  3. Some use of the REST API (either from JavaScript or remote server-side code)

As Microsoft put it, your custom code either needs to move UP to the cloud or DOWN to the user’s browser.

The JavaScript-based (JSOM) option

The idea of using JSOM code for site provisioning steps is a strange one initially. What happens if the user closes their browser as the code is running? Will the site get configured properly? What about if the user is very far away from the server on a slow connection? It was questions like these that made me sceptical initially, but with some safeguards and caveats in place I think it’s an interesting option for remote code. I’ll talk about how it works in a second, but first here’s why I think it’s interesting:

Side-stepping the “server-side remote code” issue

As a modern SharePoint developer, I’m all onboard with apps and remote code. In several cases I’ve implemented remote SharePoint code in Azure (e.g. see my Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps) post). So yep, I’m cool with that stuff! And I’m also down with kids in terms of Microsoft’s OfficeDev Patterns and Practices samples, which cover a ton of scenarios such as those I list above – they’ve effectively written a great wrapper around the .NET CSOM API calls, which make it really easy to get these tasks done.

BUT THIS TYPE OF CODE STILL HAS TO BE DEPLOYED SOMEWHERE! On-premises servers (e.g. some IIS boxes) can bring complexity in terms of high-availability, access outside the firewall, SSL certificates, backup and restore, and so on. Of course, cloud options exist too. If you (or your client) is already using Azure, then GREAT! It’s really not too difficult to get the code deployed and running there, and Microsoft take care of the operational/I.T. Pro factors listed above. But in many of the organizations I’ve worked with, unfortunately Azure (or similar cloud service) is not yet embedded in the I.T. department’s toolkit. There is no Azure subscription, with someone in I.T. already paying the monthly invoice. And there are all sorts of operational I.T. topics to discuss and get agreement on before we could get to that position – who owns responsibility, what kind of data will we be storing there, who has access, how does it fit with any existing SLAs/OLAs, which department/cost centre pays the bill and so on.

Interestingly, working in an “implementation partner” as my team does, we almost have an option of deploying some remote code for a client to Azure and not needing to involve them in this decision. The Azure Websites offering has a free option, and therefore all the billing questions/set-up could be avoided. However, you do not get high availability in free mode – so if the machines your code is running on are being patched, you’ll have downtime. So in the end, unless your code is trivial this probably isn’t an option.

Consequently, the idea of implementing remote code in JavaScript can be quite appealing in some cases.

A robust implementation for JavaScript-based provisioning

So, we’d need some way of making a JavaScript-based approach somewhat reliable. The pattern my guys have used is this:

  • The SharePoint site is created (perhaps from a custom WebTemplate which takes care of some settings in XML) – at this point, the site is NOT fully configured
  • A custom JavaScript file is referenced on every page – this checks to see if set-up steps have been completed yet. If not, the user’s permissions are checked to see if they are a Site Owner and whether we can proceed (with a “A site owner needs to visit this site to configure it” message shown if not).
  • When a user hits the site, our code runs. If we can proceed with set-up, a dialog is shown - “Please wait – we are getting your site ready”
  • The code starts to execute provisioning steps – perhaps setting a Composed Look and/or custom master page, binding taxonomy fields and so on
    • When each step completes, an entry is written to the web property bag e.g. “BrandingApplied = true”
  • When all steps are complete, a success message is shown and the “getting your site ready” dialog is automatically closed. The site is now ready for use.

As you can guess, it’s only when all confirmations are written to the property bag that we no longer attempt to run the steps. This works surprisingly well in practice – users/site owners are comfortable seeing these kind of messages, and having a site owner hit the site initially tends to work out fine, since they are the site creator. If any failures occur, well that’s fine too because the uncompleted steps will be re-tried on the next page load.

To illustrate the experience, here’s what the site owner might see during configuration (shown here on a test page, rather than a “real” site home page):

Dialog large

Of course, you could do something a bit fancier in terms of look and feel.

Some sample code for JavaScript-based provisioning

In my other article Three cloud-friendly ways for developers to provision Managed Metadata fields in SharePoint/Office 365, I show some JavaScript code which does the basics described above – specifically for binding taxonomy fields to MM Term Sets. This is the same sample as in the other article - I’ve included it again below for convenience, but I recommend reading the other article too as it has some extra notes about how the code should be used (like the fact you might want to enhance with JavaScript promises). Note that it doesn’t deal with setting confirmations in the web property bag, but it would be fairly simple to add this. The code will work with Office 365 or on-premises SharePoint:

I guess the main thing to take from this code is the approach - showing the dialog whilst several steps of JSOM code are being executed. In this simple example, when the code completes our taxonomy field will be bound and ready for use – but in our real-life projects obviously many more steps are performed.


Whilst the idea of using JavaScript/JSOM code for provisioning can seem strange, I believe it has it’s place and can be the right choice sometimes. Certainly, if you have no other need for remote code and/or do not have a hosting location (such as Azure or on-premises IIS servers) ready to go in a production sense, then the JSOM approach does offer you a way of running remote SharePoint code without the hassles that come with the alternatives. As always, consider the trade-offs and think about what’s right for you – but hopefully the information here helps you along the way.

Three cloud-friendly ways for developers to provision Managed Metadata fields in SharePoint/Office 365

Provisioning Managed Metadata fields “the developer way” brings some additional complexity where SharePoint full-trust solutions cannot be used, such as Office 365/SharePoint Online deployments or on-premises projects where cloud-friendly techniques are being used. In the past, a developer would use a Feature receiver to bind taxonomy fields to Term Sets – but this isn’t possible in cloud scenarios. I wrote about this previously in Provisioning Managed Metadata fields in Office 365 – here I discussed the problem, and provided an approach we’ve used previously which does work in the cloud. As the article shows, it’s actually possible to use a fully-declarative (XML-based) way of provisioning Managed Metadata fields, so long as some GUIDs which relate to your Office 365/SharePoint instance are inserted into the XML.

Downsides to the declarative approach

The article also discusses the fact that if you work against multiple environments (tenancies or on-premises), at least one different GUID (for the Term Store ID) needs to be used in the XML. This effectively leads to different WSP packages being deployed to different environments – of course, this is not good in ALM terms! In our case, we were fairly comfortable with this trade-off since our TFS automated build process dealt with building packages for different environments, and we didn’t feel much pain. However, without an automated build process it can be more problematic – and now that more time has passed, I think it’s worth stepping back and discussing other options for solving this common problem.

3 ways to bind taxonomy fields

Before delving into the detail of each one, here are the main options as I see them (N.B. I’ve used this list in some of my other articles):

  1. The 100% declarative approach (as discussed above and in my other article)
  2. Using some form of remote server-side .NET CSOM code:
    1. The Microsoft sample in the Office Dev Patterns and Practices samples (previously known as the AMS samples)
    2. Some custom code you wrote
    3. Some custom “PowerShell + CSOM” script you wrote
  3. Using JSOM code running in the user’s browser, which I discuss in a companion article to this one - JavaScript-based provisioning in SharePoint/Office 365

Notably, the last two options aren’t really limited to “the taxonomy field problem” – both of the remote CSOM or JSOM approaches could be used as part of a wider strategy for automated provisioning i.e. any setup steps which require code. My quick list of things which commonly need to be achieved here include:

  • Binding Managed Metadata/taxonomy fields
  • Setting a theme/Composed Look on a site
  • Setting the master page of a site
  • Modifying navigation settings
  • Branding OneDrive (personal) sites
  • [Optional] Provisioning fields and content types through code, rather than XML

So, if you have other items on this list you need to accomplish *or* you just prefer to avoid the declarative approach for Managed Metadata fields because of the ALM trade-offs, then you’re probably looking at a choice between the .NET CSOM or JSOM approaches. For .NET CSOM code, the Office Dev sample code works just fine - but, of course, you have to deploy and host it somewhere, whether it’s a cloud service like Azure or your own IIS servers. And this isn’t necessarily simple for a production-grade enterprise implementation - in the “Side-stepping the server-side remote code issue” section of JavaScript-based provisioning in SharePoint/Office 365, I cover this debate in more detail. As I conclude there:

The idea of implementing remote code in JavaScript can be quite appealing in some cases.

And if steps are taken to increase robustness, I think this approach is OK. So with all that in mind, let’s see what provisioning Managed Metadata/taxonomy fields in JSOM code looks like:

Provisioning with JSOM code

The sample below shows the code, but if you’re interested in using it also see the notes below:

Some notes on this code sample:

  • The code will work with Office 365 or on-premises SharePoint
  • One of the core benefits is that the code allows you to specify the *name* of the Term Set, rather than it’s ID. This is useful since this GUID will be different in different environments, unless you’ve taken specific steps to avoid this (i.e. by creating the Term Set programatically)
  • In fact, three things need to be specified – the name of the Term Set, the locale of the Term Set, and the internal name of the field – you can see these at the top of my code. Ultimately they get passed to the initTaxObjects() method which starts the process
  • I’ve kept the code simple by not using JavaScript promises, but that could be an exercise for the reader if you’d like to enhance it in that way
  • The method is currently set up to process one field at a time, but it could be extended to accept an array of field names/term set IDs
  • Current there’s a delay of 4 seconds inserted for illustration purposes – make sure you remove this if you implement in production :) Similarly there are lots of console.log lines which you might choose to remove
  • You may need to deal with script dependencies differently if not running in a SharePoint page (e.g. you’re code is running in an app)
  • In current form, the code above expects some calling code somewhere to trigger the execution, and a DIV with ID of "jsomProvisioningMessage" (for some simple logging messages) – you could have both of these in a custom web part or Script Editor Web Part added to a page. This is the code:

    <div id="jsomProvisioningMessage" />
    <script type="text/javascript">
    $(document).ready(function () { 

Whether it’s this approach you use or an alternative, if successful your taxonomy field should look like the following (before and after shown):

Before (unbound):

Taxonomy field before binding

After (field is now bound and can be used for tagging):

Taxonomy field after binding


Managed Metadata/taxonomy fields have always been a bit of a pain for developers, because there needs to be an association between the field and a Term Set, and these have different IDs in different environments/tenancies – so it often needs to be dynamically looked up/specified. Assuming we are implementing the site the developer way with automated provisioning, this can be done 100% declaratively (with the ALM trade-off), with .NET CSOM code or with JSOM code. In this post I’ve focused on the JSOM approach – since if you have no other requirements for remote code (and therefore no Azure subscription ready to go), this method can be appealing.

Tuesday, 22 July 2014

Remote Event Receivers on host web gotcha – no context token/ClientContext

I’ve spent quite a bit of time working with Remote Event Receivers in SharePoint 2013/Office 365, and in the early days I hit an issue that initially made me think they didn’t work properly at all. Now with some help, I realize that mainly the code just has to be written a certain way - but it’s also true to say that there is a limitation which can affect some things you might want to do. In any case, this is a gotcha that I’m starting to see other people hit so I think it’s worth talking about it.

Specifically the issue occurs when:

  • The event receiver is on the host web (rather than the app web)
  • The event receiver uses CSOM code to talk to SharePoint (rather than just using NON-SharePoint code)

Issue symptoms

When you’re trying to use RERs on the host web, what you might see is that whilst your event receiver always fires (and your remote code does execute), your code fails to authenticate back to SharePoint. You’ll get a 401 error when trying to obtain or use a ClientContext object – this could manifest itself with one of these in your code:

System.NullReferenceException: Object reference not set to an instance of an object

..but if you look at the HTTP requests/responses, you’ll see the HTTP 401 (Access Denied).

This means you effectively can’t do anything with SharePoint – and often, of course, the whole point of your event receiver is that it’s going to do something in SharePoint! Perhaps you need to create, update or delete an item somewhere, or perform some other action related to sites, users or data in SharePoint.

Other info:

  • No context token
    • If you do some debugging, you’ll find that the event receiver does NOT receive a valid context token from SharePoint/Office 365 - SPRemoteEventProperties.ContextToken is an empty string. This causes all the later code to fail. At this point, you probably think there’s a bug with SharePoint/Office 365 – even at the end of all this, I’m personally not sure whether there is a bug or it’s just a “by-design” limitation. Fortunately, as we will see there is a workaround which works for many (most?) scenarios of this type.
  • The same code works on the app web
    • If you do some wider testing, perhaps to see if there is a code issue – you’ll find the very same code works fine in other contexts. For example, if you use the same code in a RER on the app web (e.g. “ItemAdded” on a list in the app web), everything works fine. The issue is purely related to RERs on the host web.
  • No change if you use ProcessOneWayEvent (asynchronous events such as added, deleted etc.) or ProcessEvent (synchronous events such as adding, deleting etc.)
    • Remote Event Receivers are similar to traditional full-trust event receivers, in that both async and sync events can be used. An example here would be the “added” and “adding” events respectively – these correspond to the ProcessOneWayEvent and ProcessEvent methods in a Remote Event Receiver class. For reference, the issue we’re discussing here manifests itself in both cases.

So those are the symptoms. Before we dig into the “Why it happens” and “The workaround” sections, it’s worth reminding ourselves of some aspects of remote code and authentication in particular. Feel free to skip forward if you’re familiar with this territory.

A reminder on authentication of remote code

For those just starting out with Remote Event Receivers (or SharePoint apps/remote code in general), it’s worth noting that an app is required for any form of remote code. More correctly, we could expand this to say any remote code which runs on the server-side and uses “app authentication” - as opposed to “user authentication” where a username/password is specified. In any case, this can seem strange for some things like Remote Event Receivers, where there isn’t really an “app” (e.g. with some pages) that the user will enter/click around etc. All we have for a RER is a remote WCF service with no front-end! But, this is the way the trust model works for remote code – so the first thing to say is that if you don’t have an app registered and installed/trusted for your remote code, it will not work.

If we were to summarize common authentication options for remote SharePoint code, we could break it down like this:

  • User authentication
    • The username/password of a “named user” is passed – usually a very poor approach that brings security risks. Effectively this led OAuth to be invented as an alternative.
  • App authentication – this breaks down into:
    • User + app authentication – the default. Permissions of both the user *and* the app are checked
    • App-only authentication – the permissions of just the app are checked (thus allowing operations that the user themselves does not have permission to do)

Why it happens

If you’re hitting the issue, your RER most likely has code which instantiates the ClientContext object in a certain way - specifically, you’ll be using “user + app” authentication. The vast majority of MSDN and blog samples use this approach. This is ultimately the problem -  “user + app” authentication does not currently work for events related to the host web. My testing shows this is the case for both Office 365 and SharePoint 2013 on-premises installations. Along the journey of realising this, I tried some different coding approaches – and I see other developers perhaps doing the same (e.g. in forum posts). Since there are various ways the CSOM ClientContext object can be obtained, you might try different code for this – but several cannot be used in a RER and will fail. The table below shows some of the “wrong” approaches.

Code approaches which will NOT work:


Code sample

Why not

Using the typical RER approach ClientContext clientContext = TokenHelper.CreateRemoteEventReceiverClientContext(properties) This is the approach that “should work” (and the approach that most samples use). However, it only works for events on the app web – NOT events on the host web.
Creating ClientContext using constructor

ClientContext clientContext = new ClientContext(properties.ItemEventProperties.WebUrl)

The context is not obtained in a way which deals with authentication. This is what the TokenHelper methods are for.

Sidenote – it would be possible to authenticate with user authentication instead of app authentication, if the username/password of a named account was attached using the SharePointOnlineCredentials object. But that’s lame, and a big security risk! Instead, we want to use OAuth (via app authentication) of course, to avoid storing/passing passwords.
Using the “app event” approach ClientContext clientContext = TokenHelper.CreateAppEventClientContext(properties, false) The context is obtained in a way which is only appropriate for app events (e.g. AppInstalled/AppUpgraded/AppUninstalled etc.) Since we are using an event on the host web, authentication does not work.

As the table says, the one that *should* work is the first one (TokenHelper.CreateRemoteEventReceiverClientContext()) – but this relies on “user + app” authentication which appears to be the problem. So to work around this we need a different approach. 

The workaround

Instead of “user + app” authentication we can use “app-only” authentication. This requires 2 things:

  1. The app permission policy to specify that app-only calls are allowed (something the person installing the app must agree to). This is enabled in the app manifest in Visual Studio:

    Allow app-only auth
  2. The code in the app (i.e. in the Remote Event Receiver) to obtain ClientContext using an app-only access token. The expected way to do this in C# is to use the correct TokenHelper methods – big thanks to Kirk Evans for helping me with this. The code should look like this:

    ** N.B. There is a code sample here but it will not show in RSS Readers - click here for full article **
If you ensure these two things are in place, your Remote Event Receiver will work fine.

The limitation – when the workaround is no good

So, that’s the answer then. However, you might hit occasions where the app-only approach isn’t really what you want. If your remote code writes any data to SharePoint (e.g. adds a list item), you’ll notice that the SharePoint user interface makes it clear that the list item was added/modified by an app. Specifically, the username will be in the following format:

Last modified at [time] by [app name] on behalf of [username]

This is a pretty useful feature for some business requirements, since it makes it clear that although the named user was involved, they didn't make the change in SharePoint "directly". Unfortunately, use of the app-only authentication route to solve our problem changes this – specifically, with app-only auth only the app name is recorded as the user who made the change. Effectively, we lose the information which tells us which actual person was involved in the change - in some circumstances, this could be a problem if some kind of audit trail is required. To make things clear, here are two screenshots which show the difference:

List item added by "app + user" authentication:

List item added with App and User context

List item added by "app-only" authentication:

List item added with App Only context

Related sidenote (for the curious!):

If you’re wondering how SharePoint/Office 365 deals with this under the covers, you’ll find that all lists now have 2 new hidden columns - “App Created By” and “App Modified By”. These store a reference to an entry in the User Information list in the current site (just like other Person/Group columns):

AppCreatedByModifiedByColumnsOnList AppCreatedByModifiedByColumnsOnList2

So now you know ;)


It’s pretty easy to get things wrong with Remote Event Receivers, and the pitfall I’ve talked about concerning authentication and CSOM code is definitely a big gotcha – I certainly see other people hitting this. If you have remote code which modifies data in SharePoint somewhere, the way around the problem is to use app-only authentication - by ensuring the app’s permission policy allows this *and* also writing your CSOM code to authenticate in this way. This can be done by obtaining an access token using the TokenHelper.GetAppOnlyAccessToken() method, and then obtaining a ClientContext object using this token.

However, with this approach comes a trade-off - the loss of user information (in “pseudo-audit” terms) related to the user behind the change. Developers should bear this in mind when working with Remote Event Receivers.