Tuesday, 22 July 2014

Remote Event Receivers on host web gotcha – no context token/ClientContext

I’ve spent quite a bit of time working with Remote Event Receivers in SharePoint 2013/Office 365, and in the early days I hit an issue that initially made me think they didn’t work properly at all. Now with some help, I realize that mainly the code just has to be written a certain way - but it’s also true to say that there is a limitation which can affect some things you might want to do. In any case, this is a gotcha that I’m starting to see other people hit so I think it’s worth talking about it.

Specifically the issue occurs when:

  • The event receiver is on the host web (rather than the app web)
  • The event receiver uses CSOM code to talk to SharePoint (rather than just using NON-SharePoint code)

Issue symptoms

When you’re trying to use RERs on the host web, what you might see is that whilst your event receiver always fires (and your remote code does execute), your code fails to authenticate back to SharePoint. You’ll get a 401 error when trying to obtain or use a ClientContext object – this could manifest itself with one of these in your code:

System.NullReferenceException: Object reference not set to an instance of an object

..but if you look at the HTTP requests/responses, you’ll see the HTTP 401 (Access Denied).

This means you effectively can’t do anything with SharePoint – and often, of course, the whole point of your event receiver is that it’s going to do something in SharePoint! Perhaps you need to create, update or delete an item somewhere, or perform some other action related to sites, users or data in SharePoint.

Other info:

  • No context token
    • If you do some debugging, you’ll find that the event receiver does NOT receive a valid context token from SharePoint/Office 365 - SPRemoteEventProperties.ContextToken is an empty string. This causes all the later code to fail. At this point, you probably think there’s a bug with SharePoint/Office 365 – even at the end of all this, I’m personally not sure whether there is a bug or it’s just a “by-design” limitation. Fortunately, as we will see there is a workaround which works for many (most?) scenarios of this type.
  • The same code works on the app web
    • If you do some wider testing, perhaps to see if there is a code issue – you’ll find the very same code works fine in other contexts. For example, if you use the same code in a RER on the app web (e.g. “ItemAdded” on a list in the app web), everything works fine. The issue is purely related to RERs on the host web.
  • No change if you use ProcessOneWayEvent (asynchronous events such as added, deleted etc.) or ProcessEvent (synchronous events such as adding, deleting etc.)
    • Remote Event Receivers are similar to traditional full-trust event receivers, in that both async and sync events can be used. An example here would be the “added” and “adding” events respectively – these correspond to the ProcessOneWayEvent and ProcessEvent methods in a Remote Event Receiver class. For reference, the issue we’re discussing here manifests itself in both cases.

So those are the symptoms. Before we dig into the “Why it happens” and “The workaround” sections, it’s worth reminding ourselves of some aspects of remote code and authentication in particular. Feel free to skip forward if you’re familiar with this territory.

A reminder on authentication of remote code

For those just starting out with Remote Event Receivers (or SharePoint apps/remote code in general), it’s worth noting that an app is required for any form of remote code. More correctly, we could expand this to say any remote code which runs on the server-side and uses “app authentication” - as opposed to “user authentication” where a username/password is specified. In any case, this can seem strange for some things like Remote Event Receivers, where there isn’t really an “app” (e.g. with some pages) that the user will enter/click around etc. All we have for a RER is a remote WCF service with no front-end! But, this is the way the trust model works for remote code – so the first thing to say is that if you don’t have an app registered and installed/trusted for your remote code, it will not work.

If we were to summarize common authentication options for remote SharePoint code, we could break it down like this:

  • User authentication
    • The username/password of a “named user” is passed – usually a very poor approach that brings security risks. Effectively this led OAuth to be invented as an alternative.
  • App authentication – this breaks down into:
    • User + app authentication – the default. Permissions of both the user *and* the app are checked
    • App-only authentication – the permissions of just the app are checked (thus allowing operations that the user themselves does not have permission to do)

Why it happens

If you’re hitting the issue, your RER most likely has code which instantiates the ClientContext object in a certain way - specifically, you’ll be using “user + app” authentication. The vast majority of MSDN and blog samples use this approach. This is ultimately the problem -  “user + app” authentication does not currently work for events related to the host web. My testing shows this is the case for both Office 365 and SharePoint 2013 on-premises installations. Along the journey of realising this, I tried some different coding approaches – and I see other developers perhaps doing the same (e.g. in forum posts). Since there are various ways the CSOM ClientContext object can be obtained, you might try different code for this – but several cannot be used in a RER and will fail. The table below shows some of the “wrong” approaches.

Code approaches which will NOT work:


Code sample

Why not

Using the typical RER approach ClientContext clientContext = TokenHelper.CreateRemoteEventReceiverClientContext(properties) This is the approach that “should work” (and the approach that most samples use). However, it only works for events on the app web – NOT events on the host web.
Creating ClientContext using constructor

ClientContext clientContext = new ClientContext(properties.ItemEventProperties.WebUrl)

The context is not obtained in a way which deals with authentication. This is what the TokenHelper methods are for.

Sidenote – it would be possible to authenticate with user authentication instead of app authentication, if the username/password of a named account was attached using the SharePointOnlineCredentials object. But that’s lame, and a big security risk! Instead, we want to use OAuth (via app authentication) of course, to avoid storing/passing passwords.
Using the “app event” approach ClientContext clientContext = TokenHelper.CreateAppEventClientContext(properties, false) The context is obtained in a way which is only appropriate for app events (e.g. AppInstalled/AppUpgraded/AppUninstalled etc.) Since we are using an event on the host web, authentication does not work.

As the table says, the one that *should* work is the first one (TokenHelper.CreateRemoteEventReceiverClientContext()) – but this relies on “user + app” authentication which appears to be the problem. So to work around this we need a different approach. 

The workaround

Instead of “user + app” authentication we can use “app-only” authentication. This requires 2 things:

  1. The app permission policy to specify that app-only calls are allowed (something the person installing the app must agree to). This is enabled in the app manifest in Visual Studio:

    Allow app-only auth
  2. The code in the app (i.e. in the Remote Event Receiver) to obtain ClientContext using an app-only access token. The expected way to do this in C# is to use the correct TokenHelper methods – big thanks to Kirk Evans for helping me with this. The code should look like this:

    ** N.B. There is a code sample here but it will not show in RSS Readers - click here for full article **
If you ensure these two things are in place, your Remote Event Receiver will work fine.

The limitation – when the workaround is no good

So, that’s the answer then. However, you might hit occasions where the app-only approach isn’t really what you want. If your remote code writes any data to SharePoint (e.g. adds a list item), you’ll notice that the SharePoint user interface makes it clear that the list item was added/modified by an app. Specifically, the username will be in the following format:

Last modified at [time] by [app name] on behalf of [username]

This is a pretty useful feature for some business requirements, since it makes it clear that although the named user was involved, they didn't make the change in SharePoint "directly". Unfortunately, use of the app-only authentication route to solve our problem changes this – specifically, with app-only auth only the app name is recorded as the user who made the change. Effectively, we lose the information which tells us which actual person was involved in the change - in some circumstances, this could be a problem if some kind of audit trail is required. To make things clear, here are two screenshots which show the difference:

List item added by "app + user" authentication:

List item added with App and User context

List item added by "app-only" authentication:

List item added with App Only context

Related sidenote (for the curious!):

If you’re wondering how SharePoint/Office 365 deals with this under the covers, you’ll find that all lists now have 2 new hidden columns - “App Created By” and “App Modified By”. These store a reference to an entry in the User Information list in the current site (just like other Person/Group columns):

AppCreatedByModifiedByColumnsOnList AppCreatedByModifiedByColumnsOnList2

So now you know ;)


It’s pretty easy to get things wrong with Remote Event Receivers, and the pitfall I’ve talked about concerning authentication and CSOM code is definitely a big gotcha – I certainly see other people hitting this. If you have remote code which modifies data in SharePoint somewhere, the way around the problem is to use app-only authentication - by ensuring the app’s permission policy allows this *and* also writing your CSOM code to authenticate in this way. This can be done by obtaining an access token using the TokenHelper.GetAppOnlyAccessToken() method, and then obtaining a ClientContext object using this token.

However, with this approach comes a trade-off - the loss of user information (in “pseudo-audit” terms) related to the user behind the change. Developers should bear this in mind when working with Remote Event Receivers.

Monday, 14 July 2014

Renewed as SharePoint MVP for 7th time


I received the happy news recently that I continue to be a SharePoint MVP for another year – the 7th time for me. As always, it’s been great interacting with the community over the last 12 months. Every year I like to jot down on my blog the community “things” I did in the previous year - this list is mainly for me rather than you, but here are some the things I got up to:

It’s been a privilege to be part of the community, and I hope to continue to be useful over the next 12 months!

Thursday, 26 June 2014

Step-by-step video: remote SharePoint code in Azure

As I noted in my last post Modern SharePoint Development – techniques for off-box code, there is something of a movement in the SharePoint development world to what is often termed “the Cloud Application Model”. In a nutshell, this is the approach that many development tasks *must* use for Office 365, and *can* use for on-premises SharePoint. Specifically, the Cloud Application Model is about having custom code run “off-box” to SharePoint using the remote APIs rather than than the server-side API – after all, in Office 365 it’s simply not possible to deploy custom SharePoint code to Microsoft’s servers (this is not permitted because some bad code could affect the service for many clients). But as I noted last time, there can be benefits to this approach even in on-premises projects - such as:

  • Greater isolation – an app running “off-box” cannot bring down SharePoint for the entire organization
  • Ease of upgrade – less customizations made in SharePoint, the easier upgrade (e.g. to SharePoint 2015) becomes
  • Possibility of transitioning to cloud – if the organization is even half-considering a move to Office 365 (e.g. in the next few years), building solutions the “cloud-friendly” way means no/minimal re-work is required to move. Certainly we have more than one client who has asked us to build this way – because they want to leave the door open for Office 365 (or hybrid) as they move forward

Some different flavors of off-box code include:

  • Client code running in the browser - use of the JavaScript Object Model (JSOM), or the REST API from JavaScript
  • Remote server-side code – use of the .NET Client Object Model (CSOM), or the REST API from the server

For the latter case, you need to identify a non-SharePoint server where the CSOM code can run. Common choices here include using Azure (or similar cloud service) or providing your own IIS servers – since an app is typically required for the “trust” of the remote code, this is the notion of “provider-hosted apps” which you have probably heard about. As I’ve often said, I really like the Azure option because so many infrastructure things are taken care of (SSL, High Availability, ability to scale up, backup/restore, published to the internet and so on).

As a reminder, Microsoft supply *lots* of useful remote code in their Office App Model Samples pack (known as the “AMS samples”) – this deals with many common scenarios you might encounter. But you still need to figure out where to run the code from, and go through the “remote code” configuration steps to enable this (or any remote server-side code) to run.

A guide – my subtitled video showing end-to-end config of remote code in Azure

There are various resources related to apps and remote code, but the steps are fiddly and sometimes it’s difficult to follow the entire process. So I decided to produce a video, with lots of captions explaining each step – the scenario I’m using is configuring a Remote Event Receiver (RER) in Azure. Note, however, that perhaps 90% of the steps apply to *any* remote server-side code scenario where the hosting location is Azure. So if you have any kind of provider-hosted app and want to run it from Azure, you’ll need the same set of steps (even though you’re not using RERs).

The image below might give you a sense of the video/captions:

COB RER video snapshot


If you have any feedback/questions, please publish them here and I’ll try to respond. Thanks!

Monday, 16 June 2014

Presentation deck: Modern SharePoint development – techniques for off-box code

WP_20140610_003I recently gave a couple of talks at the SharePoint Evolution 2014 conference (a roadshow no less! Check out the tour bus to the left!) I spoke in London and Cambridge - I had a great time as a speaker, and had some great conversations with attendees.

I created some new materials for these talks (slide decks/demos etc.) and will publish links to these over the next couple of posts, starting here. For the first time, I’ve recorded my demos and published the videos to YouTube (links in the slide deck) – so hopefully much of the information conveyed in the demos is available too.

"Modern SharePoint development - techniques for moving code off SharePoint servers"


Almost 2 years after many of us first began to get exposure, it’s fair to say that “SharePoint apps” are not going away. Although Microsoft certainly have changed tack on some things in the SharePoint dev world, it does seem that the apps/remote code model is something that Microsoft are committed to and will continue to invest in. Whilst they’re not right for every scenario, I generally do also buy into the message that building solutions in this way can help even for on-premises deployments. Some specific points here include:

  • Greater isolation – an app running “off-box” cannot bring down SharePoint for the entire organization
  • Ease of upgrade – less customizations made in SharePoint, the easier upgrade (e.g. to SharePoint 2015) becomes
  • Possibility of transitioning to cloud – if the organization is even half-considering a move to Office 365 (e.g. in the next few years), building solutions the “cloud-friendly” way means no/minimal re-work is required to move. Certainly we have more than one client who has asked us to build this way – because they want to leave the door open for Office 365 (or hybrid) as they move forward

Speaking to attendees I realise that for many folks, Microsoft’s push to this “Cloud Application Model” hasn’t really resonated yet. They ask “why change, especially when new skills are required and many common SharePoint things become more difficult to implement?”. Well, I do agree that it doesn’t always make sense. No-one is saying you have to do things using these techniques. If, for example, the organization is sure that the cloud isn’t the right solution for them in the next few years, and they are happy with the deal they get in managing SharePoint (including any upgrade projects they might undertake), then fine – you might be able to deliver more business value using farm solutions and full-trust code.

In general, however, I think at least an awareness of this emerging trend (towards cloud-friendly SharePoint development) is beneficial. Hopefully this slide deck is of some use here, but there are many other good resources too – I list some at the end of this article


There are 3 main areas – each with an accompanying demo/YouTube video:

  • Running code in Azure (Remote Event Receivers used as example)
  • Using "PowerShell + CSOM" scripts
  • Analysis of Microsoft's "AMS Samples"
Original abstract

Those implementing Office 365 know that custom code cannot run on the SharePoint servers, and remote code alternatives must be used. However, it can be valuable to adopt the same "cloud-friendly" development techniques even for on-premises implementations. In addition to increasing platform stability, this enables you to build in flexibility to your deployment - leaving the door open to a potential move to Office 365 in the future without major re-engineering. With several code-focused demos, this session examines approaches for switching to "off-box" code - including Remote Event Receivers, PowerShell and CSOM scripts, and a nod to JavaScript-based approaches. We’ll also take an in-depth look at Microsoft’s Office App Model Samples (AMS) and why you should care about them.

Resources/further reading

Friday, 30 May 2014

Speaking on Office 365 APIs and “modern” SharePoint dev techniques at the SharePoint Evolutions 2014 Roadshow

Speaker_web_banner I’ve been in head-down mode recently preparing two new talks, which I’m looking forward to giving at the upcoming SharePoint Evolutions Roadshow here in the UK. It’s going to be a great event – pretty sure it’s the first time I’ve heard of a “roadshow” conference, but it’s a great idea! As usual for the SharePoint Evolutions event, there’s a great line-up of speakers from the UK, Europe, the U.S. and elsewhere. The roadshow visits 12 UK locations in total – I’m speaking at London and Cambridge, but many of the other speakers (including the U.S. guys) are doing LOTS of locations! Even so, no two locations will have exactly the same speakers/content, so the idea of going to more than one day works out too.

I’ll be speaking on:


An intro to the new developer APIs for Office 365 – London, June 10, 3:45pm

[This is a joint session with Wes Hackett]

Microsoft have stated that they will move away from SharePoint, Exchange and Lync APIs towards generic Office 365 APIs. The preview version of the first APIs were announced in March 2014, and cover several core building blocks for user collaboration - including mail, OneDrive files, calendar entries and contacts. Developers who need awareness of Office 365/hybrid scenarios can expect more such APIs in the future. We’ll demo the core REST API, the recently released .NET client API (which helps simplify authentication etc.) and even show accessing Office 365 from a “multi-device hybrid app” (aka Cordova) using the new JavaScript API for Office 365.

Get on the bus early - come learn about the APIs and the new permissions model ("the Common Consent framework") in this code-focused session.

Modern SharePoint Development - techniques for moving code off SharePoint servers – Cambridge, June 11, 1pm

Those implementing Office 365 know that custom code cannot run on the SharePoint servers, and remote code alternatives must be used. However, it can be valuable to adopt the same "cloud-friendly" development techniques even for on-premises implementations. In addition to increasing platform stability, this enables you to build in flexibility to your deployment - leaving the door open to a potential move to Office 365 in the future without major re-engineering. With several code-focused demos, this session examines approaches for switching to "off-box" code - including Remote Event Receivers, PowerShell and CSOM scripts, and a nod to JavaScript-based approaches. We’ll also take an in-depth look at Microsoft’s Office App Model Samples (AMS) and why you should care about them.

It’s not too late - convince your boss you need to attend for a day!

Since the roadshow is based around the idea of a series of one day events, it’s perhaps less of a big deal to escape from the office and clients to gain some key knowledge (than say, a 3 day conference). As you’d expect, the costs are also lower (£125 per day) and the organisers (Combined Knowledge) are even offering a “register 1, get 1 free” offer for a limited time as we close in on the event! 

More info at:


Thursday, 1 May 2014

Office 365 hybrid and search – presenting results from both on-premises and SharePoint Online sites

In my previous post about Office 365 and on-premises SharePoint hybrid deployments, we discussed the idea that, whilst it can be the right solution in many cases, a hybrid deployment is NOT simple and really only encompasses search, BCS, and Duet (for SAP) integration. Other aspects around document management, sites/navigation, metadata, customizations, and perhaps any other thing you care to mention are not synchronized or joined-up in any way. As a reminder, these articles are structured like this:

  1. Office 365 SharePoint hybrid – what you DO and DON’T get
  2. Office 365 hybrid and search – presenting results from both sides [this article]

Even in search, one of the cornerstones of hybrid, what you get might not be what you’re hoping for – results from the “remote” environment are displayed in a separate Result Block or Promoted Result. Here’s a reminder of the (more powerful) Result Block approach:

SharePoint hybrid search results - result block

Clearly this isn’t the best user experience. And the alternative, Promoted Results, are more about defining an individual result to appear at the top for a given search (i.e. a Best Bet) – so that’s great for individual results but not entire sets.

Last time, I discussed some of the questions and issues that can be raised by the current experience:

  • How many results should be shown (from each source) on page 1?
    • HINT – you’re only allowed a maximum of 10 for a Result Block anyway :)
  • What about paging? What happens when I go to page 2?
  • How can I easily see more results from both sources?
  • How do I determine which results are more relevant, from the local or remote environment?
  • Are there any options for seeing the results in a more integrated way?
Important news on this topic (April 2014)
At the recent SharePoint Conference 2014, Microsoft stated that they were looking at this problem (of disjointed search results in hybrid deployments), and hope to have a solution later this year. So, hopefully this whole problem will go away at some point. In any case, I think it’s still worthwhile discussing the current situation and possible interim techniques to work around it.


Investigation – can results be merged using code?

So, given the limitations of of the current out-of-the-box user experience, I was keen to see if there are other options. Indeed, for some things we are putting together for my current client, the requirement is very much “just search across everything and show me the most relevant things!” I think there’s nothing wrong with wanting that as the user experience. So as a developer, I wondered if going direct to the search API would provide any other options – perhaps the standard search page is built in a certain way, but the API has more flexibility. I’d also previously noticed a very interesting property in the API, so I started there:

Investigation 1 - using the EnableInterleaving property on the Query class

The SharePoint search API has an interesting option which can be used when writing custom code for SharePoint search. The name of the property in the API is “EnableInterleaving” on the KeywordQuery object – this sounded promising in terms of potentially being a native way of merging the result sets. The Microsoft documentation is (currently) not hugely clear on what the setting does, at least not to me anyway:

“A Boolean value that specifies whether the ResultTable objects in the ResultTableCollection produced by running this query should be interleaved.” (from http://msdn.microsoft.com/en-us/library/office/microsoft.sharepoint.client.search.query.query.enableinterleaving.aspx).

To start testing, I configured hybrid search in my dev environment – I used the outbound hybrid model, where the search center site exists on-premises but reaches out to Office 365/SharePoint Online to bring in results from there. (TechNet documentation is at Configure hybrid search for SharePoint Server 2013, or numerous sessions at the SharePoint Conference on this next week for those attending). At this point, the out-of-the-box search results page gave me the experience shown in the image at the start of the article, with the separate results blocks (highlighted in red/blue boxes).

I then wrote some custom code which allowed me to test this config option – effectively I have a custom web part with some CSOM (JSOM) code which runs a search query using the API. I have some URL parameters which enable me to specify the query text and whether EnableInterleaving is enabled or not. Unfortunately, EnableInterleaving does NOT interleave results between Office 365 and on-premises SharePoint. The main finding is that multiple result sets are still returned (i.e. for Office 365 and on-premises), and results are not combined together in any way. One difference I did observe, is that is an additional set of results is returned (a ResultTable) for personalized results (“PersonalFavoriteResults”), but it’s empty for me (most likely because this is a non-production environment where analytics aren’t in use. I’m sure SharePoint does natively populate/maintain this under real usage – it stores results previously clicked on by the user for this search).

Conclusion: - the EnableInterleaving property does NOT help us combine search results across Office 365 and on-premises SharePoint.

Investigation 2 – manually merging results in custom code

Anyone very familiar with SharePoint search will know that for any given query (e.g. a search for the query text “SharePoint”), the search engine can return different “tables” of results – perhaps think of these as “flavors” of results. They are:

  • Relevant results (“RelevantResults”)
  • Best Bet results (“SpecialTermResults”)
  • Personal favorite results (“PersonalFavoriteResults”)

Usually it’s the “relevant results” that we care most about – this table has the core set of results for the query. If our goal is to combine results from Office 365 and on-premises SharePoint, really what we want is a single “relevant results” table containing items from both locations. However, when we’re in hybrid mode, what we actually get by default is two separate “relevant results” tables – the image below shows my custom code rendering these tables, and you can see the similarities between this and the out-of-the-box search results page (which displays a separate Result Block for the Office 365 results):

Hybrid search - EnableInterleaving on 2

So, if we want to display combined results (rather than separated), we *could* write some code to merge the result tables together (e.g. in memory). But what logic would we use? How should the results be sorted? Well, one of the columns returned in the result tables is “Rank” – this is the “relevancy score” of the result compared to the query which was used (all search engines have such a calculation – like Google’s PageRank). So the question is – can we use Rank to compare results from different search indexes? Is it valid to merge the two result sets together, and then sort the full list by rank? After all, this can definitely be achieved with custom code – here’s my custom web part which goes direct to the search API to do this:

Hybrid search - custom interleaving 2

I had a feeling it wasn’t valid to do this. So I asked some folks including Microsoft, other MVPs etc. for input and my suspicions were confirmed – Wictor summed it up best with this response:

“Interleaving results from two different indexes would produce very strange results. Since the ranking is based on the analytics you basically need to "normalize" the rank and analytics data between the two (or more) different indexes. Assume the user behaves differently in the different environments (which is not uncommon - one environment might be collaboration and the other some other workload such as BI).”

So in effect, you can mash up the results, but not in a way which gives you the most relevant results first (across both environments) – the sorting by rank is artificial, and shouldn’t be trusted. However, it is worth considering that if you *don’t* care about relevance/rank, then this custom approach works fine. For example, if the requirement is to display results sorted by last modified date (or by something else, such as author, site URL or whatever) then happy days! You could use this custom code approach to merge results from your Office 365 and on-premises sites just fine in those cases.

The code

If you’re interested in this approach, here’s the code to do it. I used JSOM code to execute the search, but REST should be another option. When I have the two result tables, I iterate the results in each and add to a combined collection. Finally I sort the results, with a little help from Underscore.js. All this JavaScript is referenced by a custom web part, that I deploy to my on-premises site and add to a page there. So long as hybrid is properly configured, my code can reach across to the associated Office 365 environment and obtain search results from there too.

This is the initial code to execute the search – the main thing to note here is that there is NOTHING different needed in the code to “use hybrid mode” or “enable hybrid” or anything. The results automatically come back from both sources if the infrastructure is configured. You might notice I pick up some URL parameters for the query text and EnableInterleaving properties – you would need to integrate this with your UI as appropriate:

In the success handler, some different code *is* required here. Since 2 result tables are returned, here we do some processing to combine the results into a new JavaScript array, and implement any sorting. The code below provides different options for sorting – by rank (even though we’re saying it’s not valid under hybrid), and last modified date:

..and here’s the full JavaScript code with all my helper functions etc. (NOTE: I’m still leaving it as an exercise for the reader to implement this somewhere – I used two custom web parts which referenced this JavaScript, so providing the exact implementation you need is up to you):


At the current time the user experience for search in hybrid mode isn’t necessarily as optimal as we would like it to be, because results are displayed separately for Office 365 sites and on-premises SharePoint sites. In some cases, users will just want to “search across everything” and not care where the results live.

However, under some circumstances you might be able to use custom code to “join-up” the results. Specifically, this should work well if you are happy to display results sorted by something *other* than rank – perhaps last modified date, the author name or some other piece of metadata. Sorting by rank across two SharePoint environments is not valid, since for one reason, rank uses SharePoint’s analytics framework, which operates within the scope of an individual environment only (i.e. it doesn’t do anything magical for hybrid deployments).

In the future, Microsoft will hopefully provide capability which will address this problem “natively”, without the need for custom code. Then, it will be possible to show results across Office 365 sites and on-premises SharePoint sites together. Until then, the solution presented here might be an option for you.

Monday, 28 April 2014

Speaking at the European SharePoint Conference 2014

ESPC Speaker GraphicWe’re now just a week away from the European SharePoint Conference – held in Barcelona, the event looks great and I’m looking forward to being part of it. I understand this will be the biggest SharePoint event in Europe this year, and it certainly looks like there are some great speakers and sessions - including coverage of some stuff only recently announced at the Las Vegas conference.

The conference runs from 5th-8th of May 2014 – here is the full conference programme.

I’m presenting two sessions on the developer track:

“Developer Decisions, Tips and Tricks - Lessons Learnt from Office 365 Projects” (W13 – Wednesday May 7, 11:45)

As a developer or technical lead, your early Office 365 projects are likely to throw up some interesting questions. Should you avoid the sandbox? How should test environments be handled in the cloud? How should a site template be implemented? And just how do you provision Managed Metadata fields, when the on-premises techniques cannot be used? This session walks through the "dev strategy" decisions we made at Content and Code, and why. Over the course of several demos, we'll also discuss apps, automation scripts and also advanced techniques such as Continuous Integration for Office 365.

“Using JavaScript Templates To Customise the SharePoint 2013 User Interface” (TH20 – Thursday May 7, 14:00)

Developers often have a need to amend the HTML rendering of SharePoint, especially around lists and list forms, but in previous versions of SharePoint this was not always easy. Many developers resorted to using JavaScript to manipulate the page after it had loaded, even though this was potentially unreliable and unsupported. SharePoint 2013 solves this problem by providing the ability to specify a JavaScript 'template override' for several user interface elements. Even better, these can be applied globally or locally as needed. Come and learn how to transform the SharePoint interface the supported way!

Hopefully see you there!

Tuesday, 15 April 2014

Speaking at the SP24 online conference this week

Later this week, on April 16 and April 17, the SP24 online conference takes place and runs for, you guessed it, 24 hours. The conference is FREE. I highly recommend registering and tuning in to some sessions if you can – after all, isn’t that what that second monitor is for? :)


I’m proud to be speaking – the event looks great, and the amount of preparation by the organisers is pretty astounding. In addition to simply being streamed, each session gets a “session room” where questions can be put to the speaker, a Twitter feed runs and any supporting documents supplied by the speaker can be downloaded. For example, here is my session room:

COB SP24 session room

Not just any old online conference then – as a speaker I’m very impressed.

There’s a LONG of great speakers (Bill Baer, Jeremy Thake, Joel Oleson, Mirjam van Olst, Laura Rogers, Todd Baginski, Marc Anderson, Rene Modery, Chris Givens, Sam Hassani, Paolo Pialorsi, Dux Raymond Sy, Liam Cleary, Agnes Molnar and *many* more) – all pulled together into something like a television channel by a handful of anchors (presenters). There is a mix of live sessions (beamed to the world using Google Hangouts) and pre-recorded ones, split across two tracks:

  • Technical
  • Business

My session

My session is:

Office 365 - developer lessons learnt, tips and tricks

As a developer or technical lead, your early Office 365 projects are likely to raise some big questions. How should you interpret Microsoft's mixed messages on sandbox solutions? How should test environments be handled in the cloud? Do you need to run more than one Office 365 tenancy? When you can't deploy files to the LAYOUTS directory, where should global assets such as CSS and JavaScript be stored? What about automation, when only 30 PowerShell cmdlets are available in SharePoint Online? And just how do you provision Managed Metadata fields, when the on-premises techniques cannot be used? This session walks through the “dev strategy” decisions we made at Content and Code, and why. Over the course of several demos, we’ll also discuss apps, automation scripts and also advanced techniques such as Continuous Integration for Office 365. Hopefully even people *not* expecting to work on an Office 365 development project can come away with flavour of what such a project might look like.

Time: April 17 11am BST

Session URL: https://www.sp24conf.com/2014-1/Conf/SP24S108/ConfPages/SessionRoom.aspx

To register for SP24..

..go to:



Sunday, 6 April 2014

Using the Azure instance behind your Office 365 tenant

Something that perhaps isn’t common knowledge in the SharePoint world, is that every Office 365 tenant has an Azure instance behind it – and that if you’re doing some types of Azure stuff, sometimes you *must* use this instance. I’ve spent the last few months doing Office 365 development with some use of Azure, and this only became clear to me fairly recently. When I’ve mentioned it to other people, most have been completely unaware, and I’m not surprised when you consider that:

  • There is no link from Office 365 to the Azure instance (e.g. in the Office 365 tenant admin screens)
  • If you try to log-in to Azure as an Office 365 user, you initially get presented with a screen that makes you think you’ve done something wrong (as at spring 2014 time anyway)

Scenarios where you NEED to use this Azure instance

For many Azure usages (e.g. “externalizing” assets such as CSS/JS files like my team do, Azure Service Bus, Azure CDN etc.) it doesn’t matter which Azure instance you use. The most natural thing is to go the Azure Portal (https://manage.windowsazure.com) and sign-up if you don’t have a subscription already. Of course, this creates you a brand new Azure instance – and that works great for these types of scenarios. However, you’d need to use the “integrated” Azure instance for anything which needs to work with the directory of Azure AD/Office 365 users for your tenant. This could be things like:

  • Implementing Azure AD authentication on an external application (e.g. a provider-hosted app, or other website), where you are aiming for some kind of Single Sign-On
  • Working with some aspects of Office 365 users (e.g. the Azure AD profile which is “behind” the SharePoint Online user profile) via the Azure Graph API
  • Integrating a 3rd party application which is able to use Azure AD authentication/have SSO (e.g. salesforce.com, Dynamics CRM, Dropbox for Business etc.)
  • ..and many other scenarios involving identity

Accessing the Azure instance behind your Office 365 tenant

So let’s say one day you think, “I wonder how I’d get to this magical Azure instance?” Well, if you go to the Azure Portal, you’ll either see a sign-in screen asking you to enter a username to continue, or you might be auto-logged in if you have previously signed-in to the Microsoft gateway – for example, you’re already logged-in to Office 365. If you did that (even as an Office 365 tenant administrator) you’ll see this:


I don’t know about you, but my immediate reaction to that is “Whoah, obviously I shouldn’t be here logged-in as this user!” However, now is the time be brave and not flinch! Sometimes in life, it takes a valiant act of courage to push through the terrifying situations that can befall a modern-day I.T. warrior! The thing to do now is actually to click:

“Sign up for Windows Azure”

Yes, it’s counter-intuitive. As I mentioned earlier, the Azure instance already exists so why do we need to “sign-up”? Well, you just do - for now at least. I did hear a Microsoft speaker at SPC mention this experience as a side-note, and that they will probably improve things soon. The main thing to understand is that you have to supply payment details before this Azure instance can be accessed – because then it becomes a true pay-as-you-go instance.

Viewing the Azure AD directory behind your Office 365 tenant

Once you’ve gone though the sign-up process, you should be able to go into the Azure Portal (https://manage.windowsazure.com, or the new preview version https://portal.azure.com). Here, out of the all the things you can use in Azure, you’ll probably have just one thing – a Directory:

Azure items

..and if you go into the Directory, you should see all your Office 365 users (including those synchronized by DirSync, if configured):

Azure directory users - small

Apps which integrate with Azure AD

As I mentioned earlier, you mind yourself creating some kind of app which needs to work with Azure AD (e.g. for authentication/SSO, or some usage of the Azure AD Graph API for user records perhaps). The first step in many of  these cases is to register the app in your Azure AD directory – this is done in the “Applications” area of the directory:

Azure create directory application

Other scenarios around Azure subscriptions and identities

It turns out there are quite a few permutations of starting an Azure subscription, or joining up to an existing one. One reason for this is that in the new world of Microsoft identities, it’s possible to use a Microsoft account (formerly Live ID) OR an “Organizational Account”. If you’re an Office 365 user, your Azure AD record is an Organizational Account. Personally I’m still getting my head around some of the nuances here, but Microsoft recently released this video which talks through various Azure sign-up scenarios – if you want further information, I think it’s helpful:

Using a existing Windows Azure AD Tenant with Windows Azure


Wednesday, 19 March 2014

An implementer’s summary of the SharePoint Conference 2014

Quite a few SharePoint community folks have already published their round-up posts on the recent SharePoint Conference 2014 held in Las Vegas. I’ll SPC keynote smaller point to some good ones at the end of this article, but regardless of other posts I wanted to go ahead and publish mine because:

  • I notice I have some different items compared to other people – obviously everyone has their view of what is important to them :)
  • I think some colleagues and others in circles near me have similar views and focus areas, and might find my interpretation useful

I break my thoughts down into two sections – The Big Things, and a Dev-Focused Summary. If you feel you already understand The Big Things from SPC2014, you might want to skip ahead to my dev-focused summary:

The Big Things

  • There will be at least one more on-premises version of SharePoint, but it’s unclear how many more after that (I sense Microsoft may take a “wait and see” approach). The next version will be “SharePoint 2015” (or at least it will be released in 2015).
  • Some big changes to Office 365 which will open doors for many clients/scenarios:
  • Go Yammer!
    • The social capabilities in SharePoint 2013 will NOT be developed further
    • However in Yammer, there’s a lot of integration in the pipeline – this will mainly be on the Office 365 side until the SP2015 release though. Includes things like:
      • Groups - create a group, and automatically get a site/inbox/OneDrive library for docs etc.
      • Conversations around documents (i.e. comments displayed next to document) – in Office Online (new name for Office Web Apps), rather than Office clients on your desktop however
      • Integrated profile/identity (no longer separate Yammer and SharePoint profiles) – this will unlock other scenarios going forward
  • Forms - InfoPath is deprecated (although supported until 2023)
    • Microsoft will move away from a “one size fits all” approach to forms, and several options (some available now) will be recommended – Excel Surveys, list forms (to be enhanced), a Word-based option for “structured documents”, and “App Forms” (Access-based)
    • Microsoft are listening at http://officeforms.uservoice.com
    • Nik Patel has a great write-up here
  • The Office Graph, and Oslo app
    • This is a big new concept, about users finding content that matters to them. The Office Graph is the framework/API, and Oslo is the codename for a new app which presents this stuff. The Office Graph  “uses signals from email, social conversations, documents, sites, instant messages, meetings, and more to map the relationships between the people and things that make your business go”
    • There are lots of views such as “Shared with me”, “Modified by me”, “Presented to me” etc. – things that really make it easier for me to find “that file I was looking at last week”. The sentence that brought it home for me was “The Office Graph knows which meetings you’re in, when someone is presenting, and where the presentation is stored. Oslo just connects the dots to show what’s been presented to you.”
    • See Introducing codename Oslo and the Office Graph for more details
  • SharePoint 2013 Service Pack 1 smoothes some hybrid scenarios:
    • This goes some way to helping out with a couple of the issues I listed in Office 365/SharePoint hybrid – what you DO and DO NOT get, mainly by providing an OOTB way to swap out some links in the top navigation bar (aka “the suite bar”) displayed in your on-premises sites. Specifically: clip_image001
      • Replace "Newsfeed" with "Yammer" (if you have selected Yammer as the social option)
      • Replace link to “Sites” page – here, an admin can provide the URL to the “Sites” page in Office 365. This facilitates having ONE Sites page (in the cloud) across SharePoint Online and SharePoint on-premises – this has two benefits as far as I can see:
        • Users have one place to go to for their saved navigation links (sites/documents), rather than a very confusing situation of having them split across two Sites pages. (BUT, I’m fairly sure that “following a document” in an on-premises site does NOT magically save it to the O365 Sites page – so you may still prefer a custom approach)
        • The “Start a site..” link only exists in the cloud, so new site collections get created there
      • Replace "Skydrive Pro/OneDrive For Business" link with link to OneDrive in cloud

My dev-focused summary

  • Full-trust code (farm solutions) will continue in “SharePoint 2015”
  • New Office 365 APIs NOTE: Microsoft are starting to move away from SharePoint/Exchange/whatever-specific APIs, and more towards general “Office 365 APIs”
    • People/Contacts
    • Files (OneDrive API)
    • Mail
    • Calendar
    • [Apps which need to call into the above endpoints get registered with Windows Azure Active Directory, and use the Common Consent Framework to access Office 365 data]
  • That said, there are also new SharePoint client APIs (also usable in on-premises SharePoint with SP1/updated Client Redistributable)
    • CSOM API for creating site collections (finally!)
    • CSOM API for creating a content type with a specified ID (finally!)
    • Also being able to set things like AlternateCSSUrl, SiteLogoUrl for a site
    • Also a new SDK for creating Office 365 apps on Android. Apparently this is big news (lots of hoohaa at the conference) – no idea why, but it IS refreshing that Microsoft now talk about executives with iPads and the popularity of Android..
  • There are new JS/HTML-based controls for use in provider-hosted apps
    • The first controls available are ListView and People Picker (e.g. show a SharePoint list, with sorting/filtering etc. in your plain ASP.NET page in your app)
    • These are badged as Office.Widgets.Experimental for now
  • Auto-hosted apps are (likely to be) deprecated baby! (Microsoft have some challenges around making this production-scale ready, in terms of scalability/backup/DR etc. The long-term prognosis is being reviewed, but the recommendation is to use auto-hosted for prototyping only)    UPDATE: looks like this isn't the case (although several MSFT speakers mentioned it at SPC14) - Brian Jones (MSFT PM) chimed in on Twitter, and he's the guy who would know
    • [For me, this is a good validation of what I said in July 2013 about auto-hosted apps and why they should be avoided (in the “What about building my app/RER as an auto-hosted app?” section of this article)]  UPDATE - I guess we reserve judgement on this one for now!
  • There are GREAT new dev samples on many real-life cloud development scenarios
    • I highly recommend checking these out – see Office App Model Samples
    • Scenarios include:
      • Branding personal sites
      • Provisioning Yammer groups as part of site collection provisioning
      • Setting a theme on the host web
      • And LOTS of others
  • Some misc stuff:
    • JSON light support in Office 365 APIs
    • New apps in Excel
      • People Graph
      • Bing maps
    • New mail app in Office 365/mail
      • DocuSign!
    • New “file extension handler” - can register an app for opening/previewing files by document extension
      • e.g. CAD viewer in preview pane for CAD files
      • I think this is similar to a CustomAction – i.e. there is new Feature XML for this
  • Other news I found relevant:
    • No Microsoft solution is planned around sync’ing custom data to SharePoint Online user profiles right now
      • Microsoft do not have anything in the pipeline to facilitate sync’ing custom directory data to the user profile in SharePoint Online (a commonly-requested thing). I asked at the post-conference session on hybrid, led by Bill Baer and Steve Peschka. So, the workarounds using the old user profile ASMX web service continues to be needed if you have to do something here.
    • Microsoft plan to improve the “in SharePoint hybrid, search results are displayed in separate blocks (for on-premises and Office 365 sites)” experience (shown in my post here)
      • This was mentioned in Bill Baer’s IT Pro keynote – some sort of “remote index” solution is planned

Summary and more info

I am sure there’s lots of other stuff too, but those were probably the most relevant things for me. Some other recommended posts are: