Office 365 apps – deciding between the SharePoint app model and the Office 365 APIs

Just like me, many other SharePoint developers have spent the last year or two getting to grips with the SharePoint app model – this was introduced in SharePoint 2013 as a way of building customizations which do NOT run on the SharePoint server. Enhancements built this way can run on a remote machine (e.g. Azure, IIS, whatever) and therefore any stability/performance issues do not affect core SharePoint. If you’re using Office 365 of course, the app model is a big cornerstone of extending the platform, since Microsoft do not allow custom SharePoint code to be deployed to the Office 365 servers (server-side code at least). These were big changes in the SharePoint development world, with lots to learn around authentication, remote APIs and patterns for solving common needs.

However, there’s a new kid in town. Microsoft have introduced a series of new APIs for Office 365 which span SharePoint, Exchange and Lync – and rather than needing to know each platform as a developer, more general concepts are exposed such as:

• Mail
• Calendar events
• Contacts
• Files

Of course, those who know the territory will recognize that “mail”, “calendars” and “contacts” are related to Exchange and that “files” are related to SharePoint (N.B. right now this is primarily about files stored in the user’s OneDrive for Business area). But the point is that not everyone who wants to work with Office 365 data these days has that background. Initially this new direction can be confusing to the experienced developer – after all, can’t I access SharePoint files using the existing SharePoint CSOM or REST APIs? Also, why is authentication different? I just spent months learning about provider-hosted apps and the “new” authentication models with context tokens, access tokens and so on! Why did we need something new?  If this is a new way of building apps for Office 365, is the old model going away? These are all logical questions in my opinion, so I wanted to talk about them here, as well as discuss some things to look out for in your apps which suggest you SHOULD be using the new approach. But for sure, if you’re about to start building an app you might be faced with this decision (with a gratuitous graphic to illustrate):

After all, there are many similarities between the two – in both cases, the core implementation could be ASP.NET MVC, on Azure or some IIS servers for example. Let’s start by understanding what an Office 365 external app is about.

What an Office 365 app looks like

Here are some key tenets of these apps:

• The application is an “external” or “standalone” app of some kind, perhaps implemented in ASP.NET (most likely MVC flavor these days). And when I say website, remember that it could really be something else altogether, like an App for Office which surfaces in Word or Outlook (since these use web pages underneath). Equally it could also be a Windows Store/Phone app or one which targets other devices, such as an iOS/Android app
• We want to somehow integrate the app with Office 365 (perhaps in terms of user experience/navigation, data storage or authentication)
• The app is registered in Azure Active Directory – specifically the Azure AD behind your Office 365 tenant
• Apps can implement the same authentication mechanism as Office 365. This means your users get single sign-on, and can pass between Office 365 and your app without re-entering their username/password
• Users can navigate to the app via the Office 365 App Launcher or “My apps” page
• In coding terms, the apps are likely to use the new Office 365 APIs to access data there, but can also revert to using lower-level APIs such as the SharePoint CSOM/REST APIs
  • N.B. To do this, you use the authentication token obtained from Office 365 as the “bearer” token in the HTTP headers for the request to the other API:
    • Example code extract for a SharePoint REST API request (.NET code):
      
      var accessToken = getAccessTokenHere();
      request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);
      
      
    • Further reading on this: Using the SharePoint CSOM and REST API with Office 365 API via Azure AD (Jeremy Thake)

Developing with the Office 365 APIs

If I was to summarize the development model, I’d say there are two high-level approaches:

• Using the Office 365 REST API (more operations, but more complex since you need to craft the HTTP requests yourself and deal with authentication etc.)
• Using a flavor of the Office 365 client libraries (simpler to use, but don’t have 100% coverage of what’s possible). These flavors include:
  • .NET (for use in ASP.NET websites, Windows Store/Windows Phone apps, Xamarin apps etc.)
  • iOS
  • Android
  • Cordova apps (multi-device hybrid apps)

Key differences between Office 365 apps and Apps for SharePoint/Apps for Office

But what if your app is SharePoint/Office focused, and could therefore be implemented as EITHER an Office 365 app or an App for SharePoint/Office? Let’s consider that decision for a while - here are some of the differences as I see them (N.B. I’m also summarizing these differences in a table below):

Difference 1 - how the app is accessed (by end-users)

Office 365 app:

An Office 365 app can be accessed in a few ways:

• From any link (e.g. on your intranet home page), since it’s a standalone website
• From the Office 365 App Launcher (if the app is “pinned” there for quick access)
• From the Office 365 “My apps” page

App Launcher experience:

“My apps” page experience:

App for SharePoint:

An App for SharePoint can be accessed in the following ways:

• From the Site Contents page of a SharePoint site where the app is installed
• From a developer-provided SharePoint link, using AppRedirect.aspx?instance_id=[app instance GUID here]

In other words, these apps can only be accessed from a SharePoint/SharePoint Online site – assuming that is, that the app accesses some SharePoint data (as most would). This is because the authentication model relies on the context token being passed from SharePoint into the app – otherwise the app cannot talk to SharePoint in any way.

Site Contents page experience:

Difference 2 – app registration (for implementer)

Office 365 app:

Firstly, these apps must be registered in the Azure AD which relates to the Office 365 tenancy:

Note that Visual Studio (more specifically, the Office 365 dev tools) perform this step for you when you create an app for a single Office 365 tenancy. Azure AD administrators can also register apps, and this is what needs to happen for multi-tenant apps.

App for SharePoint:

These apps are registered in SharePoint/Office 365 using AppRegNew.aspx (or using the Office Seller Dashboard for Store apps):

Difference 3 - how authentication and authorization works e.g. for SharePoint data (for implementer)

Office 365 app:

Once the app is registered in Azure AD, you can write code using the Office 365 client libraries to authenticate to the platform (simpler), or alternatively go lower level and obtain tokens yourself through OWIN (for MVC) or ADAL for .NET (both more complex, but what is needed when you’re coding directly against the Office 365 REST APIs). In terms of the client libraries approach, here’s a snippet showing how auth works in the .NET Office 365 library (taken from the Office 365 Starter Project for ASP.NET MVC sample):

You can cache these access tokens as you see fit to avoid repeated auth prompts. But if a new token needs to be obtained, this code does the work of redirecting the user to sign-in to Office 365 (whether they are in a web browser or on a device):

App for SharePoint:

Authentication for an App for SharePoint works in the following ways:

• SharePoint-hosted app: authentication happens implicitly, since web pages are hosted in SharePoint context (and only the SharePoint JSOM API can be used)
• Provider-hosted app: app must be registered with AppRegNew.aspx (or the Office Store), and .NET code can authenticate using the SharePointContext and TokenHelper classes which are added to your Visual Studio web project on creation:
  
  

Difference 4 – app deployment (for implementer)

Office 365 app:

Whatever is needed to to copy the app to the hosting location e.g. publish to Azure (from Visual Studio – shown below) or FTP or similar:

  

App for SharePoint:

Similar to above in that app files must be published to the hosting location, but with two extra steps:

1. The app package (.app) must also be deployed to an App Catalog:
   
   
2. Once the app is in the App Catalog, it must be installed to any SharePoint site which should use the app (or, a tenant-scoped installation should be performed):
   
    

Summary table of differences/commonalities

Since the above sections are a bit difficult to cross-reference, let me try to summarize in a table:

	Office 365 app	App for SharePoint
How the app is accessed (for end-user)	The App Launcher/My Apps page Some other link (e.g. from some other intranet site you have) or bookmark – it’s a standalone website	Typically from the Site Contents page of a SharePoint site
Authentication to Office 365/SharePoint (for implementer)	App is registered in Azure AD, then authentication helper in Office 365 client libraries typically used (e.g. AuthenticationContext object in .NET Office 365 SDK)	App authentication – context token is passed from site into the app, and access token obtained from there. Helper code supplied in form of SharePointContext and TokenHelper classes (added automatically to app projects)
App hosting (for implementer)	Any web platform – Azure or other cloud-hosting, on-premises IIS, or other web servers	Any web platform – Azure or other cloud-hosting, on-premises IIS, or other web servers
App registration (for implementer)	Registered in Azure AD	Registered in SharePoint/Office 365 using AppRegNew.aspx (or using the Office Seller Dashboard for Store apps)
App deployment (for implementer)	Deployed once to the hosting platform (e.g. Azure/IIS)	Deployed to a SharePoint/Office 365 App Catalog, and then installed by site owners to various SharePoint sites or tenant-scoped install. Alternatively, submitted to the Office Store for sale there.

Other differences

There are some wider differences not covered above too. Don’t forget things like:

• For Office 365 apps, I no longer need knowledge of Exchange and SharePoint APIs to pull together an app which (for example) stores some files in an area I can access (i.e. my OneDrive) and sends an e-mail on my behalf. In other words, I don’t need to be a “career developer” in Exchange or SharePoint to make use of this stuff
• Office 365 apps are probably easier to develop for devices – having SDKs for iOS and Android means that some common dev tasks are pretty simple on those platforms. I’d say the fact that these apps don’t need to be accessed from a SharePoint site can help here too

Summary - reasons to consider the Office 365 approach

So, there are clearly some things to consider if you’re building an app which could work as either a provider-hosted SharePoint app OR an Office 365 app. Here’s my summary list of indicators that the Office 365 app could be a good fit:

• Your app doesn’t need to work with on-premises SharePoint/Exchange
• Your app doesn’t really “fit” in SharePoint:
  • It’s an external app, not associated with a “site”
  • It sits at the “Office 365 suite” level
• Your requirements match files/messages/events etc.
• You want to integrate with the Office 365 App Launcher
• You don’t want your app to need to be installed into SharePoint sites by site owners (and a tenant-scoped SharePoint app doesn’t suit your requirements)
• You’re developing for devices and want to take advantage of device-specific SDKs
• It’s the future!

In short, there will always be cases where either an Office 365 app or an App for SharePoint is a better fit – both models are valid going forward. The key is understanding the differences, both to the end-user and the implementer.

Feel free to leave a comment if you think I’m missing anything, or have any thoughts related to this!

More speaking – SharePoint Connect 2014 (Amsterdam) and SharePoint Saturday UK

I’ll be doing a couple more talks in the next few weeks – these will cover different topics, ranging from the new Office 365 APIs to techniques for cloud-friendly SharePoint development, and also Apps for Office. Hopefully see you at one of these events!

SharePoint Connect 2014 (Amsterdam)

• Modern SharePoint development - techniques for moving code off servers (Tuesday November 18, 14:50)
  • Those implementing Office 365 know that custom code cannot run on the SharePoint servers, and remote code alternatives must be used. However, it can be valuable to adopt the same "cloud-friendly" development techniques even for on-premises implementations. In addition to increasing platform stability, this enables you to build in flexibility to your deployment - leaving the door open to a potential move to Office 365 in the future without major re-engineering. With several code-focused demos, this session examines approaches for switching to "off-box" code - including Remote Event Receivers, PowerShell and CSOM scripts and client-side JavaScript.
• An intro to the new Office 365 APIs (Wednesday November 19, 11:15)
  • Microsoft have stated that they will move away from SharePoint, Exchange and Lync APIs towards generic Office 365 APIs. The preview version of the first APIs were announced in March 2014, and cover several core building blocks for user collaboration - including mail, OneDrive files, calendar entries and contacts. Developers who need awareness of Office 365/hybrid scenarios can expect more such APIs in the future.
    
    Get on the bus early - come learn about the APIs and the new permissions model ("the Common Consent framework") in this code-focused session.

SharePoint Saturday UK (Hinckley, UK)

• Apps for Office - the hidden gem that's easier than you think (Saturday November 28, 12:15)
  • Most developers know that it's now possible to develop Apps for Office which are based on core web technologies - HTML, CSS and JavaScript. But the take-up appears to be fairly limited - why is that when development and deployment (even in the enterprise) are fairly easy? This session takes a renewed look at Apps for Office, now that the Word, Excel and Mail APIs have been enhanced and the app model is no longer the weird new kid on the block. Some cool demos are involved, including a Word task pane app which analyses the contents of a document to find you similar items across your SharePoint environment/tenancy whilst you're working with the document.

Slides/video from my Apps for Office talk at TechEd 2014 now available

I presented at TechEd 2014 last week (Barcelona) on “Weave your enterprise solutions directly into Office products with the app model”, alongside Andrew Salamatov from the Exchange product group - in this post I want to provide the PowerPoint deck and link to the TechEd video, but I also to provide some thoughts on Apps for Office in general. First off, here is the slide deck:

Chris OBrien - Weaving Enterprise Solutions into Office Products from Chris O'Brien

 

Link to TechEd video

The TechEd video can be watched in the embedded video at the bottom of this post, or the link to the video over on Channel 9 is - http://channel9.msdn.com/Events/TechEd/Europe/2014/DEV-B307 

General information

If you’ve read other posts of mine, you’ll know that I’ve spent a lot of time developing Apps for SharePoint and Office 365 recently and doing “cloud-friendly” SharePoint development in general – but (like many people I suspect) I hadn’t spent too much time around Apps for Office. Having spent the last couple of months preparing to talk on the subject at TechEd last week, I feel like my eyes have been opened a bit here – Apps for Office have HUGE potential, and I feel you could easily become a hero in your client or organization with the right idea and a bit of dev effort!

The big game-changer is the development model for how Office is extended these days:

Model	Development	Deployment to users
Some weird VSTO/VSTA stuff that I never did :)	Write code in C# against slightly strange APIs specific to Word, Excel etc.	Have to get a DLL deployed to each user’s machine – good luck with that
Apps for Office	Web page hosted in client app (e.g. Word) – developer writes HTML, CSS and JavaScript. JavaScript API for interacting with document. Many methods work across all client apps (e.g. Word, Excel, PowerPoint etc.)	No deployment to user’s machine – it’s a web page

Additionally, an App for Office could optionally make use of SharePoint/Office 365 building blocks if useful. Some apps will just relate to Office (e.g. the Wikipedia app in the Office Store), but in the enterprise there are quite a few scenarios where it would be useful to bring in SharePoint/Office 365 – perhaps to store files (e.g. in a team site, or the user’s OneDrive For Business site), or access the user’s calendar, mail, contacts etc. When you need an Office app which uses these things, there are two high-level approaches:

• Implement your app as an App for SharePoint (usually provider-hosted), which encompasses an App for SharePoint
  • This works for both on-premises SharePoint and Office 365, but has some trade-offs (e.g. App for SharePoint needs to be deployed to sites, or tenant-scoped install)
• Implement your app as a standalone/external app, which consumes Office 365 services via the new Office 365 APIs and is registered in Azure AD
  • This only works for Office 365, but lots of advantages (e.g. app can be made available in the new App Launcher)

This is a decision that will come up quite frequently for Office 365 development from this point (autumn/fall 2014) onwards, and I’m sure I’ll be touching on it in quite a few future articles.

My “find similar documents” sample app

For the talk, I developed an app which surfaces in Word - specifically in the “task pane” which sits alongside the document. My app is nowhere near production-ready, but it tries to illustrate a few concepts, including use of SharePoint/Office 365 for surrounding functionality. Since I wanted it to work with both on-premises SharePoint and Office 365, I chose the provider-hosted App for SharePoint model – but my conclusion now is that the Azure AD/Office 365 APIs approach is preferable if the app only needs to work with Office 365 and you can go that way. Anyway, my app is based around the idea of “finding documents similar to the one you are working with” – perhaps to identify similar or duplicate content within your organization, find other people with expertise on the same topic, and so on. Here’s a screenshot to give you the idea (click on an image to enlarge):

Here’s what my app does:

1. Analyzes the document content (or the user can choose to analyze just a specific part), and extracts the most relevant keywords/phrases:
   
   
   
   It uses the top 3 keywords/phrases to search your SharePoint/Office 365 environment for similar content. I wanted something which worked purely on the content of the document, so that the user doesn’t need to tag the document in any way (to find related documents). By using the Office JavaScript API, I’m able to fetch the content of the document (or even just the content selected by the cursor) and then perform some analysis on it. Other, less important keywords/phrases are listed for information.
   
   
   1. As a bonus, it also analyzes the document for “sentiment” – and reports whether the document is mainly positive/negative:
      
      
      
      At this point, you might be wondering what kind of analysis I do to extract the keywords and determine the sentiment score. I’d love to tell you it’s some amazing algorithm that I spent weeks on and have registered for patent :) In reality of course, there’s actually a bunch of services on the internet that do this kind of thing – so (unfortunately) my smartness only extends to combining the Apps for Office model with one of these services! The one I chose is called AlchemyAPI, and it seems pretty cool. It’s a pay-for service to use in production, but you can obtain a free key for demos/evaluation etc.
      
      
2. As shown in the image above, those top 3 keywords/phrases are used to power a SharePoint search query. The results are shown in the task pane – each result has a “Save” button (floppy disk icon!) next to it:
   
    
   
3. The user can save any interesting references for follow-up – these get saved into a list in the user’s OneDrive For Business site (i.e. what we used to call their “My Site” in SharePoint). Here, the user can edit the list item and add notes if they choose – the main thing is they have the link and author details stored for future reference:
   
   
   
   From here I thought it might be interesting if these “saved references” could be easily searched by other users – almost in a social bookmarking kinda way. So, I thought some enhancement of the search experience would be useful here:
   
   
4. If any “saved references” appear in search results, they are highlighted with some special presentation (mainly just a grey background, some tweaks to the display so that the author’s notes and details of the “linked to” item are shown, along with a custom hover panel):
   
    

I also implemented a custom Result Source for saved references, so that a search only within these items can be done if desired.

Over the next few weeks, I’ll tidy up the implementation somewhat and publish the code somewhere.

Summary

• Apps for Office have huge potential in putting useful business tools in front of users, so that they don’t need multiple windows/applications open. We can put the experience right there in the applications they would already be in – and this makes even more sense when the functionality somehow relates to documents or e-mail
• The development model is WAY simpler than previous versions of extending Office – if you’re comfortable with a little JavaScript, you can interact with Office documents and implement apps which in Word, Excel, PowerPoint, Outlook and so on. The beauty is that your app will appear in the rich client, as well as web *and* mobile versions where appropriate – there are some caveats here, such as Word task pane apps only appearing in the rich client (for now), but for other permutations (e.g. mail apps and Excel apps) it’s a pretty good story already
• Deployment and making your app available to end–users is also much simpler than before. There *are* some things to take care of on the IT Pro side (e.g.. deploying Office Trust Center settings through Group Policy), but these aren’t too tricky – I cover them towards the end of the slides

I’ll definitely be looking for opportunities to bring in Apps for Office to my client solutions moving forward – cool stuff indeed.

TechEd Video

 

Speaking at TechEd 2014 on Apps for Office

TechEd Europe is just round the corner (October 28-31, in Barcelona), and recently I’ve been heads-down preparing to speak there on the Office 365 track. Most of the other speakers on this track are Microsoft folks from the Office 365/SharePoint/Exchange product groups (including Rob Howard, Brian Jones, Vesa Juvonen, Steve Walker, Jeremy Thake, Mauricio Ordonez and Sonya Koptyev) so there’ll be some *great* information being dispensed – my aim is to sprinkle in some field experience there too :) As an added bonus, I hear that TechEd will have an announcement or two that developers in this space will like.

My session is on Apps for Office, and I’m fortunate to be co-presenting with Andrew Salamatov from the Exchange team. Andrew is a Program Manager whose team builds the capabilities around mail apps, and he has some very cool stuff to show in this area. Apps for Office in general are becoming very interesting to me, as it kinda feels like there hasn’t been huge amounts of take-up of Apps for Office since the launch – maybe that’s confined to my corner of the Office 365/SharePoint-focused development world, but I think it’s surprising given the power and possibilities there. After all, many users LIVE in Office applications, and from the dev perspective it’s really just web development with some Office-specific JavaScript APIs these days – WAY less scary than previous models :)

Overall, Andrew and I have a couple of cool demos to show, including a Word task pane app which analyses the contents of a document to find you similar items across your SharePoint environment/tenancy whilst you're working with the document.

Here are the session details:

DEV-B307: Weave your enterprise solutions directly into Office products with the App Model (Friday, October 31 12:00 PM - 1:15 PM Room: Hall 8.0 Room D4)

Apps for SharePoint and Apps for Office allow you as an enterprise developer to weave your existing enterprise solutions throughout the Office products that your users live in day to day. In this session you will see compelling real world examples that light up existing Office products in context of key enterprise scenarios. The session will highlight tips and tricks on architecting this solution in Visual Studio along with Patterns & Practices guidance.

• Understand what scenarios are available to surface your customizations in Office products
• See real world enterprise scenarios that span multiple products stitching together an enterprise solution
• Learn tips on how to architect enterprise solutions with the App Model

Hopefully see you there!

JavaScript-based provisioning in SharePoint/Office 365

In this post I want to specifically focus on the idea of JavaScript-based provisioning code – a potentially useful approach which has a place in every modern SharePoint developer’s toolkit. This is the idea of using SharePoint’s JSOM API (i.e. the JavaScript version of CSOM) to perform various one-time configuration steps for SharePoint sites - I was initially somewhat sceptical of this technique (JavaScript? For provisioning? Would that be reliable??), but having seen some team-mates use it with success on projects (kudos guys!), I think it’s worthy of consideration.

Background

For SharePoint developers who are working with SharePoint Online/Office 365, or are working on-premises but choosing not to use full-trust WSPs simply to ensure their solution is “cloud-friendly”, the previous options for custom SharePoint code are no longer available. And some common scenarios still require code – especially when some form of custom site template is being developed, or we are working towards automated configuration of sites.

Some examples of provisioning steps that a developer might need to use code for could be:

• Binding Managed Metadata/taxonomy fields
• Setting a theme/Composed Look on a site
• Setting the master page of a site
• Modifying navigation settings
• Branding OneDrive (personal) sites
• [Optional] Provisioning fields and content types through code, rather than XML

This presents a challenge when server-side SharePoint code cannot be used. As we all know, Microsoft have deprecated the use of sandboxed SharePoint code (which we might previously have used for say, a Feature receiver), and this facility may be turned off in Office 365 in in the future. So, we have to use one of the remote APIs instead – and the code will either run from some other server (e.g. the .NET CSOM) or the user’s browser (JSOM).

Options for remote code

Broadly, the options for remote code include:

1. Some use of the .NET CSOM API:
   1. Code deployed to a cloud service such as Azure, or running off your own server somewhere. This could be an “App for SharePoint”, or it could just be remote code which does not use app authentication.
   2. A “PowerShell + CSOM” script you wrote
2. Some use of the JSOM API (i.e. the JavaScript version of CSOM)
3. Some use of the REST API (either from JavaScript or remote server-side code)

As Microsoft put it, your custom code either needs to move UP to the cloud or DOWN to the user’s browser.

The JavaScript-based (JSOM) option

The idea of using JSOM code for site provisioning steps is a strange one initially. What happens if the user closes their browser as the code is running? Will the site get configured properly? What about if the user is very far away from the server on a slow connection? It was questions like these that made me sceptical initially, but with some safeguards and caveats in place I think it’s an interesting option for remote code. I’ll talk about how it works in a second, but first here’s why I think it’s interesting:

Side-stepping the “server-side remote code” issue

As a modern SharePoint developer, I’m all onboard with apps and remote code. In several cases I’ve implemented remote SharePoint code in Azure (e.g. see my Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps) post). So yep, I’m cool with that stuff! And I’m also down with kids in terms of Microsoft’s OfficeDev Patterns and Practices samples, which cover a ton of scenarios such as those I list above – they’ve effectively written a great wrapper around the .NET CSOM API calls, which make it really easy to get these tasks done.

BUT THIS TYPE OF CODE STILL HAS TO BE DEPLOYED SOMEWHERE! On-premises servers (e.g. some IIS boxes) can bring complexity in terms of high-availability, access outside the firewall, SSL certificates, backup and restore, and so on. Of course, cloud options exist too. If you (or your client) is already using Azure, then GREAT! It’s really not too difficult to get the code deployed and running there, and Microsoft take care of the operational/I.T. Pro factors listed above. But in many of the organizations I’ve worked with, unfortunately Azure (or similar cloud service) is not yet embedded in the I.T. department’s toolkit. There is no Azure subscription, with someone in I.T. already paying the monthly invoice. And there are all sorts of operational I.T. topics to discuss and get agreement on before we could get to that position – who owns responsibility, what kind of data will we be storing there, who has access, how does it fit with any existing SLAs/OLAs, which department/cost centre pays the bill and so on.

Interestingly, working in an “implementation partner” as my team does, we almost have an option of deploying some remote code for a client to Azure and not needing to involve them in this decision. The Azure Websites offering has a free option, and therefore all the billing questions/set-up could be avoided. However, you do not get high availability in free mode – so if the machines your code is running on are being patched, you’ll have downtime. So in the end, unless your code is trivial this probably isn’t an option.

Consequently, the idea of implementing remote code in JavaScript can be quite appealing in some cases.

A robust implementation for JavaScript-based provisioning

So, we’d need some way of making a JavaScript-based approach somewhat reliable. The pattern my guys have used is this:

• The SharePoint site is created (perhaps from a custom WebTemplate which takes care of some settings in XML) – at this point, the site is NOT fully configured
• A custom JavaScript file is referenced on every page – this checks to see if set-up steps have been completed yet. If not, the user’s permissions are checked to see if they are a Site Owner and whether we can proceed (with a “A site owner needs to visit this site to configure it” message shown if not).
• When a user hits the site, our code runs. If we can proceed with set-up, a dialog is shown - “Please wait – we are getting your site ready”
• The code starts to execute provisioning steps – perhaps setting a Composed Look and/or custom master page, binding taxonomy fields and so on
  • When each step completes, an entry is written to the web property bag e.g. “BrandingApplied = true”
• When all steps are complete, a success message is shown and the “getting your site ready” dialog is automatically closed. The site is now ready for use.

As you can guess, it’s only when all confirmations are written to the property bag that we no longer attempt to run the steps. This works surprisingly well in practice – users/site owners are comfortable seeing these kind of messages, and having a site owner hit the site initially tends to work out fine, since they are the site creator. If any failures occur, well that’s fine too because the uncompleted steps will be re-tried on the next page load.

To illustrate the experience, here’s what the site owner might see during configuration (shown here on a test page, rather than a “real” site home page):

Of course, you could do something a bit fancier in terms of look and feel.

Some sample code for JavaScript-based provisioning

In my other article Three cloud-friendly ways for developers to provision Managed Metadata fields in SharePoint/Office 365, I show some JavaScript code which does the basics described above – specifically for binding taxonomy fields to MM Term Sets. This is the same sample as in the other article - I’ve included it again below for convenience, but I recommend reading the other article too as it has some extra notes about how the code should be used (like the fact you might want to enhance with JavaScript promises). Note that it doesn’t deal with setting confirmations in the web property bag, but it would be fairly simple to add this. The code will work with Office 365 or on-premises SharePoint:

I guess the main thing to take from this code is the approach - showing the dialog whilst several steps of JSOM code are being executed. In this simple example, when the code completes our taxonomy field will be bound and ready for use – but in our real-life projects obviously many more steps are performed.

Summary

Whilst the idea of using JavaScript/JSOM code for provisioning can seem strange, I believe it has it’s place and can be the right choice sometimes. Certainly, if you have no other need for remote code and/or do not have a hosting location (such as Azure or on-premises IIS servers) ready to go in a production sense, then the JSOM approach does offer you a way of running remote SharePoint code without the hassles that come with the alternatives. As always, consider the trade-offs and think about what’s right for you – but hopefully the information here helps you along the way.

Three cloud-friendly ways for developers to provision Managed Metadata fields in SharePoint/Office 365

Provisioning Managed Metadata fields “the developer way” brings some additional complexity where SharePoint full-trust solutions cannot be used, such as Office 365/SharePoint Online deployments or on-premises projects where cloud-friendly techniques are being used. In the past, a developer would use a Feature receiver to bind taxonomy fields to Term Sets – but this isn’t possible in cloud scenarios. I wrote about this previously in Provisioning Managed Metadata fields in Office 365 – here I discussed the problem, and provided an approach we’ve used previously which does work in the cloud. As the article shows, it’s actually possible to use a fully-declarative (XML-based) way of provisioning Managed Metadata fields, so long as some GUIDs which relate to your Office 365/SharePoint instance are inserted into the XML.

Downsides to the declarative approach

The article also discusses the fact that if you work against multiple environments (tenancies or on-premises), at least one different GUID (for the Term Store ID) needs to be used in the XML. This effectively leads to different WSP packages being deployed to different environments – of course, this is not good in ALM terms! In our case, we were fairly comfortable with this trade-off since our TFS automated build process dealt with building packages for different environments, and we didn’t feel much pain. However, without an automated build process it can be more problematic – and now that more time has passed, I think it’s worth stepping back and discussing other options for solving this common problem.

3 ways to bind taxonomy fields

Before delving into the detail of each one, here are the main options as I see them (N.B. I’ve used this list in some of my other articles):

1. The 100% declarative approach (as discussed above and in my other article)
2. Using some form of remote server-side .NET CSOM code:
   1. The Microsoft sample in the Office Dev Patterns and Practices samples (previously known as the AMS samples)
   2. Some custom code you wrote
   3. Some custom “PowerShell + CSOM” script you wrote
3. Using JSOM code running in the user’s browser, which I discuss in a companion article to this one - JavaScript-based provisioning in SharePoint/Office 365

Notably, the last two options aren’t really limited to “the taxonomy field problem” – both of the remote CSOM or JSOM approaches could be used as part of a wider strategy for automated provisioning i.e. any setup steps which require code. My quick list of things which commonly need to be achieved here include:

• Binding Managed Metadata/taxonomy fields
• Setting a theme/Composed Look on a site
• Setting the master page of a site
• Modifying navigation settings
• Branding OneDrive (personal) sites
• [Optional] Provisioning fields and content types through code, rather than XML

So, if you have other items on this list you need to accomplish *or* you just prefer to avoid the declarative approach for Managed Metadata fields because of the ALM trade-offs, then you’re probably looking at a choice between the .NET CSOM or JSOM approaches. For .NET CSOM code, the Office Dev sample code works just fine - but, of course, you have to deploy and host it somewhere, whether it’s a cloud service like Azure or your own IIS servers. And this isn’t necessarily simple for a production-grade enterprise implementation - in the “Side-stepping the server-side remote code issue” section of JavaScript-based provisioning in SharePoint/Office 365, I cover this debate in more detail. As I conclude there:

The idea of implementing remote code in JavaScript can be quite appealing in some cases.

And if steps are taken to increase robustness, I think this approach is OK. So with all that in mind, let’s see what provisioning Managed Metadata/taxonomy fields in JSOM code looks like:

Provisioning with JSOM code

The sample below shows the code, but if you’re interested in using it also see the notes below:

Some notes on this code sample:

• The code will work with Office 365 or on-premises SharePoint
• One of the core benefits is that the code allows you to specify the *name* of the Term Set, rather than it’s ID. This is useful since this GUID will be different in different environments, unless you’ve taken specific steps to avoid this (i.e. by creating the Term Set programatically)
• In fact, three things need to be specified – the name of the Term Set, the locale of the Term Set, and the internal name of the field – you can see these at the top of my code. Ultimately they get passed to the initTaxObjects() method which starts the process
• I’ve kept the code simple by not using JavaScript promises, but that could be an exercise for the reader if you’d like to enhance it in that way
• The method is currently set up to process one field at a time, but it could be extended to accept an array of field names/term set IDs
• Current there’s a delay of 4 seconds inserted for illustration purposes – make sure you remove this if you implement in production :) Similarly there are lots of console.log lines which you might choose to remove
• You may need to deal with script dependencies differently if not running in a SharePoint page (e.g. you’re code is running in an app)
• In current form, the code above expects some calling code somewhere to trigger the execution, and a DIV with ID of "jsomProvisioningMessage" (for some simple logging messages) – you could have both of these in a custom web part or Script Editor Web Part added to a page. This is the code:
  
  <div id="jsomProvisioningMessage" />
  <script type="text/javascript">
  $(document).ready(function () { 
    window.COB.JsomProvisioning.execute();
  }); 
  </script>

Whether it’s this approach you use or an alternative, if successful your taxonomy field should look like the following (before and after shown):

Before (unbound):

After (field is now bound and can be used for tagging):

Summary

Managed Metadata/taxonomy fields have always been a bit of a pain for developers, because there needs to be an association between the field and a Term Set, and these have different IDs in different environments/tenancies – so it often needs to be dynamically looked up/specified. Assuming we are implementing the site the developer way with automated provisioning, this can be done 100% declaratively (with the ALM trade-off), with .NET CSOM code or with JSOM code. In this post I’ve focused on the JSOM approach – since if you have no other requirements for remote code (and therefore no Azure subscription ready to go), this method can be appealing.

Remote Event Receivers on host web gotcha – no context token/ClientContext

Update August 2015 - new information!

Vesa Juvonen (Microsoft) got in touch with me to say that he and the SharePoint engineering team had been doing some testing around this, and that whilst the "app-only" workaround I discuss below does indeed work, it's actually only necessary under certain conditions. It turns out that the problem I describe in this article ONLY occurs if the event receiver is NOT registered *using app authentication*, e.g. in an AppInstalled event. If the RER is registered using *user authentication*, e.g. with the SharePointOnlineCredentials class, then you WILL see the behavior detailed here - so your choices then are either to register the RER with app authentication, or use the app-only approach detailed below in your code.

In other words, *how* the remote event receiver was originally registered has an impact when the code runs.

Thanks for the info Vesa :)

I’ve spent quite a bit of time working with Remote Event Receivers in SharePoint 2013/Office 365, and in the early days I hit an issue that initially made me think they didn’t work properly at all. Now with some help, I realize that mainly the code just has to be written a certain way - but it’s also true to say that there is a limitation which can affect some things you might want to do. In any case, this is a gotcha that I’m starting to see other people hit so I think it’s worth talking about it.
Specifically the issue occurs when:

• The event receiver is on the host web (rather than the app web)
• The event receiver uses CSOM code to talk to SharePoint (rather than just using NON-SharePoint code)

Issue symptoms

When you’re trying to use RERs on the host web, what you might see is that whilst your event receiver always fires (and your remote code does execute), your code fails to authenticate back to SharePoint. You’ll get a 401 error when trying to obtain or use a ClientContext object – this could manifest itself with one of these in your code:
System.NullReferenceException: Object reference not set to an instance of an object
..but if you look at the HTTP requests/responses, you’ll see the HTTP 401 (Access Denied).
This means you effectively can’t do anything with SharePoint – and often, of course, the whole point of your event receiver is that it’s going to do something in SharePoint! Perhaps you need to create, update or delete an item somewhere, or perform some other action related to sites, users or data in SharePoint.
Other info:

• No context token
  • If you do some debugging, you’ll find that the event receiver does NOT receive a valid context token from SharePoint/Office 365 - SPRemoteEventProperties.ContextToken is an empty string. This causes all the later code to fail. At this point, you probably think there’s a bug with SharePoint/Office 365 – even at the end of all this, I’m personally not sure whether there is a bug or it’s just a “by-design” limitation. Fortunately, as we will see there is a workaround which works for many (most?) scenarios of this type.
• The same code works on the app web
  • If you do some wider testing, perhaps to see if there is a code issue – you’ll find the very same code works fine in other contexts. For example, if you use the same code in a RER on the app web (e.g. “ItemAdded” on a list in the app web), everything works fine. The issue is purely related to RERs on the host web.
• No change if you use ProcessOneWayEvent (asynchronous events such as added, deleted etc.) or ProcessEvent (synchronous events such as adding, deleting etc.)
  • Remote Event Receivers are similar to traditional full-trust event receivers, in that both async and sync events can be used. An example here would be the “added” and “adding” events respectively – these correspond to the ProcessOneWayEvent and ProcessEvent methods in a Remote Event Receiver class. For reference, the issue we’re discussing here manifests itself in both cases.

So those are the symptoms. Before we dig into the “Why it happens” and “The workaround” sections, it’s worth reminding ourselves of some aspects of remote code and authentication in particular. Feel free to skip forward if you’re familiar with this territory.

A reminder on authentication of remote code

For those just starting out with Remote Event Receivers (or SharePoint apps/remote code in general), it’s worth noting that an app is required for any form of remote code. More correctly, we could expand this to say any remote code which runs on the server-side and uses “app authentication” - as opposed to “user authentication” where a username/password is specified. In any case, this can seem strange for some things like Remote Event Receivers, where there isn’t really an “app” (e.g. with some pages) that the user will enter/click around etc. All we have for a RER is a remote WCF service with no front-end! But, this is the way the trust model works for remote code – so the first thing to say is that if you don’t have an app registered and installed/trusted for your remote code, it will not work.
If we were to summarize common authentication options for remote SharePoint code, we could break it down like this:

• User authentication
  • The username/password of a “named user” is passed – usually a very poor approach that brings security risks. Effectively this led OAuth to be invented as an alternative.
• App authentication – this breaks down into:
  • User + app authentication – the default. Permissions of both the user *and* the app are checked
  • App-only authentication – the permissions of just the app are checked (thus allowing operations that the user themselves does not have permission to do)

Why it happens

If you’re hitting the issue, your RER most likely has code which instantiates the ClientContext object in a certain way - specifically, you’ll be using “user + app” authentication. The vast majority of MSDN and blog samples use this approach. This is ultimately the problem -  “user + app” authentication does not currently work for events related to the host web. My testing shows this is the case for both Office 365 and SharePoint 2013 on-premises installations. Along the journey of realising this, I tried some different coding approaches – and I see other developers perhaps doing the same (e.g. in forum posts). Since there are various ways the CSOM ClientContext object can be obtained, you might try different code for this – but several cannot be used in a RER and will fail. The table below shows some of the “wrong” approaches.

Code approaches which will NOT work:

Approach	Code sample	Why not
Using the typical RER approach	ClientContext clientContext = TokenHelper.CreateRemoteEventReceiverClientContext(properties)	This is the approach that “should work” (and the approach that most samples use). However, it only works for events on the app web – NOT events on the host web.
Creating ClientContext using constructor	ClientContext clientContext = new ClientContext(properties.ItemEventProperties.WebUrl)	The context is not obtained in a way which deals with authentication. This is what the TokenHelper methods are for. Sidenote – it would be possible to authenticate with user authentication instead of app authentication, if the username/password of a named account was attached using the SharePointOnlineCredentials object. But that’s lame, and a big security risk! Instead, we want to use OAuth (via app authentication) of course, to avoid storing/passing passwords.
Using the “app event” approach	ClientContext clientContext = TokenHelper.CreateAppEventClientContext(properties, false)	The context is obtained in a way which is only appropriate for app events (e.g. AppInstalled/AppUpgraded/AppUninstalled etc.) Since we are using an event on the host web, authentication does not work.

As the table says, the one that *should* work is the first one (TokenHelper.CreateRemoteEventReceiverClientContext()) – but this relies on “user + app” authentication which appears to be the problem. So to work around this we need a different approach. 

The workaround

Instead of “user + app” authentication we can use “app-only” authentication. This requires 2 things:

1. The app permission policy to specify that app-only calls are allowed (something the person installing the app must agree to). This is enabled in the app manifest in Visual Studio:
   
   
2. The code in the app (i.e. in the Remote Event Receiver) to obtain ClientContext using an app-only access token. The expected way to do this in C# is to use the correct TokenHelper methods – big thanks to Kirk Evans for helping me with this. The code should look like this:
   
   
   ** N.B. There is a code sample here but it will not show in RSS Readers - click here for full article **

If you ensure these two things are in place, your Remote Event Receiver will work fine.

The limitation – when the workaround is no good

So, that’s the answer then. However, you might hit occasions where the app-only approach isn’t really what you want. If your remote code writes any data to SharePoint (e.g. adds a list item), you’ll notice that the SharePoint user interface makes it clear that the list item was added/modified by an app. Specifically, the username will be in the following format:
Last modified at [time] by [app name] on behalf of [username]
This is a pretty useful feature for some business requirements, since it makes it clear that although the named user was involved, they didn't make the change in SharePoint "directly". Unfortunately, use of the app-only authentication route to solve our problem changes this – specifically, with app-only auth only the app name is recorded as the user who made the change. Effectively, we lose the information which tells us which actual person was involved in the change - in some circumstances, this could be a problem if some kind of audit trail is required. To make things clear, here are two screenshots which show the difference:
List item added by "app + user" authentication:

List item added by "app-only" authentication:

Related sidenote (for the curious!):
If you’re wondering how SharePoint/Office 365 deals with this under the covers, you’ll find that all lists now have 2 new hidden columns - “App Created By” and “App Modified By”. These store a reference to an entry in the User Information list in the current site (just like other Person/Group columns):
So now you know ;)

Summary

It’s pretty easy to get things wrong with Remote Event Receivers, and the pitfall I’ve talked about concerning authentication and CSOM code is definitely a big gotcha – I certainly see other people hitting this. If you have remote code which modifies data in SharePoint somewhere, the way around the problem is to use app-only authentication - by ensuring the app’s permission policy allows this *and* also writing your CSOM code to authenticate in this way. This can be done by obtaining an access token using the TokenHelper.GetAppOnlyAccessToken() method, and then obtaining a ClientContext object using this token.
However, with this approach comes a trade-off - the loss of user information (in “pseudo-audit” terms) related to the user behind the change. Developers should bear this in mind when working with Remote Event Receivers.

Renewed as SharePoint MVP for 7th time

I received the happy news recently that I continue to be a SharePoint MVP for another year – the 7th time for me. As always, it’s been great interacting with the community over the last 12 months. Every year I like to jot down on my blog the community “things” I did in the previous year - this list is mainly for me rather than you, but here are some the things I got up to:

• Making a subtitled step-by-step video on remote SharePoint code in Azure (https://www.youtube.com/watch?v=G4T1eLg0_to)
• Two talks at the recent SharePoint Evolutions roadshow conference
  • “Modern SharePoint Development - techniques for moving code off SharePoint servers”
  • “An intro to the new developer APIs for Office 365” (will be uploaded to my SlideShare soon)
• Two talks at the European SharePoint Conference 2014 (held in Barcelona)
  • “Developer Decisions, Tips and Tricks - Lessons Learnt from Office 365 Projects”
  • “Using JavaScript Templates To Customise the SharePoint 2013 User Interface”
• More blogging – nearly 200,000 unique visitors in last 12 months according to the analytics
• A session at the SP24 online conference
• A talk at the UK SharePoint user group
• An Ask The Experts session, also at SUGUK
• A talk at SharePoint Saturday UK 2013

It’s been a privilege to be part of the community, and I hope to continue to be useful over the next 12 months!

Step-by-step video: remote SharePoint code in Azure

As I noted in my last post Modern SharePoint Development – techniques for off-box code, there is something of a movement in the SharePoint development world to what is often termed “the Cloud Application Model”. In a nutshell, this is the approach that many development tasks *must* use for Office 365, and *can* use for on-premises SharePoint. Specifically, the Cloud Application Model is about having custom code run “off-box” to SharePoint using the remote APIs rather than than the server-side API – after all, in Office 365 it’s simply not possible to deploy custom SharePoint code to Microsoft’s servers (this is not permitted because some bad code could affect the service for many clients). But as I noted last time, there can be benefits to this approach even in on-premises projects - such as:

• Greater isolation – an app running “off-box” cannot bring down SharePoint for the entire organization
• Ease of upgrade – less customizations made in SharePoint, the easier upgrade (e.g. to SharePoint 2015) becomes
• Possibility of transitioning to cloud – if the organization is even half-considering a move to Office 365 (e.g. in the next few years), building solutions the “cloud-friendly” way means no/minimal re-work is required to move. Certainly we have more than one client who has asked us to build this way – because they want to leave the door open for Office 365 (or hybrid) as they move forward

Some different flavors of off-box code include:

• Client code running in the browser - use of the JavaScript Object Model (JSOM), or the REST API from JavaScript
• Remote server-side code – use of the .NET Client Object Model (CSOM), or the REST API from the server

For the latter case, you need to identify a non-SharePoint server where the CSOM code can run. Common choices here include using Azure (or similar cloud service) or providing your own IIS servers – since an app is typically required for the “trust” of the remote code, this is the notion of “provider-hosted apps” which you have probably heard about. As I’ve often said, I really like the Azure option because so many infrastructure things are taken care of (SSL, High Availability, ability to scale up, backup/restore, published to the internet and so on).

As a reminder, Microsoft supply *lots* of useful remote code in their Office App Model Samples pack (known as the “AMS samples”) – this deals with many common scenarios you might encounter. But you still need to figure out where to run the code from, and go through the “remote code” configuration steps to enable this (or any remote server-side code) to run.

A guide – my subtitled video showing end-to-end config of remote code in Azure

There are various resources related to apps and remote code, but the steps are fiddly and sometimes it’s difficult to follow the entire process. So I decided to produce a video, with lots of captions explaining each step – the scenario I’m using is configuring a Remote Event Receiver (RER) in Azure. Note, however, that perhaps 90% of the steps apply to *any* remote server-side code scenario where the hosting location is Azure. So if you have any kind of provider-hosted app and want to run it from Azure, you’ll need the same set of steps (even though you’re not using RERs).

The image below might give you a sense of the video/captions:

Links

• The YouTube link to my video is: http://cob-sp.com/RER_Config_Video
• It’s also embedded into my presentation I linked to at the beginning of this post: Modern SharePoint Development – techniques for off-box code

If you have any feedback/questions, please publish them here and I’ll try to respond. Thanks!

Presentation deck: Modern SharePoint development – techniques for off-box code

I recently gave a couple of talks at the SharePoint Evolution 2014 conference (a roadshow no less! Check out the tour bus to the left!) I spoke in London and Cambridge - I had a great time as a speaker, and had some great conversations with attendees.

I created some new materials for these talks (slide decks/demos etc.) and will publish links to these over the next couple of posts, starting here. For the first time, I’ve recorded my demos and published the videos to YouTube (links in the slide deck) – so hopefully much of the information conveyed in the demos is available too.

"Modern SharePoint development - techniques for moving code off SharePoint servers"

Background

Almost 2 years after many of us first began to get exposure, it’s fair to say that “SharePoint apps” are not going away. Although Microsoft certainly have changed tack on some things in the SharePoint dev world, it does seem that the apps/remote code model is something that Microsoft are committed to and will continue to invest in. Whilst they’re not right for every scenario, I generally do also buy into the message that building solutions in this way can help even for on-premises deployments. Some specific points here include:

• Greater isolation – an app running “off-box” cannot bring down SharePoint for the entire organization
• Ease of upgrade – less customizations made in SharePoint, the easier upgrade (e.g. to SharePoint 2015) becomes
• Possibility of transitioning to cloud – if the organization is even half-considering a move to Office 365 (e.g. in the next few years), building solutions the “cloud-friendly” way means no/minimal re-work is required to move. Certainly we have more than one client who has asked us to build this way – because they want to leave the door open for Office 365 (or hybrid) as they move forward

Speaking to attendees I realise that for many folks, Microsoft’s push to this “Cloud Application Model” hasn’t really resonated yet. They ask “why change, especially when new skills are required and many common SharePoint things become more difficult to implement?”. Well, I do agree that it doesn’t always make sense. No-one is saying you have to do things using these techniques. If, for example, the organization is sure that the cloud isn’t the right solution for them in the next few years, and they are happy with the deal they get in managing SharePoint (including any upgrade projects they might undertake), then fine – you might be able to deliver more business value using farm solutions and full-trust code.

In general, however, I think at least an awareness of this emerging trend (towards cloud-friendly SharePoint development) is beneficial. Hopefully this slide deck is of some use here, but there are many other good resources too – I list some at the end of this article

Presentation

There are 3 main areas – each with an accompanying demo/YouTube video:

• Running code in Azure (Remote Event Receivers used as example)
• Using "PowerShell + CSOM" scripts
• Analysis of Microsoft's "AMS Samples"

Chris O'Brien - Modern SharePoint development: techniques for moving code off SharePoint servers from Chris O'Brien

Original abstract

Those implementing Office 365 know that custom code cannot run on the SharePoint servers, and remote code alternatives must be used. However, it can be valuable to adopt the same "cloud-friendly" development techniques even for on-premises implementations. In addition to increasing platform stability, this enables you to build in flexibility to your deployment - leaving the door open to a potential move to Office 365 in the future without major re-engineering. With several code-focused demos, this session examines approaches for switching to "off-box" code - including Remote Event Receivers, PowerShell and CSOM scripts, and a nod to JavaScript-based approaches. We’ll also take an in-depth look at Microsoft’s Office App Model Samples (AMS) and why you should care about them.

Resources/further reading

• Office App Model Samples
• Vesa Juvonen’s blog
• Reimagine SharePoint development

Speaking on Office 365 APIs and “modern” SharePoint dev techniques at the SharePoint Evolutions 2014 Roadshow

I’ve been in head-down mode recently preparing two new talks, which I’m looking forward to giving at the upcoming SharePoint Evolutions Roadshow here in the UK. It’s going to be a great event – pretty sure it’s the first time I’ve heard of a “roadshow” conference, but it’s a great idea! As usual for the SharePoint Evolutions event, there’s a great line-up of speakers from the UK, Europe, the U.S. and elsewhere. The roadshow visits 12 UK locations in total – I’m speaking at London and Cambridge, but many of the other speakers (including the U.S. guys) are doing LOTS of locations! Even so, no two locations will have exactly the same speakers/content, so the idea of going to more than one day works out too.

I’ll be speaking on:

 

An intro to the new developer APIs for Office 365 – London, June 10, 3:45pm

[This is a joint session with Wes Hackett]

Microsoft have stated that they will move away from SharePoint, Exchange and Lync APIs towards generic Office 365 APIs. The preview version of the first APIs were announced in March 2014, and cover several core building blocks for user collaboration - including mail, OneDrive files, calendar entries and contacts. Developers who need awareness of Office 365/hybrid scenarios can expect more such APIs in the future. We’ll demo the core REST API, the recently released .NET client API (which helps simplify authentication etc.) and even show accessing Office 365 from a “multi-device hybrid app” (aka Cordova) using the new JavaScript API for Office 365.

Get on the bus early - come learn about the APIs and the new permissions model ("the Common Consent framework") in this code-focused session.

Modern SharePoint Development - techniques for moving code off SharePoint servers – Cambridge, June 11, 1pm

Those implementing Office 365 know that custom code cannot run on the SharePoint servers, and remote code alternatives must be used. However, it can be valuable to adopt the same "cloud-friendly" development techniques even for on-premises implementations. In addition to increasing platform stability, this enables you to build in flexibility to your deployment - leaving the door open to a potential move to Office 365 in the future without major re-engineering. With several code-focused demos, this session examines approaches for switching to "off-box" code - including Remote Event Receivers, PowerShell and CSOM scripts, and a nod to JavaScript-based approaches. We’ll also take an in-depth look at Microsoft’s Office App Model Samples (AMS) and why you should care about them.

It’s not too late - convince your boss you need to attend for a day!

Since the roadshow is based around the idea of a series of one day events, it’s perhaps less of a big deal to escape from the office and clients to gain some key knowledge (than say, a 3 day conference). As you’d expect, the costs are also lower (£125 per day) and the organisers (Combined Knowledge) are even offering a “register 1, get 1 free” offer for a limited time as we close in on the event! 

More info at:

https://www.sharepointevolutionconference.com

Office 365 hybrid and search – presenting results from both on-premises and SharePoint Online sites

In my previous post about Office 365 and on-premises SharePoint hybrid deployments, we discussed the idea that, whilst it can be the right solution in many cases, a hybrid deployment is NOT simple and really only encompasses search, BCS, and Duet (for SAP) integration. Other aspects around document management, sites/navigation, metadata, customizations, and perhaps any other thing you care to mention are not synchronized or joined-up in any way. As a reminder, these articles are structured like this:

1. Office 365 SharePoint hybrid – what you DO and DON’T get
2. Office 365 hybrid and search – presenting results from both sides [this article]

Even in search, one of the cornerstones of hybrid, what you get might not be what you’re hoping for – results from the “remote” environment are displayed in a separate Result Block or Promoted Result. Here’s a reminder of the (more powerful) Result Block approach:

Clearly this isn’t the best user experience. And the alternative, Promoted Results, are more about defining an individual result to appear at the top for a given search (i.e. a Best Bet) – so that’s great for individual results but not entire sets.

Last time, I discussed some of the questions and issues that can be raised by the current experience:

• How many results should be shown (from each source) on page 1?
  • HINT – you’re only allowed a maximum of 10 for a Result Block anyway :)
• What about paging? What happens when I go to page 2?
• How can I easily see more results from both sources?
• How do I determine which results are more relevant, from the local or remote environment?
• Are there any options for seeing the results in a more integrated way?

Important news on this topic (April 2014)

At the recent SharePoint Conference 2014, Microsoft stated that they were looking at this problem (of disjointed search results in hybrid deployments), and hope to have a solution later this year. So, hopefully this whole problem will go away at some point. In any case, I think it’s still worthwhile discussing the current situation and possible interim techniques to work around it.

 

Investigation – can results be merged using code?

So, given the limitations of of the current out-of-the-box user experience, I was keen to see if there are other options. Indeed, for some things we are putting together for my current client, the requirement is very much “just search across everything and show me the most relevant things!” I think there’s nothing wrong with wanting that as the user experience. So as a developer, I wondered if going direct to the search API would provide any other options – perhaps the standard search page is built in a certain way, but the API has more flexibility. I’d also previously noticed a very interesting property in the API, so I started there:

Investigation 1 - using the EnableInterleaving property on the Query class

The SharePoint search API has an interesting option which can be used when writing custom code for SharePoint search. The name of the property in the API is “EnableInterleaving” on the KeywordQuery object – this sounded promising in terms of potentially being a native way of merging the result sets. The Microsoft documentation is (currently) not hugely clear on what the setting does, at least not to me anyway:

“A Boolean value that specifies whether the ResultTable objects in the ResultTableCollection produced by running this query should be interleaved.” (from http://msdn.microsoft.com/en-us/library/office/microsoft.sharepoint.client.search.query.query.enableinterleaving.aspx).

To start testing, I configured hybrid search in my dev environment – I used the outbound hybrid model, where the search center site exists on-premises but reaches out to Office 365/SharePoint Online to bring in results from there. (TechNet documentation is at Configure hybrid search for SharePoint Server 2013, or numerous sessions at the SharePoint Conference on this next week for those attending). At this point, the out-of-the-box search results page gave me the experience shown in the image at the start of the article, with the separate results blocks (highlighted in red/blue boxes).

I then wrote some custom code which allowed me to test this config option – effectively I have a custom web part with some CSOM (JSOM) code which runs a search query using the API. I have some URL parameters which enable me to specify the query text and whether EnableInterleaving is enabled or not. Unfortunately, EnableInterleaving does NOT interleave results between Office 365 and on-premises SharePoint. The main finding is that multiple result sets are still returned (i.e. for Office 365 and on-premises), and results are not combined together in any way. One difference I did observe, is that is an additional set of results is returned (a ResultTable) for personalized results (“PersonalFavoriteResults”), but it’s empty for me (most likely because this is a non-production environment where analytics aren’t in use. I’m sure SharePoint does natively populate/maintain this under real usage – it stores results previously clicked on by the user for this search).

Conclusion: - the EnableInterleaving property does NOT help us combine search results across Office 365 and on-premises SharePoint.

Investigation 2 – manually merging results in custom code

Anyone very familiar with SharePoint search will know that for any given query (e.g. a search for the query text “SharePoint”), the search engine can return different “tables” of results – perhaps think of these as “flavors” of results. They are:

• Relevant results (“RelevantResults”)
• Best Bet results (“SpecialTermResults”)
• Personal favorite results (“PersonalFavoriteResults”)

Usually it’s the “relevant results” that we care most about – this table has the core set of results for the query. If our goal is to combine results from Office 365 and on-premises SharePoint, really what we want is a single “relevant results” table containing items from both locations. However, when we’re in hybrid mode, what we actually get by default is two separate “relevant results” tables – the image below shows my custom code rendering these tables, and you can see the similarities between this and the out-of-the-box search results page (which displays a separate Result Block for the Office 365 results):

So, if we want to display combined results (rather than separated), we *could* write some code to merge the result tables together (e.g. in memory). But what logic would we use? How should the results be sorted? Well, one of the columns returned in the result tables is “Rank” – this is the “relevancy score” of the result compared to the query which was used (all search engines have such a calculation – like Google’s PageRank). So the question is – can we use Rank to compare results from different search indexes? Is it valid to merge the two result sets together, and then sort the full list by rank? After all, this can definitely be achieved with custom code – here’s my custom web part which goes direct to the search API to do this:

I had a feeling it wasn’t valid to do this. So I asked some folks including Microsoft, other MVPs etc. for input and my suspicions were confirmed – Wictor summed it up best with this response:

“Interleaving results from two different indexes would produce very strange results. Since the ranking is based on the analytics you basically need to "normalize" the rank and analytics data between the two (or more) different indexes. Assume the user behaves differently in the different environments (which is not uncommon - one environment might be collaboration and the other some other workload such as BI).”

So in effect, you can mash up the results, but not in a way which gives you the most relevant results first (across both environments) – the sorting by rank is artificial, and shouldn’t be trusted. However, it is worth considering that if you *don’t* care about relevance/rank, then this custom approach works fine. For example, if the requirement is to display results sorted by last modified date (or by something else, such as author, site URL or whatever) then happy days! You could use this custom code approach to merge results from your Office 365 and on-premises sites just fine in those cases.

The code

If you’re interested in this approach, here’s the code to do it. I used JSOM code to execute the search, but REST should be another option. When I have the two result tables, I iterate the results in each and add to a combined collection. Finally I sort the results, with a little help from Underscore.js. All this JavaScript is referenced by a custom web part, that I deploy to my on-premises site and add to a page there. So long as hybrid is properly configured, my code can reach across to the associated Office 365 environment and obtain search results from there too.

This is the initial code to execute the search – the main thing to note here is that there is NOTHING different needed in the code to “use hybrid mode” or “enable hybrid” or anything. The results automatically come back from both sources if the infrastructure is configured. You might notice I pick up some URL parameters for the query text and EnableInterleaving properties – you would need to integrate this with your UI as appropriate:

In the success handler, some different code *is* required here. Since 2 result tables are returned, here we do some processing to combine the results into a new JavaScript array, and implement any sorting. The code below provides different options for sorting – by rank (even though we’re saying it’s not valid under hybrid), and last modified date:

..and here’s the full JavaScript code with all my helper functions etc. (NOTE: I’m still leaving it as an exercise for the reader to implement this somewhere – I used two custom web parts which referenced this JavaScript, so providing the exact implementation you need is up to you):

Summary

At the current time the user experience for search in hybrid mode isn’t necessarily as optimal as we would like it to be, because results are displayed separately for Office 365 sites and on-premises SharePoint sites. In some cases, users will just want to “search across everything” and not care where the results live.

However, under some circumstances you might be able to use custom code to “join-up” the results. Specifically, this should work well if you are happy to display results sorted by something *other* than rank – perhaps last modified date, the author name or some other piece of metadata. Sorting by rank across two SharePoint environments is not valid, since for one reason, rank uses SharePoint’s analytics framework, which operates within the scope of an individual environment only (i.e. it doesn’t do anything magical for hybrid deployments).

In the future, Microsoft will hopefully provide capability which will address this problem “natively”, without the need for custom code. Then, it will be possible to show results across Office 365 sites and on-premises SharePoint sites together. Until then, the solution presented here might be an option for you.