Wednesday, 15 October 2014

Speaking at TechEd 2014 on Apps for Office

TechEd Europe is just round the corner (October 28-31, in Barcelona), and recently I’ve been heads-down preparing TEEU_2014_Im speaking_3to speak there on the Office 365 track. Most of the other speakers on this track are Microsoft folks from the Office 365/SharePoint/Exchange product groups (including Rob Howard, Brian Jones, Vesa Juvonen, Steve Walker, Jeremy Thake, Mauricio Ordonez and Sonya Koptyev) so there’ll be some *great* information being dispensed – my aim is to sprinkle in some field experience there too :) As an added bonus, I hear that TechEd will have an announcement or two that developers in this space will like.

My session is on Apps for Office, and I’m fortunate to be co-presenting with Andrew Salamatov from the Exchange team. Andrew is a Program Manager whose team builds the capabilities around mail apps, and he has some very cool stuff to show in this area. Apps for Office in general are becoming very interesting to me, as it kinda feels like there hasn’t been huge amounts of take-up of Apps for Office since the launch – maybe that’s confined to my corner of the Office 365/SharePoint-focused development world, but I think it’s surprising given the power and possibilities there. After all, many users LIVE in Office applications, and from the dev perspective it’s really just web development with some Office-specific JavaScript APIs these days – WAY less scary than previous models :)

Overall, Andrew and I have a couple of cool demos to show, including a Word task pane app which analyses the contents of a document to find you similar items across your SharePoint environment/tenancy whilst you're working with the document.

Here are the session details:

DEV-B307: Weave your enterprise solutions directly into Office products with the App Model (Friday, October 31 12:00 PM - 1:15 PM Room: Hall 8.0 Room D4)

Apps for SharePoint and Apps for Office allow you as an enterprise developer to weave your existing enterprise solutions throughout the Office products that your users live in day to day. In this session you will see compelling real world examples that light up existing Office products in context of key enterprise scenarios. The session will highlight tips and tricks on architecting this solution in Visual Studio along with Patterns & Practices guidance.

  • Understand what scenarios are available to surface your customizations in Office products
  • See real world enterprise scenarios that span multiple products stitching together an enterprise solution
  • Learn tips on how to architect enterprise solutions with the App Model

Hopefully see you there!

Monday, 8 September 2014

JavaScript-based provisioning in SharePoint/Office 365

In this post I want to specifically focus on the idea of JavaScript-based provisioning code – a potentially useful approach which has a place in every modern SharePoint developer’s toolkit. This is the idea of using SharePoint’s JSOM API (i.e. the JavaScript version of CSOM) to perform various one-time configuration steps for SharePoint sites - I was initially somewhat sceptical of this technique (JavaScript? For provisioning? Would that be reliable??), but having seen some team-mates use it with success on projects (kudos guys!), I think it’s worthy of consideration.

Background

For SharePoint developers who are working with SharePoint Online/Office 365, or are working on-premises but choosing not to use full-trust WSPs simply to ensure their solution is “cloud-friendly”, the previous options for custom SharePoint code are no longer available. And some common scenarios still require code – especially when some form of custom site template is being developed, or we are working towards automated configuration of sites.

Some examples of provisioning steps that a developer might need to use code for could be:

  • Binding Managed Metadata/taxonomy fields
  • Setting a theme/Composed Look on a site
  • Setting the master page of a site
  • Modifying navigation settings
  • Branding OneDrive (personal) sites
  • [Optional] Provisioning fields and content types through code, rather than XML

This presents a challenge when server-side SharePoint code cannot be used. As we all know, Microsoft have deprecated the use of sandboxed SharePoint code (which we might previously have used for say, a Feature receiver), and this facility may be turned off in Office 365 in in the future. So, we have to use one of the remote APIs instead – and the code will either run from some other server (e.g. the .NET CSOM) or the user’s browser (JSOM).

Options for remote code

Broadly, the options for remote code include:

  1. Some use of the .NET CSOM API:
    1. Code deployed to a cloud service such as Azure, or running off your own server somewhere. This could be an “App for SharePoint”, or it could just be remote code which does not use app authentication.
    2. A “PowerShell + CSOM” script you wrote
  2. Some use of the JSOM API (i.e. the JavaScript version of CSOM)
  3. Some use of the REST API (either from JavaScript or remote server-side code)

As Microsoft put it, your custom code either needs to move UP to the cloud or DOWN to the user’s browser.

The JavaScript-based (JSOM) option

The idea of using JSOM code for site provisioning steps is a strange one initially. What happens if the user closes their browser as the code is running? Will the site get configured properly? What about if the user is very far away from the server on a slow connection? It was questions like these that made me sceptical initially, but with some safeguards and caveats in place I think it’s an interesting option for remote code. I’ll talk about how it works in a second, but first here’s why I think it’s interesting:

Side-stepping the “server-side remote code” issue

As a modern SharePoint developer, I’m all onboard with apps and remote code. In several cases I’ve implemented remote SharePoint code in Azure (e.g. see my Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps) post). So yep, I’m cool with that stuff! And I’m also down with kids in terms of Microsoft’s OfficeDev Patterns and Practices samples, which cover a ton of scenarios such as those I list above – they’ve effectively written a great wrapper around the .NET CSOM API calls, which make it really easy to get these tasks done.

BUT THIS TYPE OF CODE STILL HAS TO BE DEPLOYED SOMEWHERE! On-premises servers (e.g. some IIS boxes) can bring complexity in terms of high-availability, access outside the firewall, SSL certificates, backup and restore, and so on. Of course, cloud options exist too. If you (or your client) is already using Azure, then GREAT! It’s really not too difficult to get the code deployed and running there, and Microsoft take care of the operational/I.T. Pro factors listed above. But in many of the organizations I’ve worked with, unfortunately Azure (or similar cloud service) is not yet embedded in the I.T. department’s toolkit. There is no Azure subscription, with someone in I.T. already paying the monthly invoice. And there are all sorts of operational I.T. topics to discuss and get agreement on before we could get to that position – who owns responsibility, what kind of data will we be storing there, who has access, how does it fit with any existing SLAs/OLAs, which department/cost centre pays the bill and so on.

Interestingly, working in an “implementation partner” as my team does, we almost have an option of deploying some remote code for a client to Azure and not needing to involve them in this decision. The Azure Websites offering has a free option, and therefore all the billing questions/set-up could be avoided. However, you do not get high availability in free mode – so if the machines your code is running on are being patched, you’ll have downtime. So in the end, unless your code is trivial this probably isn’t an option.

Consequently, the idea of implementing remote code in JavaScript can be quite appealing in some cases.

A robust implementation for JavaScript-based provisioning

So, we’d need some way of making a JavaScript-based approach somewhat reliable. The pattern my guys have used is this:

  • The SharePoint site is created (perhaps from a custom WebTemplate which takes care of some settings in XML) – at this point, the site is NOT fully configured
  • A custom JavaScript file is referenced on every page – this checks to see if set-up steps have been completed yet. If not, the user’s permissions are checked to see if they are a Site Owner and whether we can proceed (with a “A site owner needs to visit this site to configure it” message shown if not).
  • When a user hits the site, our code runs. If we can proceed with set-up, a dialog is shown - “Please wait – we are getting your site ready”
  • The code starts to execute provisioning steps – perhaps setting a Composed Look and/or custom master page, binding taxonomy fields and so on
    • When each step completes, an entry is written to the web property bag e.g. “BrandingApplied = true”
  • When all steps are complete, a success message is shown and the “getting your site ready” dialog is automatically closed. The site is now ready for use.

As you can guess, it’s only when all confirmations are written to the property bag that we no longer attempt to run the steps. This works surprisingly well in practice – users/site owners are comfortable seeing these kind of messages, and having a site owner hit the site initially tends to work out fine, since they are the site creator. If any failures occur, well that’s fine too because the uncompleted steps will be re-tried on the next page load.

To illustrate the experience, here’s what the site owner might see during configuration (shown here on a test page, rather than a “real” site home page):

Dialog large

Of course, you could do something a bit fancier in terms of look and feel.

Some sample code for JavaScript-based provisioning

In my other article Three cloud-friendly ways for developers to provision Managed Metadata fields in SharePoint/Office 365, I show some JavaScript code which does the basics described above – specifically for binding taxonomy fields to MM Term Sets. This is the same sample as in the other article - I’ve included it again below for convenience, but I recommend reading the other article too as it has some extra notes about how the code should be used (like the fact you might want to enhance with JavaScript promises). Note that it doesn’t deal with setting confirmations in the web property bag, but it would be fairly simple to add this. The code will work with Office 365 or on-premises SharePoint:

I guess the main thing to take from this code is the approach - showing the dialog whilst several steps of JSOM code are being executed. In this simple example, when the code completes our taxonomy field will be bound and ready for use – but in our real-life projects obviously many more steps are performed.

Summary

Whilst the idea of using JavaScript/JSOM code for provisioning can seem strange, I believe it has it’s place and can be the right choice sometimes. Certainly, if you have no other need for remote code and/or do not have a hosting location (such as Azure or on-premises IIS servers) ready to go in a production sense, then the JSOM approach does offer you a way of running remote SharePoint code without the hassles that come with the alternatives. As always, consider the trade-offs and think about what’s right for you – but hopefully the information here helps you along the way.

Three cloud-friendly ways for developers to provision Managed Metadata fields in SharePoint/Office 365

Provisioning Managed Metadata fields “the developer way” brings some additional complexity where SharePoint full-trust solutions cannot be used, such as Office 365/SharePoint Online deployments or on-premises projects where cloud-friendly techniques are being used. In the past, a developer would use a Feature receiver to bind taxonomy fields to Term Sets – but this isn’t possible in cloud scenarios. I wrote about this previously in Provisioning Managed Metadata fields in Office 365 – here I discussed the problem, and provided an approach we’ve used previously which does work in the cloud. As the article shows, it’s actually possible to use a fully-declarative (XML-based) way of provisioning Managed Metadata fields, so long as some GUIDs which relate to your Office 365/SharePoint instance are inserted into the XML.

Downsides to the declarative approach

The article also discusses the fact that if you work against multiple environments (tenancies or on-premises), at least one different GUID (for the Term Store ID) needs to be used in the XML. This effectively leads to different WSP packages being deployed to different environments – of course, this is not good in ALM terms! In our case, we were fairly comfortable with this trade-off since our TFS automated build process dealt with building packages for different environments, and we didn’t feel much pain. However, without an automated build process it can be more problematic – and now that more time has passed, I think it’s worth stepping back and discussing other options for solving this common problem.

3 ways to bind taxonomy fields

Before delving into the detail of each one, here are the main options as I see them (N.B. I’ve used this list in some of my other articles):

  1. The 100% declarative approach (as discussed above and in my other article)
  2. Using some form of remote server-side .NET CSOM code:
    1. The Microsoft sample in the Office Dev Patterns and Practices samples (previously known as the AMS samples)
    2. Some custom code you wrote
    3. Some custom “PowerShell + CSOM” script you wrote
  3. Using JSOM code running in the user’s browser, which I discuss in a companion article to this one - JavaScript-based provisioning in SharePoint/Office 365

Notably, the last two options aren’t really limited to “the taxonomy field problem” – both of the remote CSOM or JSOM approaches could be used as part of a wider strategy for automated provisioning i.e. any setup steps which require code. My quick list of things which commonly need to be achieved here include:

  • Binding Managed Metadata/taxonomy fields
  • Setting a theme/Composed Look on a site
  • Setting the master page of a site
  • Modifying navigation settings
  • Branding OneDrive (personal) sites
  • [Optional] Provisioning fields and content types through code, rather than XML

So, if you have other items on this list you need to accomplish *or* you just prefer to avoid the declarative approach for Managed Metadata fields because of the ALM trade-offs, then you’re probably looking at a choice between the .NET CSOM or JSOM approaches. For .NET CSOM code, the Office Dev sample code works just fine - but, of course, you have to deploy and host it somewhere, whether it’s a cloud service like Azure or your own IIS servers. And this isn’t necessarily simple for a production-grade enterprise implementation - in the “Side-stepping the server-side remote code issue” section of JavaScript-based provisioning in SharePoint/Office 365, I cover this debate in more detail. As I conclude there:

The idea of implementing remote code in JavaScript can be quite appealing in some cases.

And if steps are taken to increase robustness, I think this approach is OK. So with all that in mind, let’s see what provisioning Managed Metadata/taxonomy fields in JSOM code looks like:

Provisioning with JSOM code

The sample below shows the code, but if you’re interested in using it also see the notes below:

Some notes on this code sample:

  • The code will work with Office 365 or on-premises SharePoint
  • One of the core benefits is that the code allows you to specify the *name* of the Term Set, rather than it’s ID. This is useful since this GUID will be different in different environments, unless you’ve taken specific steps to avoid this (i.e. by creating the Term Set programatically)
  • In fact, three things need to be specified – the name of the Term Set, the locale of the Term Set, and the internal name of the field – you can see these at the top of my code. Ultimately they get passed to the initTaxObjects() method which starts the process
  • I’ve kept the code simple by not using JavaScript promises, but that could be an exercise for the reader if you’d like to enhance it in that way
  • The method is currently set up to process one field at a time, but it could be extended to accept an array of field names/term set IDs
  • Current there’s a delay of 4 seconds inserted for illustration purposes – make sure you remove this if you implement in production :) Similarly there are lots of console.log lines which you might choose to remove
  • You may need to deal with script dependencies differently if not running in a SharePoint page (e.g. you’re code is running in an app)
  • In current form, the code above expects some calling code somewhere to trigger the execution, and a DIV with ID of "jsomProvisioningMessage" (for some simple logging messages) – you could have both of these in a custom web part or Script Editor Web Part added to a page. This is the code:

    <div id="jsomProvisioningMessage" />
    <script type="text/javascript">
    $(document).ready(function () { 
      window.COB.JsomProvisioning.execute();
    }); 
    </script>

Whether it’s this approach you use or an alternative, if successful your taxonomy field should look like the following (before and after shown):

Before (unbound):

Taxonomy field before binding

After (field is now bound and can be used for tagging):

Taxonomy field after binding

Summary

Managed Metadata/taxonomy fields have always been a bit of a pain for developers, because there needs to be an association between the field and a Term Set, and these have different IDs in different environments/tenancies – so it often needs to be dynamically looked up/specified. Assuming we are implementing the site the developer way with automated provisioning, this can be done 100% declaratively (with the ALM trade-off), with .NET CSOM code or with JSOM code. In this post I’ve focused on the JSOM approach – since if you have no other requirements for remote code (and therefore no Azure subscription ready to go), this method can be appealing.

Tuesday, 22 July 2014

Remote Event Receivers on host web gotcha – no context token/ClientContext

I’ve spent quite a bit of time working with Remote Event Receivers in SharePoint 2013/Office 365, and in the early days I hit an issue that initially made me think they didn’t work properly at all. Now with some help, I realize that mainly the code just has to be written a certain way - but it’s also true to say that there is a limitation which can affect some things you might want to do. In any case, this is a gotcha that I’m starting to see other people hit so I think it’s worth talking about it.

Specifically the issue occurs when:

  • The event receiver is on the host web (rather than the app web)
  • The event receiver uses CSOM code to talk to SharePoint (rather than just using NON-SharePoint code)

Issue symptoms

When you’re trying to use RERs on the host web, what you might see is that whilst your event receiver always fires (and your remote code does execute), your code fails to authenticate back to SharePoint. You’ll get a 401 error when trying to obtain or use a ClientContext object – this could manifest itself with one of these in your code:

System.NullReferenceException: Object reference not set to an instance of an object

..but if you look at the HTTP requests/responses, you’ll see the HTTP 401 (Access Denied).

This means you effectively can’t do anything with SharePoint – and often, of course, the whole point of your event receiver is that it’s going to do something in SharePoint! Perhaps you need to create, update or delete an item somewhere, or perform some other action related to sites, users or data in SharePoint.

Other info:

  • No context token
    • If you do some debugging, you’ll find that the event receiver does NOT receive a valid context token from SharePoint/Office 365 - SPRemoteEventProperties.ContextToken is an empty string. This causes all the later code to fail. At this point, you probably think there’s a bug with SharePoint/Office 365 – even at the end of all this, I’m personally not sure whether there is a bug or it’s just a “by-design” limitation. Fortunately, as we will see there is a workaround which works for many (most?) scenarios of this type.
  • The same code works on the app web
    • If you do some wider testing, perhaps to see if there is a code issue – you’ll find the very same code works fine in other contexts. For example, if you use the same code in a RER on the app web (e.g. “ItemAdded” on a list in the app web), everything works fine. The issue is purely related to RERs on the host web.
  • No change if you use ProcessOneWayEvent (asynchronous events such as added, deleted etc.) or ProcessEvent (synchronous events such as adding, deleting etc.)
    • Remote Event Receivers are similar to traditional full-trust event receivers, in that both async and sync events can be used. An example here would be the “added” and “adding” events respectively – these correspond to the ProcessOneWayEvent and ProcessEvent methods in a Remote Event Receiver class. For reference, the issue we’re discussing here manifests itself in both cases.

So those are the symptoms. Before we dig into the “Why it happens” and “The workaround” sections, it’s worth reminding ourselves of some aspects of remote code and authentication in particular. Feel free to skip forward if you’re familiar with this territory.

A reminder on authentication of remote code

For those just starting out with Remote Event Receivers (or SharePoint apps/remote code in general), it’s worth noting that an app is required for any form of remote code. More correctly, we could expand this to say any remote code which runs on the server-side and uses “app authentication” - as opposed to “user authentication” where a username/password is specified. In any case, this can seem strange for some things like Remote Event Receivers, where there isn’t really an “app” (e.g. with some pages) that the user will enter/click around etc. All we have for a RER is a remote WCF service with no front-end! But, this is the way the trust model works for remote code – so the first thing to say is that if you don’t have an app registered and installed/trusted for your remote code, it will not work.

If we were to summarize common authentication options for remote SharePoint code, we could break it down like this:

  • User authentication
    • The username/password of a “named user” is passed – usually a very poor approach that brings security risks. Effectively this led OAuth to be invented as an alternative.
  • App authentication – this breaks down into:
    • User + app authentication – the default. Permissions of both the user *and* the app are checked
    • App-only authentication – the permissions of just the app are checked (thus allowing operations that the user themselves does not have permission to do)

Why it happens

If you’re hitting the issue, your RER most likely has code which instantiates the ClientContext object in a certain way - specifically, you’ll be using “user + app” authentication. The vast majority of MSDN and blog samples use this approach. This is ultimately the problem -  “user + app” authentication does not currently work for events related to the host web. My testing shows this is the case for both Office 365 and SharePoint 2013 on-premises installations. Along the journey of realising this, I tried some different coding approaches – and I see other developers perhaps doing the same (e.g. in forum posts). Since there are various ways the CSOM ClientContext object can be obtained, you might try different code for this – but several cannot be used in a RER and will fail. The table below shows some of the “wrong” approaches.

Code approaches which will NOT work:

Approach

Code sample

Why not

Using the typical RER approach ClientContext clientContext = TokenHelper.CreateRemoteEventReceiverClientContext(properties) This is the approach that “should work” (and the approach that most samples use). However, it only works for events on the app web – NOT events on the host web.
Creating ClientContext using constructor

ClientContext clientContext = new ClientContext(properties.ItemEventProperties.WebUrl)

The context is not obtained in a way which deals with authentication. This is what the TokenHelper methods are for.

Sidenote – it would be possible to authenticate with user authentication instead of app authentication, if the username/password of a named account was attached using the SharePointOnlineCredentials object. But that’s lame, and a big security risk! Instead, we want to use OAuth (via app authentication) of course, to avoid storing/passing passwords.
Using the “app event” approach ClientContext clientContext = TokenHelper.CreateAppEventClientContext(properties, false) The context is obtained in a way which is only appropriate for app events (e.g. AppInstalled/AppUpgraded/AppUninstalled etc.) Since we are using an event on the host web, authentication does not work.

As the table says, the one that *should* work is the first one (TokenHelper.CreateRemoteEventReceiverClientContext()) – but this relies on “user + app” authentication which appears to be the problem. So to work around this we need a different approach. 

The workaround

Instead of “user + app” authentication we can use “app-only” authentication. This requires 2 things:

  1. The app permission policy to specify that app-only calls are allowed (something the person installing the app must agree to). This is enabled in the app manifest in Visual Studio:

    Allow app-only auth
  2. The code in the app (i.e. in the Remote Event Receiver) to obtain ClientContext using an app-only access token. The expected way to do this in C# is to use the correct TokenHelper methods – big thanks to Kirk Evans for helping me with this. The code should look like this:

    ** N.B. There is a code sample here but it will not show in RSS Readers - click here for full article **
If you ensure these two things are in place, your Remote Event Receiver will work fine.

The limitation – when the workaround is no good

So, that’s the answer then. However, you might hit occasions where the app-only approach isn’t really what you want. If your remote code writes any data to SharePoint (e.g. adds a list item), you’ll notice that the SharePoint user interface makes it clear that the list item was added/modified by an app. Specifically, the username will be in the following format:

Last modified at [time] by [app name] on behalf of [username]

This is a pretty useful feature for some business requirements, since it makes it clear that although the named user was involved, they didn't make the change in SharePoint "directly". Unfortunately, use of the app-only authentication route to solve our problem changes this – specifically, with app-only auth only the app name is recorded as the user who made the change. Effectively, we lose the information which tells us which actual person was involved in the change - in some circumstances, this could be a problem if some kind of audit trail is required. To make things clear, here are two screenshots which show the difference:

List item added by "app + user" authentication:

List item added with App and User context

List item added by "app-only" authentication:

List item added with App Only context

Related sidenote (for the curious!):

If you’re wondering how SharePoint/Office 365 deals with this under the covers, you’ll find that all lists now have 2 new hidden columns - “App Created By” and “App Modified By”. These store a reference to an entry in the User Information list in the current site (just like other Person/Group columns):

AppCreatedByModifiedByColumnsOnList AppCreatedByModifiedByColumnsOnList2

So now you know ;)

Summary

It’s pretty easy to get things wrong with Remote Event Receivers, and the pitfall I’ve talked about concerning authentication and CSOM code is definitely a big gotcha – I certainly see other people hitting this. If you have remote code which modifies data in SharePoint somewhere, the way around the problem is to use app-only authentication - by ensuring the app’s permission policy allows this *and* also writing your CSOM code to authenticate in this way. This can be done by obtaining an access token using the TokenHelper.GetAppOnlyAccessToken() method, and then obtaining a ClientContext object using this token.

However, with this approach comes a trade-off - the loss of user information (in “pseudo-audit” terms) related to the user behind the change. Developers should bear this in mind when working with Remote Event Receivers.

Monday, 14 July 2014

Renewed as SharePoint MVP for 7th time

MVP_header_logo

I received the happy news recently that I continue to be a SharePoint MVP for another year – the 7th time for me. As always, it’s been great interacting with the community over the last 12 months. Every year I like to jot down on my blog the community “things” I did in the previous year - this list is mainly for me rather than you, but here are some the things I got up to:

It’s been a privilege to be part of the community, and I hope to continue to be useful over the next 12 months!

Thursday, 26 June 2014

Step-by-step video: remote SharePoint code in Azure

As I noted in my last post Modern SharePoint Development – techniques for off-box code, there is something of a movement in the SharePoint development world to what is often termed “the Cloud Application Model”. In a nutshell, this is the approach that many development tasks *must* use for Office 365, and *can* use for on-premises SharePoint. Specifically, the Cloud Application Model is about having custom code run “off-box” to SharePoint using the remote APIs rather than than the server-side API – after all, in Office 365 it’s simply not possible to deploy custom SharePoint code to Microsoft’s servers (this is not permitted because some bad code could affect the service for many clients). But as I noted last time, there can be benefits to this approach even in on-premises projects - such as:

  • Greater isolation – an app running “off-box” cannot bring down SharePoint for the entire organization
  • Ease of upgrade – less customizations made in SharePoint, the easier upgrade (e.g. to SharePoint 2015) becomes
  • Possibility of transitioning to cloud – if the organization is even half-considering a move to Office 365 (e.g. in the next few years), building solutions the “cloud-friendly” way means no/minimal re-work is required to move. Certainly we have more than one client who has asked us to build this way – because they want to leave the door open for Office 365 (or hybrid) as they move forward

Some different flavors of off-box code include:

  • Client code running in the browser - use of the JavaScript Object Model (JSOM), or the REST API from JavaScript
  • Remote server-side code – use of the .NET Client Object Model (CSOM), or the REST API from the server

For the latter case, you need to identify a non-SharePoint server where the CSOM code can run. Common choices here include using Azure (or similar cloud service) or providing your own IIS servers – since an app is typically required for the “trust” of the remote code, this is the notion of “provider-hosted apps” which you have probably heard about. As I’ve often said, I really like the Azure option because so many infrastructure things are taken care of (SSL, High Availability, ability to scale up, backup/restore, published to the internet and so on).

As a reminder, Microsoft supply *lots* of useful remote code in their Office App Model Samples pack (known as the “AMS samples”) – this deals with many common scenarios you might encounter. But you still need to figure out where to run the code from, and go through the “remote code” configuration steps to enable this (or any remote server-side code) to run.

A guide – my subtitled video showing end-to-end config of remote code in Azure

There are various resources related to apps and remote code, but the steps are fiddly and sometimes it’s difficult to follow the entire process. So I decided to produce a video, with lots of captions explaining each step – the scenario I’m using is configuring a Remote Event Receiver (RER) in Azure. Note, however, that perhaps 90% of the steps apply to *any* remote server-side code scenario where the hosting location is Azure. So if you have any kind of provider-hosted app and want to run it from Azure, you’ll need the same set of steps (even though you’re not using RERs).

The image below might give you a sense of the video/captions:

COB RER video snapshot

Links

If you have any feedback/questions, please publish them here and I’ll try to respond. Thanks!

Monday, 16 June 2014

Presentation deck: Modern SharePoint development – techniques for off-box code

WP_20140610_003I recently gave a couple of talks at the SharePoint Evolution 2014 conference (a roadshow no less! Check out the tour bus to the left!) I spoke in London and Cambridge - I had a great time as a speaker, and had some great conversations with attendees.

I created some new materials for these talks (slide decks/demos etc.) and will publish links to these over the next couple of posts, starting here. For the first time, I’ve recorded my demos and published the videos to YouTube (links in the slide deck) – so hopefully much of the information conveyed in the demos is available too.


"Modern SharePoint development - techniques for moving code off SharePoint servers"

Background

Almost 2 years after many of us first began to get exposure, it’s fair to say that “SharePoint apps” are not going away. Although Microsoft certainly have changed tack on some things in the SharePoint dev world, it does seem that the apps/remote code model is something that Microsoft are committed to and will continue to invest in. Whilst they’re not right for every scenario, I generally do also buy into the message that building solutions in this way can help even for on-premises deployments. Some specific points here include:

  • Greater isolation – an app running “off-box” cannot bring down SharePoint for the entire organization
  • Ease of upgrade – less customizations made in SharePoint, the easier upgrade (e.g. to SharePoint 2015) becomes
  • Possibility of transitioning to cloud – if the organization is even half-considering a move to Office 365 (e.g. in the next few years), building solutions the “cloud-friendly” way means no/minimal re-work is required to move. Certainly we have more than one client who has asked us to build this way – because they want to leave the door open for Office 365 (or hybrid) as they move forward

Speaking to attendees I realise that for many folks, Microsoft’s push to this “Cloud Application Model” hasn’t really resonated yet. They ask “why change, especially when new skills are required and many common SharePoint things become more difficult to implement?”. Well, I do agree that it doesn’t always make sense. No-one is saying you have to do things using these techniques. If, for example, the organization is sure that the cloud isn’t the right solution for them in the next few years, and they are happy with the deal they get in managing SharePoint (including any upgrade projects they might undertake), then fine – you might be able to deliver more business value using farm solutions and full-trust code.

In general, however, I think at least an awareness of this emerging trend (towards cloud-friendly SharePoint development) is beneficial. Hopefully this slide deck is of some use here, but there are many other good resources too – I list some at the end of this article

Presentation

There are 3 main areas – each with an accompanying demo/YouTube video:

  • Running code in Azure (Remote Event Receivers used as example)
  • Using "PowerShell + CSOM" scripts
  • Analysis of Microsoft's "AMS Samples"
Original abstract

Those implementing Office 365 know that custom code cannot run on the SharePoint servers, and remote code alternatives must be used. However, it can be valuable to adopt the same "cloud-friendly" development techniques even for on-premises implementations. In addition to increasing platform stability, this enables you to build in flexibility to your deployment - leaving the door open to a potential move to Office 365 in the future without major re-engineering. With several code-focused demos, this session examines approaches for switching to "off-box" code - including Remote Event Receivers, PowerShell and CSOM scripts, and a nod to JavaScript-based approaches. We’ll also take an in-depth look at Microsoft’s Office App Model Samples (AMS) and why you should care about them.

Resources/further reading

Friday, 30 May 2014

Speaking on Office 365 APIs and “modern” SharePoint dev techniques at the SharePoint Evolutions 2014 Roadshow

Speaker_web_banner I’ve been in head-down mode recently preparing two new talks, which I’m looking forward to giving at the upcoming SharePoint Evolutions Roadshow here in the UK. It’s going to be a great event – pretty sure it’s the first time I’ve heard of a “roadshow” conference, but it’s a great idea! As usual for the SharePoint Evolutions event, there’s a great line-up of speakers from the UK, Europe, the U.S. and elsewhere. The roadshow visits 12 UK locations in total – I’m speaking at London and Cambridge, but many of the other speakers (including the U.S. guys) are doing LOTS of locations! Even so, no two locations will have exactly the same speakers/content, so the idea of going to more than one day works out too.

I’ll be speaking on:

 

An intro to the new developer APIs for Office 365 – London, June 10, 3:45pm

[This is a joint session with Wes Hackett]

Microsoft have stated that they will move away from SharePoint, Exchange and Lync APIs towards generic Office 365 APIs. The preview version of the first APIs were announced in March 2014, and cover several core building blocks for user collaboration - including mail, OneDrive files, calendar entries and contacts. Developers who need awareness of Office 365/hybrid scenarios can expect more such APIs in the future. We’ll demo the core REST API, the recently released .NET client API (which helps simplify authentication etc.) and even show accessing Office 365 from a “multi-device hybrid app” (aka Cordova) using the new JavaScript API for Office 365.

Get on the bus early - come learn about the APIs and the new permissions model ("the Common Consent framework") in this code-focused session.

Modern SharePoint Development - techniques for moving code off SharePoint servers – Cambridge, June 11, 1pm

Those implementing Office 365 know that custom code cannot run on the SharePoint servers, and remote code alternatives must be used. However, it can be valuable to adopt the same "cloud-friendly" development techniques even for on-premises implementations. In addition to increasing platform stability, this enables you to build in flexibility to your deployment - leaving the door open to a potential move to Office 365 in the future without major re-engineering. With several code-focused demos, this session examines approaches for switching to "off-box" code - including Remote Event Receivers, PowerShell and CSOM scripts, and a nod to JavaScript-based approaches. We’ll also take an in-depth look at Microsoft’s Office App Model Samples (AMS) and why you should care about them.

It’s not too late - convince your boss you need to attend for a day!

Since the roadshow is based around the idea of a series of one day events, it’s perhaps less of a big deal to escape from the office and clients to gain some key knowledge (than say, a 3 day conference). As you’d expect, the costs are also lower (£125 per day) and the organisers (Combined Knowledge) are even offering a “register 1, get 1 free” offer for a limited time as we close in on the event! 

More info at:

https://www.sharepointevolutionconference.com

Thursday, 1 May 2014

Office 365 hybrid and search – presenting results from both on-premises and SharePoint Online sites

In my previous post about Office 365 and on-premises SharePoint hybrid deployments, we discussed the idea that, whilst it can be the right solution in many cases, a hybrid deployment is NOT simple and really only encompasses search, BCS, and Duet (for SAP) integration. Other aspects around document management, sites/navigation, metadata, customizations, and perhaps any other thing you care to mention are not synchronized or joined-up in any way. As a reminder, these articles are structured like this:

  1. Office 365 SharePoint hybrid – what you DO and DON’T get
  2. Office 365 hybrid and search – presenting results from both sides [this article]

Even in search, one of the cornerstones of hybrid, what you get might not be what you’re hoping for – results from the “remote” environment are displayed in a separate Result Block or Promoted Result. Here’s a reminder of the (more powerful) Result Block approach:

SharePoint hybrid search results - result block

Clearly this isn’t the best user experience. And the alternative, Promoted Results, are more about defining an individual result to appear at the top for a given search (i.e. a Best Bet) – so that’s great for individual results but not entire sets.

Last time, I discussed some of the questions and issues that can be raised by the current experience:

  • How many results should be shown (from each source) on page 1?
    • HINT – you’re only allowed a maximum of 10 for a Result Block anyway :)
  • What about paging? What happens when I go to page 2?
  • How can I easily see more results from both sources?
  • How do I determine which results are more relevant, from the local or remote environment?
  • Are there any options for seeing the results in a more integrated way?
Important news on this topic (April 2014)
At the recent SharePoint Conference 2014, Microsoft stated that they were looking at this problem (of disjointed search results in hybrid deployments), and hope to have a solution later this year. So, hopefully this whole problem will go away at some point. In any case, I think it’s still worthwhile discussing the current situation and possible interim techniques to work around it.

 

Investigation – can results be merged using code?

So, given the limitations of of the current out-of-the-box user experience, I was keen to see if there are other options. Indeed, for some things we are putting together for my current client, the requirement is very much “just search across everything and show me the most relevant things!” I think there’s nothing wrong with wanting that as the user experience. So as a developer, I wondered if going direct to the search API would provide any other options – perhaps the standard search page is built in a certain way, but the API has more flexibility. I’d also previously noticed a very interesting property in the API, so I started there:

Investigation 1 - using the EnableInterleaving property on the Query class

The SharePoint search API has an interesting option which can be used when writing custom code for SharePoint search. The name of the property in the API is “EnableInterleaving” on the KeywordQuery object – this sounded promising in terms of potentially being a native way of merging the result sets. The Microsoft documentation is (currently) not hugely clear on what the setting does, at least not to me anyway:

“A Boolean value that specifies whether the ResultTable objects in the ResultTableCollection produced by running this query should be interleaved.” (from http://msdn.microsoft.com/en-us/library/office/microsoft.sharepoint.client.search.query.query.enableinterleaving.aspx).

To start testing, I configured hybrid search in my dev environment – I used the outbound hybrid model, where the search center site exists on-premises but reaches out to Office 365/SharePoint Online to bring in results from there. (TechNet documentation is at Configure hybrid search for SharePoint Server 2013, or numerous sessions at the SharePoint Conference on this next week for those attending). At this point, the out-of-the-box search results page gave me the experience shown in the image at the start of the article, with the separate results blocks (highlighted in red/blue boxes).

I then wrote some custom code which allowed me to test this config option – effectively I have a custom web part with some CSOM (JSOM) code which runs a search query using the API. I have some URL parameters which enable me to specify the query text and whether EnableInterleaving is enabled or not. Unfortunately, EnableInterleaving does NOT interleave results between Office 365 and on-premises SharePoint. The main finding is that multiple result sets are still returned (i.e. for Office 365 and on-premises), and results are not combined together in any way. One difference I did observe, is that is an additional set of results is returned (a ResultTable) for personalized results (“PersonalFavoriteResults”), but it’s empty for me (most likely because this is a non-production environment where analytics aren’t in use. I’m sure SharePoint does natively populate/maintain this under real usage – it stores results previously clicked on by the user for this search).

Conclusion: - the EnableInterleaving property does NOT help us combine search results across Office 365 and on-premises SharePoint.

Investigation 2 – manually merging results in custom code

Anyone very familiar with SharePoint search will know that for any given query (e.g. a search for the query text “SharePoint”), the search engine can return different “tables” of results – perhaps think of these as “flavors” of results. They are:

  • Relevant results (“RelevantResults”)
  • Best Bet results (“SpecialTermResults”)
  • Personal favorite results (“PersonalFavoriteResults”)

Usually it’s the “relevant results” that we care most about – this table has the core set of results for the query. If our goal is to combine results from Office 365 and on-premises SharePoint, really what we want is a single “relevant results” table containing items from both locations. However, when we’re in hybrid mode, what we actually get by default is two separate “relevant results” tables – the image below shows my custom code rendering these tables, and you can see the similarities between this and the out-of-the-box search results page (which displays a separate Result Block for the Office 365 results):

Hybrid search - EnableInterleaving on 2

So, if we want to display combined results (rather than separated), we *could* write some code to merge the result tables together (e.g. in memory). But what logic would we use? How should the results be sorted? Well, one of the columns returned in the result tables is “Rank” – this is the “relevancy score” of the result compared to the query which was used (all search engines have such a calculation – like Google’s PageRank). So the question is – can we use Rank to compare results from different search indexes? Is it valid to merge the two result sets together, and then sort the full list by rank? After all, this can definitely be achieved with custom code – here’s my custom web part which goes direct to the search API to do this:

Hybrid search - custom interleaving 2

I had a feeling it wasn’t valid to do this. So I asked some folks including Microsoft, other MVPs etc. for input and my suspicions were confirmed – Wictor summed it up best with this response:

“Interleaving results from two different indexes would produce very strange results. Since the ranking is based on the analytics you basically need to "normalize" the rank and analytics data between the two (or more) different indexes. Assume the user behaves differently in the different environments (which is not uncommon - one environment might be collaboration and the other some other workload such as BI).”

So in effect, you can mash up the results, but not in a way which gives you the most relevant results first (across both environments) – the sorting by rank is artificial, and shouldn’t be trusted. However, it is worth considering that if you *don’t* care about relevance/rank, then this custom approach works fine. For example, if the requirement is to display results sorted by last modified date (or by something else, such as author, site URL or whatever) then happy days! You could use this custom code approach to merge results from your Office 365 and on-premises sites just fine in those cases.

The code

If you’re interested in this approach, here’s the code to do it. I used JSOM code to execute the search, but REST should be another option. When I have the two result tables, I iterate the results in each and add to a combined collection. Finally I sort the results, with a little help from Underscore.js. All this JavaScript is referenced by a custom web part, that I deploy to my on-premises site and add to a page there. So long as hybrid is properly configured, my code can reach across to the associated Office 365 environment and obtain search results from there too.

This is the initial code to execute the search – the main thing to note here is that there is NOTHING different needed in the code to “use hybrid mode” or “enable hybrid” or anything. The results automatically come back from both sources if the infrastructure is configured. You might notice I pick up some URL parameters for the query text and EnableInterleaving properties – you would need to integrate this with your UI as appropriate:

In the success handler, some different code *is* required here. Since 2 result tables are returned, here we do some processing to combine the results into a new JavaScript array, and implement any sorting. The code below provides different options for sorting – by rank (even though we’re saying it’s not valid under hybrid), and last modified date:

..and here’s the full JavaScript code with all my helper functions etc. (NOTE: I’m still leaving it as an exercise for the reader to implement this somewhere – I used two custom web parts which referenced this JavaScript, so providing the exact implementation you need is up to you):


Summary

At the current time the user experience for search in hybrid mode isn’t necessarily as optimal as we would like it to be, because results are displayed separately for Office 365 sites and on-premises SharePoint sites. In some cases, users will just want to “search across everything” and not care where the results live.

However, under some circumstances you might be able to use custom code to “join-up” the results. Specifically, this should work well if you are happy to display results sorted by something *other* than rank – perhaps last modified date, the author name or some other piece of metadata. Sorting by rank across two SharePoint environments is not valid, since for one reason, rank uses SharePoint’s analytics framework, which operates within the scope of an individual environment only (i.e. it doesn’t do anything magical for hybrid deployments).

In the future, Microsoft will hopefully provide capability which will address this problem “natively”, without the need for custom code. Then, it will be possible to show results across Office 365 sites and on-premises SharePoint sites together. Until then, the solution presented here might be an option for you.

Monday, 28 April 2014

Speaking at the European SharePoint Conference 2014

ESPC Speaker GraphicWe’re now just a week away from the European SharePoint Conference – held in Barcelona, the event looks great and I’m looking forward to being part of it. I understand this will be the biggest SharePoint event in Europe this year, and it certainly looks like there are some great speakers and sessions - including coverage of some stuff only recently announced at the Las Vegas conference.

The conference runs from 5th-8th of May 2014 – here is the full conference programme.

I’m presenting two sessions on the developer track:

“Developer Decisions, Tips and Tricks - Lessons Learnt from Office 365 Projects” (W13 – Wednesday May 7, 11:45)

As a developer or technical lead, your early Office 365 projects are likely to throw up some interesting questions. Should you avoid the sandbox? How should test environments be handled in the cloud? How should a site template be implemented? And just how do you provision Managed Metadata fields, when the on-premises techniques cannot be used? This session walks through the "dev strategy" decisions we made at Content and Code, and why. Over the course of several demos, we'll also discuss apps, automation scripts and also advanced techniques such as Continuous Integration for Office 365.

“Using JavaScript Templates To Customise the SharePoint 2013 User Interface” (TH20 – Thursday May 7, 14:00)

Developers often have a need to amend the HTML rendering of SharePoint, especially around lists and list forms, but in previous versions of SharePoint this was not always easy. Many developers resorted to using JavaScript to manipulate the page after it had loaded, even though this was potentially unreliable and unsupported. SharePoint 2013 solves this problem by providing the ability to specify a JavaScript 'template override' for several user interface elements. Even better, these can be applied globally or locally as needed. Come and learn how to transform the SharePoint interface the supported way!

Hopefully see you there!