Thursday 16 April 2015

Improving the Office 365 SharePoint hybrid experience

Back in February 2014, I wrote Office 365 SharePoint hybrid – what you DO and DO NOT get. In that article, I discuss the fact that whilst there’s lots of discussion about hybrid and indeed lots of organizations doing *something* in this space – in actual fact, what Microsoft provide is far from a turn-key solution and there’s quite a long list of things you need to work on. That is, if your organization wants to provide any kind of “joined-up” experience that works well for end-users, across sites in the cloud and sites in an on-premises SharePoint environment.

It’s fair to say that at the current time, hybrid from Microsoft really just consists of:

  • Search
  • BCS
  • A model for personal sites (OneDrive for Business sites) to live in the cloud, with some navigation for users, when other things are on-premises

The list of things you don’t get includes things like (N.B. the earlier article has a more complete list):

  • Global navigation
  • A global site directory
  • A joined-up social experience (unless you use Yammer)
  • Any kind of synchronisation of:
    • Branding
    • Customizations
    • Taxonomy/Managed Metadata
    • Content types
    • Apps
    • User profile data

..and so on.

I fully expect SharePoint 2016 to improve on this situation – I think it will be a key focus area for the product. But here in April 2015, having SP2016 in production is still some considerable time away for most.

Common requests for improving hybrid – user profiles, taxonomy and user photos

Since then I’ve worked on several hybrid deployments, and have been involved in work to close the hybrid gap and improve the experience. In this post I want to talk about some of the ground my team and I have covered – what we’ve implemented, and what our experiences have been. From where I stand now, I still believe that hybrid is the most complex option in some ways (for now at least). If you have a choice of simply running all of your SharePoint “things” in Office 365, at a stroke you will reduce complexity, cost and effort! However, this isn’t always possible and indeed, I talk to more and more organizations who are on a roadmap to eliminating on-premises SharePoint in favour of SharePoint Online – but it will take time to get there. So, hybrid definitely makes sense in many cases.

For one client in particular, we focused on solving three core problems:

User profiles

Since hybrid effectively gives each user two completely independent/disconnected user profiles, we wanted to provide one place for a user to edit their profile, with changes replicated to the other. We picked the on-premises SharePoint profile as the “editable master”, since some investment had already been made in integrating data from other on-premises data sources in this case (via BCS) and also custom AD fields. Our job was to replicate the changes to the SharePoint Online user profile (particularly custom/organization-specific fields not sync’d by Azure Active Directory sync – more on this later), so that user data was consistent wherever it was accessed

Taxonomy data

There were two drivers behind the need to sync taxonomy data from the on-premises SP2013 term store to the SharePoint Online term store – tagging of documents (as you’d perhaps expect), but also the fact that several term sets were used in custom user profile properties. Examples would be user profile properties for, say, “Business unit” or “Division” – values for these come from custom term sets to ensure consistency and valid input. In order for us to sync user profile data, this taxonomy data would need to also exist in Office 365, otherwise the profile cannot be saved because it’s pointing to a missing value. In other words, there was a dependency on taxonomy data so that we could sync user profile data.

User photos

On the surface this was simpler than the others – the client didn’t want to allow users to edit their own photos, and had some processes to deal with user photos in existing systems (including resizing, applying a border for consistency etc.). The desire was for the SharePoint Online user profile to also use this photo. If you’re going to bulk sync photos up to SharePoint Online, in reality the best way to do this is to deal with this at the Office 365 level (so that other service elements also show the photo), and the Set-UserPhoto cmdlet is provided in this space. However, we found that there was more complexity here than we expected – more later.

What we implemented – leveraging the OfficeDev Patterns and Practices code

Overall, custom code was needed to bridge these gaps. From the start, we knew that the OfficeDev Patterns and Practices libraries had code which would help, and we designed our solution around some pieces in there. With much of the heavy lifting taken care of, our focus was implementing some of their “engine code” in an enterprise kinda way – something that could run on a schedule, have the right level of resilience, and be monitorable so that any failures could be identified. Helpdesk calls of the “hey, when I go here it doesn’t have the right value for me, but when I go there it does!” and “I can’t tag this document properly” would no doubt occur otherwise – to a certain extent we definitely expect this occasionally, but we want to be able to identify/manage any issues proactively.

High-level architecture

Whilst the various bits of code supplied by the OfficeDev P and P team do their particular job (e.g. help sync taxonomy or user profile data), exactly how they are implemented and where they run from is up to you – options could include:

  • A simple console app, installed to an on-premises server
  • As above, but wrapped in a PowerShell script
  • A cloud option, e.g. WebJobs in Azure (but you might need to do extra work, e.g. with the Data Management Gateway or similar if access to on-premises data is needed)
  • Some combination of the above e.g. middle-tier code in Azure, “client” code or script on-premises

In our case, because all three aspects start with on-premises data we needed the code to talk to on-premises systems – be it the SharePoint 2013 environment, or on-premises fileshares (e.g. for photos). So, we used the “simple console app” option - this gave us the connectivity we needed, but also could be easily scheduled/monitored, and offered a familiar architecture that everyone was comfortable with. In code terms, however, we did separate out the core engine code, so that it could be hosted somewhere else (e.g. Azure) if ever required.

User profiles

The solution here focused on sync’ing custom user profile data from on-premises to SharePoint Online. Of course, Azure Active Directory sync (was DirSync) is used to synchronize some user data from on-premises systems (Active Directory) to Office 365, including the initial creation of profiles, in fact AAD Sync only deals with a core set of fields/attributes! Organizations commonly define *many* custom user profile fields (e.g. division, department, employee ID etc.) – and so a custom solution is currently needed if you want this data imported to Office 365/SharePoint Online. Our approach uses the relatively recent (November 2014) CSOM remote code methods to write the data to user profiles in SharePoint Online (not to Azure AD in this case). I implemented this as a two-step process, where a CSV file is generated of users whose profile has been updated (whether through the user, BCS fields or a sync from AD). The import then picks up this data as a 2nd step.

In fact, because CSOM is used for both reading from the source and writing the target, simply by changing values in a config file (e.g. URLs, credentials) we can actually synchronize profile data between ANY SharePoint environments we have connectivity to – either side could be an on-premises or Office 365 environment. This mean we have a pretty good solution which could be adapted if ever the needs change.

The end result is that custom profile data is synchronised to Office 365, and effectively users “look the same” in both environments - for example in people searches, or where a “created by” or “last modified by” link is clicked, and so on. In other words, the user experience just makes sense. (N.B. the images shown below are of the new profile experience in Office 365 [April 2015]):



Specific fields can be set to "do not import" (e.g. to not conflict with fields updated by AAD Sync), or for any other reason: clip_image004
User accounts can also be "mapped" between the source/target environments - this is useful where * accounts are used/AAD Sync and identity integration isn't implemented (e.g. test environments):


Any users who fail to import for some reason will automatically be re-tried (even if their profile hasn't changed since the last attempt). Information on which users are failing, how many times they've failed, and the last failure reason is maintained as the import jobs run:


Since they run on an ongoing basis, the tools are designed to be monitorable. They write a summary to the Windows Event Log, ready for SCOM or another monitoring tool to pick up and send e-mail notifications of any failures:


Further to this, Log4Net is implemented to provide rich logging to help troubleshoot any problems:


Taxonomy data (e.g. term sets for tagging)

Taxonomy sync can synchronize most types of changes (new term sets, new terms, renames etc.) from source to target. The heavy lifting is actually done by Microsoft code (the OfficeDev Patterns & Practices Core.MMSSync solution).

The end result is that any term set created in the source environment (on-premises SP2013 in this case) is replicated to the target environment (Office 365):

This allows documents to be tagged in a consistent way across online/on-premises sites, and also user profile fields which are bound to term sets to work properly in both environments.


Synchronization of user photos

Our mechanism here was mainly a PowerShell script which used the Set-UserPhoto PowerShell cmdlet (which is actually an Exchange Online cmdlet – and requires the user to have an Exchange Online mailbox in Office 365), but integrates with some existing photo-handling processes. However, it was decided to change this approach entirely in the end, due to some issues we hit – more on this in the next section.

Challenges we encountered

For the most part, the development of these tools went fine – however, we definitely did encounter a couple of road-bumps along the way. These were things like:

  • Typical client infrastructure issues – tools would run fine in our environments, but would be less reliable communicating with the cloud when running on more locked down servers and running through additional network infrastructure (e.g. reverse proxies).
    • In the end, this proved a significant issue for this client (but perhaps somewhat specific to their case). The user photo PowerShell script in particular would fall over with authentication problems when the network connection was less than perfect – and it was hard to “engineer in” absolute reliability here. In the end, it was decided a simpler approach would be to change policy to allow users to edit their own photo (in SharePoint Online) :)
  • Some taxonomy sync challenges
    • Although the P and P code is pretty resilient here, we did hit the occasional error (most likely related to connectivity/the previous point). For the most part, these would get resolved on the next run however.
  • Throttling and batch issues with remote code
    • During the development process, clearer guidance emerged on how to handle throttling in Office 365 code. This was useful, but we also found that some bits of the OfficeDev Patterns and Practices internal code caused problems in the way we were using it (even though the later versions implemented the throttling/retry pattern). There was nothing invalid about the PnP code *or* the calls we were making – it was more that some tweaking was required to be able to successfully process everything we needed to.


Ultimately we were able to implement our sync tools so that the end-user’s experience of hybrid SharePoint is improved. I think that any form of cloud development will always encounter occasional errors in communication, and at enterprise scale it’s usually a good idea to build in some kind of tolerance to this (e.g. our re-try mechanism for failed user profile updates). The OfficeDev Patterns and Practices code helped us out a ton – there’s no doubt that closing these gaps would have required WAY more effort without it.

As mentioned earlier, I’d expect SharePoint 2016 to improve on the hybrid situation natively – but this work wasn’t a huge effort to make big steps towards this right now.

Monday 6 April 2015

Speaking at Ignite and SharePoint Evolutions 2015 Conferences

As you might know if you’re in the SharePoint/Office 365 space, there’s a couple of big conferences around the corner – I’ll be delivering sessions at Microsoft’s Ignite conference in Chicago in May, and also the most excellent SharePoint Evolutions conference before then in April, here in London. It always takes quite a lot of preparation to pull new talks together, but after the conference I’ll be looking forward to writing in more detail about some of the topics I’ve been exploring. Before then, I think both conferences are going to be pretty special, and if you’re going to one of them (or Build, in San Francisco in April) I think you’ll have a great time and learn a lot. Here are the details of my sessions (N.B. there were some updates to these summaries which hadn’t made it to the websites at the time of writing):


MS_Ignite_400pxDealing with Application Lifecycle Management in Office 365 app development 

Thursday, May 7th, 10:45AM - 12:00AM

For teams who are undertaking cloud-friendly SharePoint or Office 365 development, apps are likely to be a key area of focus - be they Apps for SharePoint or newer Office 365/Azure AD apps. One of the benefits of stepping outside the traditional SharePoint development model is that aspects such as ALM and Continuous Integration become easier - finally, an end to "development is harder because it's SharePoint!". In this session we'll talk about how to do app development the "right" way - with a focus on ASP.NET as the framework and Azure as the hosting platform. We'll cover good practices such as implementing automated builds, take a close look at “continuous deployment” for apps, and also discuss the upgrade/push-to-live process for existing applications. Over several demos, we'll show how Visual Studio Online and Azure are a winning combination, and see how concepts such as "Deployment Slots" in Azure can help.

Transforming Your SharePoint Full Trust Code to the Office App Model (panel discussion)

Tuesday, May 5th - 5:00PM - 6:15PM
Panel – Vesa Juvonen, Frank Marasco, Bob German, Erwin van Hunen, Chris O’Brien

This session is a panel discussion covering examples and patterns for moving SharePoint customizations to the cloud app model - for use either in Office 365 or "cloud-friendly" implementations of on-premises SharePoint. The panel comprises members of the Microsoft OfficeDev Patterns and Practices team and independent MVPs. Both bring their experiences from the field, but different perspectives too. Since the landscape changes when developers must leave behind server-side SharePoint API code and feature framework elements, the discussion will centre around 5 related hot topics - branding, options around remote code (including .NET, JavaScript and PowerShell), provisioning of customized sites, the story with sandbox solutions and finally how the Office 365 APIs fit in with the app model. We promise a lively discussion, examples of code from the field, and time for Q&A!


SharePoint Evolutions


Dealing with Application Lifecycle Management in Office 365 app development

Same as Ignite session above

New developer choices - comparing Apps for SharePoint with external Office 365 apps

No sooner have we all learnt how to develop Apps for SharePoint, when Microsoft go and release a new development model. Don’t you just love it when they do that?! In an "external" or "standalone" app, there's no need for the app to be installed to a SharePoint site - and this can provide huge benefits. However, new APIs must be used and the developer must understand how Azure AD fits in. Whilst this model is currently Office 365 only, Microsoft have hinted that this approach may come to on-premises SharePoint soon - so this will become important to all developers. This session is designed to get you up to speed.

Closing notes

I also plan to publish videos of the demos from these sessions. There have been quite a few changes in Azure recently, and my ALM session covers some of the cool things such as Deployment Slots, with a hat tip to other related capabilities such as the “Traffic Routing” feature which can be used for A/B testing and the like.

I’m really looking forward to the panel discussion at Ignite too. Like the other panel members I’ve definitely got some opinions on the topics being discussed :)

Anyway, hope to see you at one of the conferences. Please come and say hello if you’re there!