Thursday, 16 April 2015

Improving the Office 365 SharePoint hybrid experience

Back in February 2014, I wrote Office 365 SharePoint hybrid – what you DO and DO NOT get. In that article, I discuss the fact that whilst there’s lots of discussion about hybrid and indeed lots of organizations doing *something* in this space – in actual fact, what Microsoft provide is far from a turn-key solution and there’s quite a long list of things you need to work on. That is, if your organization wants to provide any kind of “joined-up” experience that works well for end-users, across sites in the cloud and sites in an on-premises SharePoint environment.

It’s fair to say that at the current time, hybrid from Microsoft really just consists of:

  • Search
  • BCS
  • A model for personal sites (OneDrive for Business sites) to live in the cloud, with some navigation for users, when other things are on-premises

The list of things you don’t get includes things like (N.B. the earlier article has a more complete list):

  • Global navigation
  • A global site directory
  • A joined-up social experience (unless you use Yammer)
  • Any kind of synchronisation of:
    • Branding
    • Customizations
    • Taxonomy/Managed Metadata
    • Content types
    • Apps
    • User profile data

..and so on.

I fully expect SharePoint 2016 to improve on this situation – I think it will be a key focus area for the product. But here in April 2015, having SP2016 in production is still some considerable time away for most.

Common requests for improving hybrid – user profiles, taxonomy and user photos

Since then I’ve worked on several hybrid deployments, and have been involved in work to close the hybrid gap and improve the experience. In this post I want to talk about some of the ground my team and I have covered – what we’ve implemented, and what our experiences have been. From where I stand now, I still believe that hybrid is the most complex option in some ways (for now at least). If you have a choice of simply running all of your SharePoint “things” in Office 365, at a stroke you will reduce complexity, cost and effort! However, this isn’t always possible and indeed, I talk to more and more organizations who are on a roadmap to eliminating on-premises SharePoint in favour of SharePoint Online – but it will take time to get there. So, hybrid definitely makes sense in many cases.

For one client in particular, we focused on solving three core problems:

User profiles

Since hybrid effectively gives each user two completely independent/disconnected user profiles, we wanted to provide one place for a user to edit their profile, with changes replicated to the other. We picked the on-premises SharePoint profile as the “editable master”, since some investment had already been made in integrating data from other on-premises data sources in this case (via BCS) and also custom AD fields. Our job was to replicate the changes to the SharePoint Online user profile (particularly custom/organization-specific fields not sync’d by Azure Active Directory sync – more on this later), so that user data was consistent wherever it was accessed

Taxonomy data

There were two drivers behind the need to sync taxonomy data from the on-premises SP2013 term store to the SharePoint Online term store – tagging of documents (as you’d perhaps expect), but also the fact that several term sets were used in custom user profile properties. Examples would be user profile properties for, say, “Business unit” or “Division” – values for these come from custom term sets to ensure consistency and valid input. In order for us to sync user profile data, this taxonomy data would need to also exist in Office 365, otherwise the profile cannot be saved because it’s pointing to a missing value. In other words, there was a dependency on taxonomy data so that we could sync user profile data.

User photos

On the surface this was simpler than the others – the client didn’t want to allow users to edit their own photos, and had some processes to deal with user photos in existing systems (including resizing, applying a border for consistency etc.). The desire was for the SharePoint Online user profile to also use this photo. If you’re going to bulk sync photos up to SharePoint Online, in reality the best way to do this is to deal with this at the Office 365 level (so that other service elements also show the photo), and the Set-UserPhoto cmdlet is provided in this space. However, we found that there was more complexity here than we expected – more later.

What we implemented – leveraging the OfficeDev Patterns and Practices code

Overall, custom code was needed to bridge these gaps. From the start, we knew that the OfficeDev Patterns and Practices libraries had code which would help, and we designed our solution around some pieces in there. With much of the heavy lifting taken care of, our focus was implementing some of their “engine code” in an enterprise kinda way – something that could run on a schedule, have the right level of resilience, and be monitorable so that any failures could be identified. Helpdesk calls of the “hey, when I go here it doesn’t have the right value for me, but when I go there it does!” and “I can’t tag this document properly” would no doubt occur otherwise – to a certain extent we definitely expect this occasionally, but we want to be able to identify/manage any issues proactively.

High-level architecture

Whilst the various bits of code supplied by the OfficeDev P and P team do their particular job (e.g. help sync taxonomy or user profile data), exactly how they are implemented and where they run from is up to you – options could include:

  • A simple console app, installed to an on-premises server
  • As above, but wrapped in a PowerShell script
  • A cloud option, e.g. WebJobs in Azure (but you might need to do extra work, e.g. with the Data Management Gateway or similar if access to on-premises data is needed)
  • Some combination of the above e.g. middle-tier code in Azure, “client” code or script on-premises

In our case, because all three aspects start with on-premises data we needed the code to talk to on-premises systems – be it the SharePoint 2013 environment, or on-premises fileshares (e.g. for photos). So, we used the “simple console app” option - this gave us the connectivity we needed, but also could be easily scheduled/monitored, and offered a familiar architecture that everyone was comfortable with. In code terms, however, we did separate out the core engine code, so that it could be hosted somewhere else (e.g. Azure) if ever required.

User profiles

The solution here focused on sync’ing custom user profile data from on-premises to SharePoint Online. Of course, Azure Active Directory sync (was DirSync) is used to synchronize some user data from on-premises systems (Active Directory) to Office 365, including the initial creation of profiles, in fact AAD Sync only deals with a core set of fields/attributes! Organizations commonly define *many* custom user profile fields (e.g. division, department, employee ID etc.) – and so a custom solution is currently needed if you want this data imported to Office 365/SharePoint Online. Our approach uses the relatively recent (November 2014) CSOM remote code methods to write the data to user profiles in SharePoint Online (not to Azure AD in this case). I implemented this as a two-step process, where a CSV file is generated of users whose profile has been updated (whether through the user, BCS fields or a sync from AD). The import then picks up this data as a 2nd step.

In fact, because CSOM is used for both reading from the source and writing the target, simply by changing values in a config file (e.g. URLs, credentials) we can actually synchronize profile data between ANY SharePoint environments we have connectivity to – either side could be an on-premises or Office 365 environment. This mean we have a pretty good solution which could be adapted if ever the needs change.

The end result is that custom profile data is synchronised to Office 365, and effectively users “look the same” in both environments - for example in people searches, or where a “created by” or “last modified by” link is clicked, and so on. In other words, the user experience just makes sense. (N.B. the images shown below are of the new profile experience in Office 365 [April 2015]):



Specific fields can be set to "do not import" (e.g. to not conflict with fields updated by AAD Sync), or for any other reason: clip_image004
User accounts can also be "mapped" between the source/target environments - this is useful where * accounts are used/AAD Sync and identity integration isn't implemented (e.g. test environments):


Any users who fail to import for some reason will automatically be re-tried (even if their profile hasn't changed since the last attempt). Information on which users are failing, how many times they've failed, and the last failure reason is maintained as the import jobs run:


Since they run on an ongoing basis, the tools are designed to be monitorable. They write a summary to the Windows Event Log, ready for SCOM or another monitoring tool to pick up and send e-mail notifications of any failures:


Further to this, Log4Net is implemented to provide rich logging to help troubleshoot any problems:


Taxonomy data (e.g. term sets for tagging)

Taxonomy sync can synchronize most types of changes (new term sets, new terms, renames etc.) from source to target. The heavy lifting is actually done by Microsoft code (the OfficeDev Patterns & Practices Core.MMSSync solution).

The end result is that any term set created in the source environment (on-premises SP2013 in this case) is replicated to the target environment (Office 365):

This allows documents to be tagged in a consistent way across online/on-premises sites, and also user profile fields which are bound to term sets to work properly in both environments.


Synchronization of user photos

Our mechanism here was mainly a PowerShell script which used the Set-UserPhoto PowerShell cmdlet (which is actually an Exchange Online cmdlet – and requires the user to have an Exchange Online mailbox in Office 365), but integrates with some existing photo-handling processes. However, it was decided to change this approach entirely in the end, due to some issues we hit – more on this in the next section.

Challenges we encountered

For the most part, the development of these tools went fine – however, we definitely did encounter a couple of road-bumps along the way. These were things like:

  • Typical client infrastructure issues – tools would run fine in our environments, but would be less reliable communicating with the cloud when running on more locked down servers and running through additional network infrastructure (e.g. reverse proxies).
    • In the end, this proved a significant issue for this client (but perhaps somewhat specific to their case). The user photo PowerShell script in particular would fall over with authentication problems when the network connection was less than perfect – and it was hard to “engineer in” absolute reliability here. In the end, it was decided a simpler approach would be to change policy to allow users to edit their own photo (in SharePoint Online) :)
  • Some taxonomy sync challenges
    • Although the P and P code is pretty resilient here, we did hit the occasional error (most likely related to connectivity/the previous point). For the most part, these would get resolved on the next run however.
  • Throttling and batch issues with remote code
    • During the development process, clearer guidance emerged on how to handle throttling in Office 365 code. This was useful, but we also found that some bits of the OfficeDev Patterns and Practices internal code caused problems in the way we were using it (even though the later versions implemented the throttling/retry pattern). There was nothing invalid about the PnP code *or* the calls we were making – it was more that some tweaking was required to be able to successfully process everything we needed to.


Ultimately we were able to implement our sync tools so that the end-user’s experience of hybrid SharePoint is improved. I think that any form of cloud development will always encounter occasional errors in communication, and at enterprise scale it’s usually a good idea to build in some kind of tolerance to this (e.g. our re-try mechanism for failed user profile updates). The OfficeDev Patterns and Practices code helped us out a ton – there’s no doubt that closing these gaps would have required WAY more effort without it.

As mentioned earlier, I’d expect SharePoint 2016 to improve on the hybrid situation natively – but this work wasn’t a huge effort to make big steps towards this right now.


finarne said...

Wondering what your thoughts are around the "move" away from the SharePoint features in O365 to more disparate services.

Looking at the Delve announcements it's as if we can say MySites are almost gone now, and so the notion of tagging is perhaps going to shift to adding to a Delve board (note I don't know where the Delve note "tags" get stored, think they're still referred to as tags in the api).

I sat in on the Yammer training at the SharePoint conference last year and when the SharePoint folk in the room brought up questions on how would Yammer tags and SharePoint managed metadata tags going to be aligned it was a "we aren't really thinking about that" answer.

With Delve boards and the new people page I can see that the notion of tagging could be elevated out of Managed Metadata entirely for O365 at least.

SharePoint user profile data is of course imported into Managed Metadata (done this loads of times myself and used the data for site navigation and search personalisation) but do you see a need to get the Managed Metadata and Delve board tags aligned?

Maybe we'll see a way to configure user profile properties as a delve tag type?

Kirk said...

Nice post, Chris. We worked through some of these problems in the past, but it was before PnP was available, I believe, and I'm sure we had different requirements as well so we took some different paths...

One approach for profiles is that we decided to have the O365 profile as the one source of truth. We copied profile properties from on-prem to online (pre PnP, unfortunately) and then created on-prem redirects to take users to the online profile from on-prem. Since we only had one source of the truth and users never accessed their on-prem profiles there was no need for a sync, but our code that did a one-time copy was able to work either way similar to your approach.

When it came to user profile images we found a lot of nuances that were undesirable. These may be better today, but I suspect they are not. I blogged about it here:

For search we used the hybrid search approach, but since on-prem was not accessible through the firewall online was unable to show on-prem search results. Therefore, we made an on-prem search center the primary search center. For online search results we would attempt to redirect the user to the on-prem search center if they were already logged into on-prem. The on-prem search results would show hybrid results from both on-prem and online.

- Kirk

Kirk said...

Oh, I should have also mentioned that we created a user profile image proxy to run on-prem so that if the user was not authenticated into SharePoint online, he/she could still see the online user photo. This is in addition to the redirection link to the profile mentioned in my last comment and is to support having the O365 profile as the "one source of truth".

- Kirk

Nico said...

Thanks for sharing Chris. Wished I knew this half a year ago when I designed a large hybrid cloud solution with SharePoint 2013 and Office365. Could have saved me a lot of time.

Just want to share with you a few additional experiences with the taxonomy sync challenge. The P&P code did not worked for us, as it lacked support for some important Term Store properties. However, we stumbled across a very affordable product called ServiceAware Term Store Sync that did exactly what we needed to address the challenge of keeping terms in sync between on-premises SharePoint 2013 and Office365.