Tuesday, 15 April 2014

Speaking at the SP24 online conference this week

Later this week, on April 16 and April 17, the SP24 online conference takes place and runs for, you guessed it, 24 hours. The conference is FREE. I highly recommend registering and tuning in to some sessions if you can – after all, isn’t that what that second monitor is for? :)


I’m proud to be speaking – the event looks great, and the amount of preparation by the organisers is pretty astounding. In addition to simply being streamed, each session gets a “session room” where questions can be put to the speaker, a Twitter feed runs and any supporting documents supplied by the speaker can be downloaded. For example, here is my session room:

COB SP24 session room

Not just any old online conference then – as a speaker I’m very impressed.

There’s a LONG of great speakers (Bill Baer, Jeremy Thake, Joel Oleson, Mirjam van Olst, Laura Rogers, Todd Baginski, Marc Anderson, Rene Modery, Chris Givens, Sam Hassani, Paolo Pialorsi, Dux Raymond Sy, Liam Cleary, Agnes Molnar and *many* more) – all pulled together into something like a television channel by a handful of anchors (presenters). There is a mix of live sessions (beamed to the world using Google Hangouts) and pre-recorded ones, split across two tracks:

  • Technical
  • Business

My session

My session is:

Office 365 - developer lessons learnt, tips and tricks

As a developer or technical lead, your early Office 365 projects are likely to raise some big questions. How should you interpret Microsoft's mixed messages on sandbox solutions? How should test environments be handled in the cloud? Do you need to run more than one Office 365 tenancy? When you can't deploy files to the LAYOUTS directory, where should global assets such as CSS and JavaScript be stored? What about automation, when only 30 PowerShell cmdlets are available in SharePoint Online? And just how do you provision Managed Metadata fields, when the on-premises techniques cannot be used? This session walks through the “dev strategy” decisions we made at Content and Code, and why. Over the course of several demos, we’ll also discuss apps, automation scripts and also advanced techniques such as Continuous Integration for Office 365. Hopefully even people *not* expecting to work on an Office 365 development project can come away with flavour of what such a project might look like.

Time: April 17 11am BST

Session URL: https://www.sp24conf.com/2014-1/Conf/SP24S108/ConfPages/SessionRoom.aspx

To register for SP24..

..go to:



Sunday, 6 April 2014

Using the Azure instance behind your Office 365 tenant

Something that perhaps isn’t common knowledge in the SharePoint world, is that every Office 365 tenant has an Azure instance behind it – and that if you’re doing some types of Azure stuff, sometimes you *must* use this instance. I’ve spent the last few months doing Office 365 development with some use of Azure, and this only became clear to me fairly recently. When I’ve mentioned it to other people, most have been completely unaware, and I’m not surprised when you consider that:

  • There is no link from Office 365 to the Azure instance (e.g. in the Office 365 tenant admin screens)
  • If you try to log-in to Azure as an Office 365 user, you initially get presented with a screen that makes you think you’ve done something wrong (as at spring 2014 time anyway)

Scenarios where you NEED to use this Azure instance

For many Azure usages (e.g. “externalizing” assets such as CSS/JS files like my team do, Azure Service Bus, Azure CDN etc.) it doesn’t matter which Azure instance you use. The most natural thing is to go the Azure Portal (https://manage.windowsazure.com) and sign-up if you don’t have a subscription already. Of course, this creates you a brand new Azure instance – and that works great for these types of scenarios. However, you’d need to use the “integrated” Azure instance for anything which needs to work with the directory of Azure AD/Office 365 users for your tenant. This could be things like:

  • Implementing Azure AD authentication on an external application (e.g. a provider-hosted app, or other website), where you are aiming for some kind of Single Sign-On
  • Working with some aspects of Office 365 users (e.g. the Azure AD profile which is “behind” the SharePoint Online user profile) via the Azure Graph API
  • Integrating a 3rd party application which is able to use Azure AD authentication/have SSO (e.g. salesforce.com, Dynamics CRM, Dropbox for Business etc.)
  • ..and many other scenarios involving identity

Accessing the Azure instance behind your Office 365 tenant

So let’s say one day you think, “I wonder how I’d get to this magical Azure instance?” Well, if you go to the Azure Portal, you’ll either see a sign-in screen asking you to enter a username to continue, or you might be auto-logged in if you have previously signed-in to the Microsoft gateway – for example, you’re already logged-in to Office 365. If you did that (even as an Office 365 tenant administrator) you’ll see this:


I don’t know about you, but my immediate reaction to that is “Whoah, obviously I shouldn’t be here logged-in as this user!” However, now is the time be brave and not flinch! Sometimes in life, it takes a valiant act of courage to push through the terrifying situations that can befall a modern-day I.T. warrior! The thing to do now is actually to click:

“Sign up for Windows Azure”

Yes, it’s counter-intuitive. As I mentioned earlier, the Azure instance already exists so why do we need to “sign-up”? Well, you just do - for now at least. I did hear a Microsoft speaker at SPC mention this experience as a side-note, and that they will probably improve things soon. The main thing to understand is that you have to supply payment details before this Azure instance can be accessed – because then it becomes a true pay-as-you-go instance.

Viewing the Azure AD directory behind your Office 365 tenant

Once you’ve gone though the sign-up process, you should be able to go into the Azure Portal (https://manage.windowsazure.com, or the new preview version https://portal.azure.com). Here, out of the all the things you can use in Azure, you’ll probably have just one thing – a Directory:

Azure items

..and if you go into the Directory, you should see all your Office 365 users (including those synchronized by DirSync, if configured):

Azure directory users - small

Apps which integrate with Azure AD

As I mentioned earlier, you mind yourself creating some kind of app which needs to work with Azure AD (e.g. for authentication/SSO, or some usage of the Azure AD Graph API for user records perhaps). The first step in many of  these cases is to register the app in your Azure AD directory – this is done in the “Applications” area of the directory:

Azure create directory application

Other scenarios around Azure subscriptions and identities

It turns out there are quite a few permutations of starting an Azure subscription, or joining up to an existing one. One reason for this is that in the new world of Microsoft identities, it’s possible to use a Microsoft account (formerly Live ID) OR an “Organizational Account”. If you’re an Office 365 user, your Azure AD record is an Organizational Account. Personally I’m still getting my head around some of the nuances here, but Microsoft recently released this video which talks through various Azure sign-up scenarios – if you want further information, I think it’s helpful:

Using a existing Windows Azure AD Tenant with Windows Azure


Wednesday, 19 March 2014

An implementer’s summary of the SharePoint Conference 2014

Quite a few SharePoint community folks have already published their round-up posts on the recent SharePoint Conference 2014 held in Las Vegas. I’ll SPC keynote smaller point to some good ones at the end of this article, but regardless of other posts I wanted to go ahead and publish mine because:

  • I notice I have some different items compared to other people – obviously everyone has their view of what is important to them :)
  • I think some colleagues and others in circles near me have similar views and focus areas, and might find my interpretation useful

I break my thoughts down into two sections – The Big Things, and a Dev-Focused Summary. If you feel you already understand The Big Things from SPC2014, you might want to skip ahead to my dev-focused summary:

The Big Things

  • There will be at least one more on-premises version of SharePoint, but it’s unclear how many more after that (I sense Microsoft may take a “wait and see” approach). The next version will be “SharePoint 2015” (or at least it will be released in 2015).
  • Some big changes to Office 365 which will open doors for many clients/scenarios:
  • Go Yammer!
    • The social capabilities in SharePoint 2013 will NOT be developed further
    • However in Yammer, there’s a lot of integration in the pipeline – this will mainly be on the Office 365 side until the SP2015 release though. Includes things like:
      • Groups - create a group, and automatically get a site/inbox/OneDrive library for docs etc.
      • Conversations around documents (i.e. comments displayed next to document) – in Office Online (new name for Office Web Apps), rather than Office clients on your desktop however
      • Integrated profile/identity (no longer separate Yammer and SharePoint profiles) – this will unlock other scenarios going forward
  • Forms - InfoPath is deprecated (although supported until 2023)
    • Microsoft will move away from a “one size fits all” approach to forms, and several options (some available now) will be recommended – Excel Surveys, list forms (to be enhanced), a Word-based option for “structured documents”, and “App Forms” (Access-based)
    • Microsoft are listening at http://officeforms.uservoice.com
    • Nik Patel has a great write-up here
  • The Office Graph, and Oslo app
    • This is a big new concept, about users finding content that matters to them. The Office Graph is the framework/API, and Oslo is the codename for a new app which presents this stuff. The Office Graph  “uses signals from email, social conversations, documents, sites, instant messages, meetings, and more to map the relationships between the people and things that make your business go”
    • There are lots of views such as “Shared with me”, “Modified by me”, “Presented to me” etc. – things that really make it easier for me to find “that file I was looking at last week”. The sentence that brought it home for me was “The Office Graph knows which meetings you’re in, when someone is presenting, and where the presentation is stored. Oslo just connects the dots to show what’s been presented to you.”
    • See Introducing codename Oslo and the Office Graph for more details
  • SharePoint 2013 Service Pack 1 smoothes some hybrid scenarios:
    • This goes some way to helping out with a couple of the issues I listed in Office 365/SharePoint hybrid – what you DO and DO NOT get, mainly by providing an OOTB way to swap out some links in the top navigation bar (aka “the suite bar”) displayed in your on-premises sites. Specifically: clip_image001
      • Replace "Newsfeed" with "Yammer" (if you have selected Yammer as the social option)
      • Replace link to “Sites” page – here, an admin can provide the URL to the “Sites” page in Office 365. This facilitates having ONE Sites page (in the cloud) across SharePoint Online and SharePoint on-premises – this has two benefits as far as I can see:
        • Users have one place to go to for their saved navigation links (sites/documents), rather than a very confusing situation of having them split across two Sites pages. (BUT, I’m fairly sure that “following a document” in an on-premises site does NOT magically save it to the O365 Sites page – so you may still prefer a custom approach)
        • The “Start a site..” link only exists in the cloud, so new site collections get created there
      • Replace "Skydrive Pro/OneDrive For Business" link with link to OneDrive in cloud

My dev-focused summary

  • Full-trust code (farm solutions) will continue in “SharePoint 2015”
  • New Office 365 APIs NOTE: Microsoft are starting to move away from SharePoint/Exchange/whatever-specific APIs, and more towards general “Office 365 APIs”
    • People/Contacts
    • Files (OneDrive API)
    • Mail
    • Calendar
    • [Apps which need to call into the above endpoints get registered with Windows Azure Active Directory, and use the Common Consent Framework to access Office 365 data]
  • That said, there are also new SharePoint client APIs (also usable in on-premises SharePoint with SP1/updated Client Redistributable)
    • CSOM API for creating site collections (finally!)
    • CSOM API for creating a content type with a specified ID (finally!)
    • Also being able to set things like AlternateCSSUrl, SiteLogoUrl for a site
    • Also a new SDK for creating Office 365 apps on Android. Apparently this is big news (lots of hoohaa at the conference) – no idea why, but it IS refreshing that Microsoft now talk about executives with iPads and the popularity of Android..
  • There are new JS/HTML-based controls for use in provider-hosted apps
    • The first controls available are ListView and People Picker (e.g. show a SharePoint list, with sorting/filtering etc. in your plain ASP.NET page in your app)
    • These are badged as Office.Widgets.Experimental for now
  • Auto-hosted apps are (likely to be) deprecated baby! (Microsoft have some challenges around making this production-scale ready, in terms of scalability/backup/DR etc. The long-term prognosis is being reviewed, but the recommendation is to use auto-hosted for prototyping only)    UPDATE: looks like this isn't the case (although several MSFT speakers mentioned it at SPC14) - Brian Jones (MSFT PM) chimed in on Twitter, and he's the guy who would know
    • [For me, this is a good validation of what I said in July 2013 about auto-hosted apps and why they should be avoided (in the “What about building my app/RER as an auto-hosted app?” section of this article)]  UPDATE - I guess we reserve judgement on this one for now!
  • There are GREAT new dev samples on many real-life cloud development scenarios
    • I highly recommend checking these out – see Office App Model Samples
    • Scenarios include:
      • Branding personal sites
      • Provisioning Yammer groups as part of site collection provisioning
      • Setting a theme on the host web
      • And LOTS of others
  • Some misc stuff:
    • JSON light support in Office 365 APIs
    • New apps in Excel
      • People Graph
      • Bing maps
    • New mail app in Office 365/mail
      • DocuSign!
    • New “file extension handler” - can register an app for opening/previewing files by document extension
      • e.g. CAD viewer in preview pane for CAD files
      • I think this is similar to a CustomAction – i.e. there is new Feature XML for this
  • Other news I found relevant:
    • No Microsoft solution is planned around sync’ing custom data to SharePoint Online user profiles right now
      • Microsoft do not have anything in the pipeline to facilitate sync’ing custom directory data to the user profile in SharePoint Online (a commonly-requested thing). I asked at the post-conference session on hybrid, led by Bill Baer and Steve Peschka. So, the workarounds using the old user profile ASMX web service continues to be needed if you have to do something here.
    • Microsoft plan to improve the “in SharePoint hybrid, search results are displayed in separate blocks (for on-premises and Office 365 sites)” experience (shown in my post here)
      • This was mentioned in Bill Baer’s IT Pro keynote – some sort of “remote index” solution is planned

Summary and more info

I am sure there’s lots of other stuff too, but those were probably the most relevant things for me. Some other recommended posts are:

Wednesday, 12 March 2014

First Release and NDA Preview for Office 365 – early access to Microsoft’s changes for testing

One of the most interesting sessions I attended at SPC 2014 was around change management in Office 365. For those of us who have worked on Office 365 (i.e. SharePoint Online) or hybrid projects, it’s been a challenging time recently – Microsoft have made various updates and patches to Office 365, which have subsequently caused lots of customizations to break and ultimately made it hard to support organizations on Office 365. And I think it’s normal to have *some* level of customization in an Office 365 project – for some it might just be branding, but since the SharePoint user experience is still not ideal in places, our clients are often requesting enhancements around search, user profiles, sites, OneDrive for Business (the new name for My Sites) and so on. Some examples of unannounced Microsoft changes which have caused valid customizations to break are:

Of course, a big part of this is that Microsoft have taken a leaf from the Facebook/Yammer book, and have shifted their “release cadence” – updates are deployed to Office 365 in small batches very frequently(every month), which is very different to the on-premises world with a big release every 3 years and service packs and cumulative updates within that.

Taking a step back – lack of Office 365 test environments

Clearly, Microsoft are always going to need to update Office 365/SharePoint Online. And in many cases, all we need is a way of testing the change (and any impact on our customizations) before it’s in production. I recently had a meeting with some Microsoft UK folks about this, and I summarized this aspect as:

  • Lack of test environments in Office 365
    • Whilst we can (and do) create and pay for multiple O365 tenancies/environments to simulate non-production environments, we have no control of the *sequence* that tenants are patched in – in other words, a Microsoft update can hit our “production” tenant before our dev/test tenants
  • No notice/awareness of the date that an update will occur on (for our tenants)

Announced at SPC 2014 - “First Release” and “NDA Preview” options

The good news is that Microsoft are taking big steps to improve this situation – it’s a topic close to my heart, so I wanted to provide my interpretation here (as well as publicize the initiative), but will also link to official TechNet pages when they arrive. In a slightly “under the radar” session on the schedule, a few changes were announced by Jake Zborowski (Group Product Manager, Office 365) at the conference:

  1. Improved communication about updates
  2. A more structured approach to releasing updates
  3. “First Release” for Office 365
  4. “Office 365 NDA Preview”

The next sections go into some more detail on each one.

Improved communication about updates

This includes:

  • A published *public* roadmap – used to communicate some types of upcoming updates
    • This is targeted for April 2014
  • Use of the Office 365 updates blog to communicate changes
  • Use of the Office 365 System Requirements page to communicate changes
  • Changes to the Message Center
    • Ability to display messages/warnings specific to the tenant (currently cannot)
    • Will show near-term warnings, recommended actions etc.
    • New API, so that developers can write notification apps
    • Enhancements to the existing Office 365 Admin mobile app

A more structured approach to releasing updates

Changes will be classified into different types, and each given a different communication and release plan. The slide below summarizes the communication aspect - but note that at a lower level, communications will actually appear at different times in the different communication vehicles (e.g. an upcoming change will be announced on the public roadmap before it shows up in Message Center etc.):


“First Release” for Office 365 

This is the ability to switch mode for a certain Office 365 tenant – effectively you are signing-up to be “first in the queue” for many types of changes/updates, before they get made generally available in Office 365. The scope of this is for changes to SharePoint Online and Exchange Online. This is AWESOME news, and should go a long way to solving some of the issues for us. Some more details:

  • Opt-in switch (off by default):
    • The switch will appear in the Office 365 admin center around early April 2014, and then changes will start to be rolled through it around 4 weeks after that
    • Changes to the switch will be respected within 24 hours – you can switch First Release on/off for your tenancy as you like
    • If you opt-out, any changes already deployed will not be pulled back but future updates will no longer be applied early to your tenant
  • Types of updates:
    • Updates which do get pushed through will be supported/vetted/documented and end-user facing
    • Not all updates will be pushed through First Release – I wasn’t 100% clear on what is/isn’t, but I believe something like storage increases (e.g. for OneDrive, or general site collections) would not be
  • Timing:
    • You’ll get a warning of at least 7 days alerting you that a change is going be rolled out to First Release
    • An update will be in First Release for a minimum of 2 weeks before standard rollout – however, for most tenants it could work out to be around 1 month until the update lands
    • N.B. at the time of writing, the Product Group are currently working on the ability to know for sure when an update will hit a specific tenant (and consider that for some updates this could actually be different for individual site collections and/or users!). Obviously this is crucial to implementers (like us), because sometimes we may need to deploy a code change to "match up" with a change in Office 365 (otherwise there will be a period where something is not working properly

Here’s a mock-up of how the First Release switch might appear (apologies for the low-quality image):

First Release switch - mockup

A note on how I expect WE will use First Release
As I discuss in my Office 365 developer decisions, tips and tricks presentation, we long ago decided to use multiple tenants for each client we work with (representing dev, test and production at least). With the introduction of First Release, I expect we will enable First Release on dev and/or test but leave it off for our “production” tenants. I have some thoughts on whether *both* dev/test should be enabled

“NDA Preview” for Office 365

NDA preview is another useful option for early access to changes – here, Microsoft’s changes are rolled out to a subset of users in your tenant. So whereas First Release affects an entire tenancy, NDA Preview affects only nominated users. There is more involved than just flicking a switch here – I don’t yet know for sure whether it will be open to everyone, but it is a TAP-like program where the organization needs to be able to dedicate time to providing good information to MSFT engineering and working together on issues. Just like a TAP, contracts have to be signed etc. 

Some more details:

  • Up to 100 users (TBC) can be nominated
  • All changes going through  First Release will also go through NDA Preview
    • BUT, some new changes will go through NDA Preview only, and NOT First Release – an example could be the storage increase change I mentioned earlier
  • Sign up at http://prereleaseprograms-public.sharepoint.com
    • Microsoft will then be in touch at a later date – as I say, contracts need to be signed (possibly because of the different nature of some changes pushed through NDA Preview)
A note on how I expect WE will use NDA Preview:
Even with First Release, I think the NDA Preview option will be very valuable. and I expect we WILL enable it for each of our production tenants and nominate some users from at least the implementation team. The production tenant is sometimes the only environment which truly “has everything” – directory synchronization, federated authentication, Yammer SSO/integration to the Yammer Enterprise network, two-way hybrid config (for hybrid deployments) are all examples of things which may not be in place elsewhere if you run multiple tenants. As such, the option of testing Microsoft’s updates in production before full rollout is a very good thing.

Timelines and NDA Preview/First Release

The following diagram gives a sense of how the two programs will work in terms of the timeline for updates:

Office 365 updates process - small


I’m really happy to see Microsoft start addressing the issue of Office 365 change management. My fear had been that the platform had not been engineered to release updates to tenants in different ways (and you get a sense of that when the Product Manager mentions that big back-end changes are required for Microsoft to predict when a patch will hit a certain tenant). However, it seems Microsoft *do* appreciate that there is a huge need for a better process around Office 365 patches and updates – and indeed Microsoft’s message these days is “we do expect you to build customizations on Office 365” (but no doubt the usual caveats about what is sensible apply). This is positive for organizations using Office 365 and partners who support them.


Friday, 21 February 2014

Office 365 SharePoint hybrid – what you DO and DO NOT get

As far as I can see, more and more clients are interested in “hybrid” these days – the idea of running some of their SharePoint sites in Office 365 (SharePoint Online), and some on-premises. For any organization where the I.T. group is thinking is “we like cloud, but not for everything”, the hybrid idea can be quite appealing. There can many reasons for hybrid – companies wanting to supplement their on-premises capability (perhaps for the “quick spin-up” that Office 365 offers, or to leverage particular features/functionality such as Power BI), companies doing phased migrations to the cloud, or companies with regulatory or data sovereignty constraints – hybrid can work well for these scenarios and more.

TechNet has some great coverage of hybrid configuration process (start with Hybrid for SharePoint Server 2013), but folks who are new to the landscape (especially less technical types) are often surprised by what you actually “get”, and what you don’t, in hybrid mode. Search is a particular area where you get a certain deal – and this may or may not match up with what users are expecting. This post started out as an exploration of that, but I realised that some folks might appreciate some background on hybrid before that, so I’ve split the content up into:

  1. Office 365 SharePoint hybrid – what you DO and DON’T get [this article]
  2. Office 365 hybrid and search – presenting results from both sides [not yet published]

In many ways, hybrid looks SO SIMPLE! I mean, as TechNet shows, you have some sites in Office 365 and some on-premises – frankly, my mother could probably architect this stuff, right?

Simplified SharePoint hybrid diagram_thumb[2]

In reality of course, there’s a HUGE amount to consider for a hybrid deployment! In a straight comparison of “100% on-premises”, “100% SharePoint Online” and hybrid, I think most would agree that hybrid is the most complex option. After all, you have to do all the on-premises planning (e.g. planning the SQL layer, planning for service applications etc. etc.), in addition to understanding how you will configure and use Office 365. Also, hybrid typically requires extra on-premises configuration (DirSync, maybe ADFS, and maybe reverse proxy infrastructure). And if you’re doing any kind of customisation, there is complexity there around how you build for on-premises vs. SharePoint Online (or perhaps your customisations need to always work in both) – the list goes on.

Hybrid – what you DO get

So, hybrid isn’t necessarily the magical solution it sounds like it could be. That’s not to say it’s not the right solution for you/your client – as an aside, it is for my current client. But it’s not a turn-key solution where you set “Mode = hybrid” and Office 365/SharePoint magically sets things up and all aspects of your deployment are covered. Instead, SharePoint hybrid really covers three areas:

  • Search
    • Being able to search across both environments (with caveats – see later)
  • BCS
    • Being able to access data in on-premises applications/systems from Office 365
  • Duet integration (SAP)
    • Being able to access data in on-premises SAP from Office 365

And that’s really it.

It’s true to say that some aspects of SharePoint can easily be made to work in a way which suits hybrid – My Sites, for example, can be made to exist only in SharePoint Online (if that’s the preferred option) by redirecting the My Site host to the URL in the cloud.

But otherwise, that’s basically what hybrid looks like.

Hybrid – what you DON’T get

So now let’s think about what you don’t get through “native support” – in other words, things you will need to plan for and spend time implementing yourself.

You don’t get:

  • Any kind of global navigation
    • A key issue here is the “Sites” page (where a user can find to links to”bookmarked” sites/documents i.e. things they have followed) – if a user follows a site/library/document in on-premises, that will NOT show up in the Office 365 sites page!
  • Any kind of global site directory
    • Indeed, SharePoint 2013 no longer has an out-of-the-box site directory of course, it was removed from the product
  • A joined-up social/newsfeed experience (if you are using SharePoint social rather than Yammer) across Office 365/on-premises – there are numerous issues here, but essentially any activity such as likes, comments, follows etc. in on on-premises environment will not appear in the newsfeed in SharePoint Online
  • A great search experience
    • Instead, search results are displayed in separate blocks from Office 365/on-premises – more on this later
  • Any kind of auto-deployment/synchronization of:
    • Branding
    • Master pages
    • Any other customisations packaged as (sandbox) WSPs
    • Taxonomy/Managed Metadata
      • There are numerous considerations here – all related to having two Managed Metadata Service Applications (one in Office 365 and one on-premises)
    • Content types
      • (And if you’re wondering if you can leverage the Content Type Hub, by “chaining” together an on-premise CT Hub with one in SharePoint Online – no you cannot)
    • Document templates (e.g. MyCompanyProposalTemplate.docx)
    • Many aspects of configuration:
      • e.g. Search settings (best bets, search suggestions, search schema such as managed properties etc.)
      • e.g. List and library settings (workflow, versioning settings, policies etc.)
    • Many aspects related to apps
      • Apps available in the App Catalog (because you cannot share an App Catalog between SPO and on-premises)
      • App settings (e.g. whether apps can be installed from the Store)
      • Apps which are being monitored
      • Authentication of provider-hosted apps across SPO and on-premises usually requires some planning
    • Some aspects of user profiles
      • Custom properties
      • Sync across Office 365 and on-premises profiles (i.e. you need to separately configure sync from AD to both locations)
    • [… and no doubt there are lots of other things to add to this list too]

Hopefully you get the idea - in summary, there really is only a minimal link between your Office 365 tenant and your on-premises SharePoint environment (covering search, BCS, etc.). If you need something in both places - and you often do – you’ll need to take care of that yourself.

The hybrid search experience

Search is high on that list of things which get “interesting” for hybrid. After all, your SharePoint sites are split across (at least) two environments, but you’ll usually want users to be able to search across all content easily – without having to go to two different search engines. Providing a global search can certainly be done, and if you’re new to this area there are several options (all of which require some configuration to be done in the on-premises environment at least):

  • Outbound hybrid – search center in on-premises environment brings in results from Office 365
  • Inbound hybrid – search center in Office 365 brings in results from on-premises environment (requires a supported reverse proxy device)
  • Two-way hybrid – both search centers can display results from the other environment

Configure hybrid Search for SharePoint Server 2013 is the best place to start on TechNet.

In all cases, there are limitations - most folks don’t realise that there is no way to “merge” the search results together. There are two possible options:

  1. Add a Promoted Result (i.e. a “best bet”)
  2. Add a Result Block
  3. [OK, so you can also transform the query and therefore change the results – but this doesn’t help in merging the search results.]

The Result Block approach will probably be the most common. But both this and Promoted Results are poor for the client who is expecting to “just search across everything and show me the most relevant things!” This is what the Result Block approach looks like (Office 365 Result Block in red, results from on-premises content in blue):

SharePoint hybrid search results - result block

In other words, it’s very much like a “federated” search experience – the search is executed in two stages:

  • The local SharePoint environment is searched and results returned
  • The query is also “sent” to the remote SharePoint environment, and results returned

The page then loads and shows both sets of results (assuming some items match in both places). But this experience can raise some questions:

  • How many results should be shown (from each source) on page 1?
    • HINT – you’re only allowed a maximum of 10 for a Result Block anyway :)
  • What about paging? What happens when I go to page 2?
  • How can I easily see more results from both sources?
  • How do I determine which results are more relevant, from the local or remote environment?
  • Are there any options for seeing the results in a more integrated way?

In my next post, I’ll talk about some options for merging the results – including how you might do it, caveats around this, and so on. But hopefully this post provides some background information on hybrid before then. If it sounds like I’ve painted a negative view of hybrid, that’s not my intention. It can work great, but I think it's worth noting these considerations - I do find many folks have expectations that don’t necessarily match how the bigger picture works in practice.

Sunday, 26 January 2014

Add/delete and list Remote Event Receivers with PowerShell/CSOM

Recently I’ve been looking at Remote Event Receivers more and more, as I think they are important (and somewhat unavoidable) if you are trying to build “cloud-friendly” SharePoint solutions – either because you’re on Office 365, or simply want to leave the door open for such a move. You might need to use a RER for something in the host web or app web. I’ll most likely write more on RERs in the future, but if you need a basic step-by-step guide to Remote Event Receivers, my previous post Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps) may be useful. In this post I want to share some potentially useful PowerShell (and background info), for those who work with RERs.

As I note in the article at the previous link, when you create an RER you actually need to create an app for SharePoint, even if it is an RER to be used in the host web. The app must be either provider-hosted or auto-hosted. For production use I like the provider-hosted option, even if the code is deployed to a cloud service such as Azure. This offers much more flexibility than auto-hosted, which doesn’t have the same options for configuration/monitoring/scaling-up etc. An app is needed because you are creating remote code which most likely needs to use OAuth to communicate back to SharePoint, and the trust model means that (in the standard case at least) an administrator such as the site owner needs to agree to the permissions requested by the app. This could be “Web – Read” or “Site Collection – FullControl” and so on.

Registering Remote Event Receivers

There are several ways to register RERs:

  • Declaratively
  • With CSOM code e.g. in the AppInstalled event
  • With a PowerShell/CSOM script

One big downside to the declarative approach is that it can only be used for lists in an app web (i.e. a SharePoint list deployed by the app). It cannot be used for lists in the host web (e.g. to add a RER to a list in a team site). For completeness, the declarative XML is shown below – it’s just like the XML you might have used for any other event receiver, the only difference is the new “Url” element in the XML:

** N.B. There is a code sample here but it will not show in RSS Readers - click here for full article **

I’ve found more and more that as I’m doing development, the PowerShell approach is highly useful. Sometimes you just need to directly interact with the list - perhaps to see if your RER really is registered properly, or change the URL to a different WCF service for testing. Since there is no user interface for setting RERs, some quick PowerShell is ideal – this post is to share my functions in case they’re useful. I’ll list the different ones for adding/deleting/listing RERs, then provide a full/combined script at the end.

By the way, if you’re new to the idea of PowerShell scripts which have CSOM code embedded in them, see Using CSOM in PowerShell scripts with Office 365.

Adding a new Remote Event Receiver

The function below adds a remote event receiver to a named SharePoint list.

(N.B. Note that all the functions below require a valid ClientContext object to be passed in – see the Prerequisites section towards the end of this post for more details.)

** N.B. There is a code sample here but it will not show in RSS Readers - click here for full article **

Delete a Remote Event Receiver

The function below removes a remote event receiver from a named SharePoint list.

** N.B. There is a code sample here but it will not show in RSS Readers - click here for full article **

List Remote Event Receivers on a SharePoint list

In fact, RER declarations are just like regular event receivers – the SPList class just has one “EventReceivers” property, rather than separate collections for remote event receivers and regular event receivers:

** N.B. There is a code sample here but it will not show in RSS Readers - click here for full article **

Combined/all-up script

So having walked through the functions individually, here's a combined script for download/copy and paste:

** N.B. There is a code sample here but it will not show in RSS Readers - click here for full article **


To get started with these scripts you should follow the “Getting Started” section of Using CSOM in PowerShell scripts with Office 365, including the “The top of your script – referencing DLLs and authentication” section.

All of the samples above need you to pass a valid ClientContext object – which you have once you have provided the credentials and URL, and have authenticated to SharePoint/Office 365. As in my previous posts, I obtain this in a separate file which I call something like TopOfScript_PSCSOM.ps1. For clarity, here’s an idea of what you need:

** N.B. There is a code sample here but it will not show in RSS Readers - click here for full article **


Hopefully this script is useful to those working with RERs. Using PowerShell/CSOM isn’t the only way to declare a Remote Event Receiver – in production you might use CSOM code in the AppInstalled event (which is itself declared declaratively in AppManifest.xml – the only way for that event), but the PowerShell/CSOM approach is definitely useful in dev. Happy coding!

Monday, 6 January 2014

Extending the SP2013/Office 365 search hover panel (e.g. adding “post to my feed” functionality)

The enhanced search experience is one of the great things about SharePoint 2013, and the “hover panel”, which shows more information about the result when you hover over it, is a key part of that. Being able to peek inside a document or presentation, or see a preview of a web page without clicking on the result makes for way more efficient searching. Just like search results themselves, the appearance and functionality of the hover panel can be easily extended with HTML, CSS and JavaScript – in this post I am providing some sample code I’ve used in conference talks, which I think highlights things in this area quite well. My sample could be used as-is or modified to provide some other customization to the hover panel.

For context, here’s a reminder of what the hover panel looks like:

Standard hover panel 2 
Actually that image shows a document preview *without* Office Web Apps being available – just a simple summary of the document is shown. Arguably the more normal experience is to have OWA (for example it is standard in Office 365), and in this case you can page through the document/presentation to see if it’s what you’re looking for:

Standard hover panel - OWA

Great. So now let’s look at tweaking the hover panel to add some custom behavior.

Customizing the hover panel – adding an “action”

[Minor diversion/story]: In September 2013, Microsoft added some new functionality to Office 365 which allows a user to “post to Yammer” from the search hover panel. The entry point is a new “action” in the bottom part of the hover panel:

O365 post to Yammer

As a coincidence, a few months earlier in April 2013 I had demo’d a very similar customization at the SharePoint Evolutions Conference in London, but in my case my code was posting to the SharePoint 2013 social feed (i.e. not Yammer). Spookily enough, I had even chosen the same word, “POST”:

Hover Panel actions - with guide - edges
Seeing the Office 365 change reminded me that I should post this code somewhere – maybe it could be useful to you if your organization uses the SharePoint newsfeed rather than Yammer, or you’re just generally looking for technical info on the hover panel.

[End of minor diversion/story]

My customization

My code modifies the hover panel for all item types (e.g. web page, Word doc, Excel doc etc.). As described above, the “POST” action is added to the actions bar. When clicked, a textbox appears *in the hover panel* to allow the user to type a message to post (to the accompany link):

Hover panel - before post to feed - large

[I’m sure the UX team at my place of work would have something (very bad!) to say about this, but really I’m just trying to show that you can do all sorts of things with the hover panel :)]

I display a new button to do the actual “Post” at this point. Once a message is typed and this button is clicked, an AJAX request is fired to the SharePoint social API, and a success or failure confirmation is displayed in the hover panel:

Hover panel - after post to feed - small
And as you’d expect, the message has now been broadcast to my followers (with a link to the original document) and is now listed in the activities section of my profile:

Message shown in social feed

Search hover panel – key info

The hover panel is broken down into three areas:

Hover Panel - with guide with edge 
All this stuff comes from HTML/JavaScript files in the Master Page Gallery of your SharePoint search site – so to be clear, let’s say you have a Search Center site at a URL of “/search” (as in Office 365), then these files can be found in that site’s Master Page Gallery. You’ll find the files under the “Display Templates/Search” folder path:

Search display templates
Firstly you’ll notice above that there is a HTML file for every JavaScript file – this is the case for any site which has the publishing feature enabled and therefore Design Manager support for display templates. Past that, there are three broad types of file:

  • “Item” display templates - for each item type, such as Item_PDF.js, Item_Word.js, Item_Person.js and so on. Customize these if you wish to tweak how that particular Result Type looks in search results
  • “Hover panel” templates – again for each item type, such as Item_PDF_HoverPanel.js and so on. Customize these to tweak the hover panel (which is what we’ll be doing in this post)
  • Files for “misc” things - like best bets, the search box control etc.

Importantly, as well as the “per item type” files there are some “common” files – these are hooks which can be used to implement customizations across all Result Types, without having to edit the files for each one. Effectively there is some pseudo-inheritance implemented in JavaScript by Microsoft, where both the “common” and “specific” templates are used provide the UI for a specific item (nice work Product Group). For the hover panel, this works in the following way:

  • Each “specific” file provides some outer HTML which may be specific to that item type, but also calls functions called:
    • ctx.RenderHeader()
    • ctx.RenderBody()
    • ctx.RenderFooter()
  • These functions are implemented in the “common” files

Effectively you could throw away the idea of calling the common functions for a specific item type if you wanted, and provide an entirely custom hover panel - usually you’ll want to keep the consistency though. The common files are:

Common display templates
It’s these files we’ll be modifying.

Code for my customization

So my sample adds some HTML/CSS, but also has some custom JavaScript code – effectively the moving parts are:

  • Some new CSS and JavaScript files, which need to be referenced on the page
  • Some changes to the HTML/JS files in the Master Page Gallery which are used for the hover panel

Here’s a run-down of the changes I make to which file:


Change description

Item_CommonHoverPanel_Actions.html Adds the “POST” action link which sits in the “actions” bar
Item_CommonHoverPanel_Body.html Adds the core customization – the textbox which appears for the comment to be typed into, and the button which does the actual “Post”.

Also has the JavaScript call to my custom JavaScript function which does the processing work.
COB-js-demos.js Custom JavaScript file holding my custom functions such as:

I added this to the page using a CustomAction.

As you can see, I’m modifying the HTML file and allowing the JS file to be generated. In production you may want to package the JS files as a WSP and overwrite the OOTB ones (keeping a copy of the original of course). I’ll provide a link to a zip file with all files at the end, but here is the new code in each:

** N.B. There is a code sample here but it will not show in RSS Readers - click here for full article **
** N.B. There is a code sample here but it will not show in RSS Readers - click here for full article **
** N.B. There is a code sample here but it will not show in RSS Readers - click here for full article **

So in summary we reference the custom JS file on the page (e.g. in a ScriptLink, CustomAction or script tag in the master page) and make the changes to the hover panel files. I’m supplying all the files which make up my solution in the zip linked below, but just as a set of files - you’ll need to package  them into a WSP if that’s what you need (and choose where you want CSS/JS files to deploy to). 


The search hover panel is a key part of the SharePoint 2013/Office 365 search experience, and is designed to be customized to meet different needs. A new behavior can be implemented across all item types or just for a specific one, such as a Word document or “person” result. In this sample I have shown not only how to tweak the HTML/CSS, but also provide some custom AJAX-based functionality in the hover panel. I chose to use the SharePoint social feed, but the mechanics could be adapted to any behaviour which SharePoint exposes through REST/CSOM/other client APIs.



Thursday, 5 December 2013

Using CSOM in PowerShell scripts with Office 365

In my recent “Office 365 developer decisions, tips and tricks” talk I mentioned that we’d been doing a lot of “PowerShell with CSOM” work, and this was enabling us to run scripts against SharePoint Online in the same way that we are used to (for on-premises SharePoint). This is becoming a popular approach, but since I got questions on it I thought it worth writing about.

When we first started to work with Office 365, I remember being quite concerned at the lack of PowerShell cmdlets – basically all the commands we’re used to using do not exist there. Here’s a gratuitous graph to illustrate the point:


So yes, nearly 800 PowerShell commands in SP2013 (up from around 530 in SP2010) down to a measly 30 in SharePoint Online. And those 30 mainly cover basic operations with sites, users and permissions – no scripting of, say, Managed Metadata, user profiles, search and so on. It’s true to say that some of these things are now available down at site-collection scope (needed, of course, when you don’t have a true “Central Admin” site but there are still “tenant-level” settings that you want to use script for rather than make manual changes through the UI.

So what’s a poor developer/administrator to do?

The answer is to write PowerShell as you always did, but embed CSOM code in there. More examples later, but here’s a small illustration:

# get the site collection scoped Features collections (e.g. to activate one) – not showing how to obtain $clientContext here..
$siteFeatures = $clientContext.Site.Features

So we’re using the .NET CSOM, but instead of C# we are using PowerShell’s ability to call any .NET object (indeed, nearly every script will use PowerShell’s New-Object command). All the things we came to love about PowerShell are back on the table:

  • Scripts can be easily amended, no need to recompile (or open Visual Studio)
  • We can debug with PowerGui or PowerShell ISE
  • We can leverage other things PowerShell is good at e.g. easily reading from XML files, using other PowerShell modules and other APIs (including .NET) etc.

Of course, we can only perform operations where the method exists in the .NET CSOM – that’s the boundary of what we can do.

Getting started

Step 1 – understand the landscape

The first thing to understand is that there are actually 3 different approaches for scripting against Office 365/SharePoint Online, depending on what you need to do. It might just be me, but I think that when you start it’s easy to get confused between them, or not fully appreciate that they all exist. The 3 approaches I’m thinking of are:

  • SharePoint Online cmdlets
  • MSOL cmdlets
  • PowerShell + CSOM

This post focuses on the last flavor. I also wrote a short companion post about the overall landscape and with some details/examples on the other flavors, at Using SharePoint Online and MSOL cmdlets in PowerShell with Office 365

Step 2 – prepare the machine you will run scripts against SharePoint Online

Option 1 - if you will NOT run scripts from a SP2013 box (e.g. a SP2013 VM):

You need to obtain the SharePoint DLLs which comprise the .NET CSOM, and copy them to a folder on your machine – your scripts will reference these DLLs.

  1. Go to any SharePoint 2013 server, and copy any DLL
  2. which starts with Microsoft.SharePoint.Client*.dll from the C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI folder.
  3. Store them in a folder on your machine e.g. C:\Lib – make a note of this location.


Option 2 - if you WILL run scripts from a SP2013 box (e.g. a SP2013 VM):

In this case, there is no need to copy the DLLs – your scripts will reference them in the original SharePoint install location (C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI).

The top of your script – referencing DLLs and authentication

Each .ps1 file which calls the SharePoint CSOM needs to deal with two things before you can use the API – loading the CSOM types, and authenticating/obtaining a ClientContext object. So, you’ll need this at the top of your script:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

In the scripts which follow, we’ll include this “top of script” stuff by dot-sourcing TopOfScript.ps1 in every script below – you could follow a similar approach (perhaps with a different name!) or simply paste that stuff into every script you create. If you enter a valid set of credentials and URL, running the script above should see you ready to rumble:

PS CSOM got context

Script examples

Activating a Feature in SPO

Something you might want to do at some point is enable or disable a Feature using script. The script below, like the others that follow it, all reference my TopOfScript.ps1 script above:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

PS CSOM activate feature

Enable side-loading (for app deployment)

Along very similar lines (because it also involves activating a Feature), is the idea of enabling “side-loading” on a site. By default, if you’re developing a SharePoint app it can only be F5 deployed from Visual Studio to a site created from the Developer Site template, but by enabling “side-loading” you can do it on (say) a team site too. Since the Feature isn’t visible (in the UI), you’ll need a script like this:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

PS CSOM enable side loading

Iterating webs

Sometimes you might want to loop through all the webs in a site collection, or underneath a particular web:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

PS CSOM iterate webs

(Worth noting that you also see SharePoint-hosted app webs also in the image above, since these are just subwebs (albeit ones which get accessed on the app domain URL rather than the actual host site’s web application URL).

Iterating webs, then lists, and updating a property on each list

Or how about extending the sample above to not only iterate webs, but also the lists in each - the property I'm updating on each list is the EnableVersioning property, but you easily use any other property or method in the same way:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

 PS CSOM iterate lists enable versioning

Import search schema XML

In SharePoint 2013 and Office 365, many aspects of search configuration (such as Managed Properties and Crawled Properties, Query Rules, Result Sources and Result Types) can be exported and importing between environments as an XML file. The sample below shows the import operation handled with PS + CSOM: 

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

PS CSOM import search schema


As you can hopefully see, there’s lots you can accomplish with the PowerShell and CSOM combination. Anything that can be done with CSOM API can be wrapped into a script, and you can build up a library of useful PowerShell snippets just like the old days. There are some interesting things that you CANNOT do with CSOM (such as automating the process of uploading/deploying a sandboxed WSP to Office 365), but there ARE approaches for solving even these problems, and I’ll most likely cover this (and our experiences) in future posts.

A final idea on the PowerShell + CSOM front is the idea that you can have “hybrid” scripts which can deal with both SharePoint Online and on-premises SharePoint. For example, on my current project everything we build must be deployable to both SPO and on-premises, and our scripts take a “DeploymentTarget” parameter where the values can be “Online” or “OnPremises”. There are some differences (i.e. branching) in the scripts, but for many operations the same commands can be run.

Related post - Using SharePoint Online and MSO cmdlets in PowerShell with Office 365

Using SharePoint Online and MSOL/WAAD cmdlets in PowerShell with Office 365

This post is a companion post to Using CSOM in PowerShell scripts with Office 365. As I mention over in that article, broadly there are 3 different flavors to writing PowerShell for Office 365 – exactly what commands you need to run will dictate which you use, but it’s also conceivable that you might use all 3 in the same script. When you start with Office 365, I think it’s easy to get confused between them, or not fully appreciate that they all exist. What I’m thinking of here is:




When installed you’ll have:

SharePoint Online (SPO) cmdlets These are SharePoint Online specific, and can be identified by “SPO” in the noun part of the cmdlet. 
  • Get-SPOSite (to list your site collections in Office 365)
  • New-SPOSite (to create a new site collection)
 SPO shell
MS Online (MSOL)/WAAD cmdlets These are commands related to an Office 365 tenancy (but not necessarily specific to Exchange, Lync or SharePoint) and can be identified by “Msol” in the noun part of the cmdlet.
  • Get-MSolUser (to list users in your tenancy)
  • Set-MsolUserPassword (to update a password)
 MSO shell
Using SP CSOM in PS scripts The main focus of my other post, Using CSOM in PowerShell scripts with Office 365
  • Activating a Feature in SharePoint Online
  • Updating webs or lists in SharePoint Online
No install needed – you can run this type of scripts in a regular Windows PowerShell command prompt.


A note on MSOL/Windows Azure AD cmdlets

You might be wondering why the MSOL cmdlets show “Windows Azure Active Directory..” in the shortcut title (full name is “Windows Azure Active Directory Module for Windows PowerShell”), despite everything else being labelled “MSOnline” or “MSOL”. The answer is that originally these cmdlets were known as the “Microsoft Online Services Module for Windows PowerShell cmdlets”, but since that time Microsoft have introduced Windows Azure Active Directory as a formal service offering. Every Office 365 tenancy is backed by Windows Azure Active Directory (WAAD) – and since the MSOL cmdlets always were focused around directory stuff (managing users/groups, managing synchronization with your Active Directory and so on), these have now been absorbed into the WAAD offering.

Getting started

If you’re a developer or administrator who will be working with Office 365 regularly, I recommend installing the the ‘shells’ for both the SharePoint Online and Office 365 PowerShell commands – you’ll probably need them at some point.

  1. Install PowerShell 3.0 if you don’t already have it. It’s included in the Windows Management Framework 3.0 - http://www.microsoft.com/en-us/download/details.aspx?id=34595
  2. Install the SPO cmdlets - http://office.microsoft.com/en-gb/sharepoint-help/redir/XT102919083.aspx?CTT=5&origin=HA102919087
  3. Install the MSOL cmdlets - http://technet.microsoft.com/en-us/library/jj151815.aspx#bkmk_installmodule

Once installed, you’re ready to start thinking about the “top of script” stuff (e.g. authenticating to Office 365). You’ll find that it’s very similar for both the SPO and MSOL scripts, but a different cmdlet must be run to start the session:

  • Connect-SPOService
  • Connect-MsolService

Script examples – SPO scripts

Authenticating to SharePoint Online to run SPO cmdlets:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

List all site collections in SharePoint Online:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

Recreate a site collection in SharePoint Online:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

Script examples – MSOL/WAAD scripts

Authenticating to Office 365 to run MSOL/WAAD cmdlets:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

Getting all users in in the directory

A simple MSOL example, just for completeness:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

Further reading

Appendix – full lists of SPO and MSOL/WAAD cmdlets

To help you get a sense of the full range of commands in each family (in case you don’t already have them installed), I’m listing them below:

SPO cmdlets


MSOL/WAAD cmdlets