Provisioning Managed Metadata fields in Office 365 - Part 1: dealing with multiple environments

Managed Metadata fields have always been slightly painful for SharePoint developers, but if you did any kind of site templating or Feature development in SharePoint 2010, chances are that you did some research, read some blog articles and came to understand the solution. Here I’d like to talk about it for Office 365, and show a little customization I built to help with the process we currently use. This article became long, so I split it over two articles:

1. Provisioning Managed Metadata fields in Office 365 – dealing with multiple environments [this article]
2. Provisioning Managed Metadata fields in Office 365 – building WSP packages for a specific environment/tenancy

So, back on the SharePoint 2010 situation - the deal is that some Managed Metadata details change between SharePoint environments – so unlike other fields, the same provisioning XML could not be used across dev/test/UAT/production. To be specific, the IDs of the Term Store, Group and Term Set all change between environments. As a reminder, broadly the solution was to:

• Use XML to define the “static” details of the field
• Use server-side code to “finish the job” – i.e. use the API to ask SharePoint what the IDs are (for the current environment), and then update the field config with that value

Without the 2nd step, the field will be created but will be broken – it will be grayed out and users cannot use it (SP2010 example shown here):

Posts by Ari Bakker (whose image I’m using above, thanks Ari), Wictor Wilen and Andrew Connell were popular in discussing the steps to solve this.

It’s the same deal in SharePoint 2013/Office 365 – but it turns out we need different techniques.

Why the approach doesn’t work for Office 365/SharePoint Online

Well firstly, code in sandboxed solutions is deprecated. Full-stop. [RELATED - did you hear? Microsoft are starting to clarify that sandboxed solutions without code aren’t really deprecated after all (and hopefully MSDN will reflect this soon), but CODE in sandboxed solutions is deprecated and could be phased out in future versions. Clearly this is a very important distinction.]

But even if we were happy to use sandboxed code - in Office 365/SharePoint Online, we cannot use the Microsoft.SharePoint.Taxonomy namespace in server-side code anyway – the net result is that we are unable to “finish the job” in this way to ensure the field is correctly bound to the Term Store. This is a problem! Even worse, whilst it is possible in the CSOM API to bind the field, having this execute in the provisioning process (e.g. as a site is being created from the template) is challenging, maybe impossible. Maybe you could come up with some imaginative hack, but that’s probably what it would be. And what happens if this remote code (e.g. a Remote Event Receiver) fails?

Possible solutions

A colleague of mine, Luis Mañez, did some great research – I’ll give you a quick summary here, but I strongly recommend reading his article - Deploying Managed Metadata Fields declaratively in SharePoint 2013 Online (Office 365). Here’s a summary:

In fact, it IS possible to provision Managed Metadata fields without any code, if you are willing to accept a big trade-off – you can declaratively specify the key details (such as the Term Store ID (also known as the SspId), the Group ID, the Term Set ID etc.) into your XML field definitions. Wictor alluded to this possibility in his post. But remember, these details change between environments!

So in other words, the trade-off is that you would need to rebuild your WSPs for each environment.

This is tricky for us, because on this project we choose to run multiple Office 365 tenancies, for development/test/production (something I’ll talk about in future posts) – just like a traditional mature process. So at first we said “No way! That’s against all of our ALM principles!  The exact same packages MUST move between the environments!”. But then we rationally looked at the alternatives we could see:

• Option 1 - Some elaborate “remote code” solution, perhaps involving code running separately AFTER the site has been provisioned. Until this code executed, it would not be possible to upload documents to libraries with MM fields within the sites (and similarly if this remote call actually fails for any reason,  these libraries would not function correctly until an administrator intervenes).
• Option 2 - The client needs to fix-up any Managed Metadata fields manually – across all 5000 sites we were expecting. In every list and library. Knowing that some lists/libraries have up to 5 such fields. Yeah….

Since neither of these was attractive, we continued looking at this idea of a 100% declarative definition of Managed Metadata fields. And then we realized that..

..if you do things in a certain way, you can get to the point where ONLY THE TERM STORE ID (SspId) CHANGES BETWEEN ENVIRONMENTS. That’s kinda interesting. It means that just one find/replace operation is all that’s needed – assuming you’re happy to accept the overall trade-off. Of course, having to replace the SspId is still sub-optimal, error-prone and less than awesome. But maybe we could work on that too – and that’s what these posts are really about - to show a Visual Studio customization we made to simplify this process, and make it less prone to human-error. If you want to skip ahead to this, see Provisioning Managed Metadata fields in Office 365 – Part 2: building WSP packages for a specific environment/tenancy. 

But first, let’s talk about that “if you do things in a certain way” thing (which means that the only the SspId changes between environments)..

The full recipe for Managed Metadata fields

Taking a step back for a second, if you are in “development mode” (e.g. creating a template for SharePoint sites), then successful provisioning actually involves more than just provisioning the field itself in a certain way. Effectively you should seek to provision both the Term Sets AND the fields. Do not allow administrators to create new Term Sets in the admin interface. This is because:

• This way, you can control the ID of all your Term Sets – rather than let SharePoint generate that GUID
• Because this is a static “known” ID, we can reference it elsewhere

Here’s what needs to happen:

• Term Sets are provisioned into the Term Store with “known” IDs
  • We create Term Sets (often with some default Terms) in Office 365 using PowerShell and CSOM, instead of through the admin interface. In the article I referenced earlier, Luis provides you with the core PowerShell to do “taxonomy provisioning” in this way. In our project, we basically hook this up to an XML file which defines the desired Term Store structure (e.g. Groups, Term Sets etc.) – and this XML file defines the GUIDs of such objects; WE generated them – not SharePoint/Office 365
• The “known IDs” are then used in the XML definition of the fields
  • The code sample below is an example of a Managed Metadata field being provisioned the 100% declarative way. Notice all the properties being specified in the ‘Customization’ section (something a field using the combined declarative + code approach does not have):
    
    ** N.B. My newer code samples do not show in RSS Readers - click here for full article **

If you do this, your Managed Metadata fields will work just fine:

Great. It’s certainly very valuable to know this is possible for Office 365. So now we have things so that only the SspId value needs to change between environments. But that’s still a nasty find/replace operation – how could we make this situation better?

I describe the mechanism we use in the next post - Provisioning Managed Metadata fields in Office 365 – Part 2: building WSP packages for a specific environment/tenancy.

Working with web parts within a SharePoint app

I’ve previously mentioned that “you don’t get very much” when you start creating an app which has some SharePoint-hosted components (e.g. pages). You get a bare bones ASPX page, and from there it’s down to you as the developer. Whilst playing around with apps I’ve found it interesting to experiment by adding out-of-the-box components (e.g. web parts/controls in the SharePoint namespace) to app pages. The results are quite interesting, and lead to a couple of findings which might be useful tricks in the toolbox. If nothing else, you might find this an interesting “leftfield” post about some questions you hadn’t thought to ask.

Before we dive in, here’s where we are in my overall app series:

1. SharePoint 2013 apps – architecture, capability and UX considerations
2. Getting started – creating lists, content types, fields etc. within a SharePoint app (provisioning)
3. Working with data in the app web, and why you should
4. Access end-user data (in the host web) from a SharePoint 2013 app
5. Rolling out SharePoint 2013 apps to the enterprise - tenant scope and PowerShell installs
6. Azure is the new SharePoint ‘_layouts’ directory
7. “Host web apps” – provisioning files (e.g. master pages) to the host web
8. “Host web apps” – provisioning fields and content types
9. Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps) 
10. Working with web parts within a SharePoint app [this article]

Deploying web parts within an app

If you create a SharePoint-hosted app, you might notice that the default page you get is an ASPX page rather than HTML. Also, it has several declarations at the top, allowing SharePoint controls to be used:

 

The idea here is that you might want your app to take advantage of some SharePoint building blocks (web parts/controls) rather than recreating that functionality from scratch. But since you don’t have the full page editing model in an app (it’s not SharePoint content that a contributor would edit, after all), any web parts would have to be included in the markup of the page (by the developer). I did some testing to see which web parts appear to work, and this post is really about my findings.

I expanded the markup to include some common web parts within web part zones (and added any @Register directives which I started to need). My markup was then:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

Which web parts can I (probably) use within an app?

Here’s a summary of my results:

Web part	Usable?
Content Query web part	No
Content Search web part	Yes
XsltListView web part	Yes
Core Search Results web part	Yes
Page Viewer web part	Yes
Data View web part	No
Result Script web part (for search results)	No EDIT: Yes

Content Query web part:

The CQWP appears to be a flat “no”. If you don’t have publishing enabled on the host web, you’ll get this:

This seems to be related to being unable to fetch ContentQueryMain.xsl. I notice it is a 401 rather than a 404, which is kinda interesting, but anyway if you activate publishing you then get:

So, it feels like the Content Query web part may not have received any attention to make it work within an app web. It’s worth noting here that path to the XSL references the app web (in my case, https://cob-387fbe6473fe0b.spapps.dev/Style%20Library/XSL%20Style%20Sheets/ContentQueryMain.xsl)– although it is “structurally” correct (in that the Style Library is referenced at the root), I think effectively it’s not possible to “browse” host web content on the app web domain. I even tried providing an absolute host web URL to the XSL file, but that fails also with the ‘page not found’ error.

So, perhaps the CQWP would need to be re-engineered for it to work within an app.

Content Search web part:

This web part appears to work fine, which is good news:

It’s worth knowing that Display Templates are still pulled from the host web (master page gallery), but it seems no problems arise from that.

XsltListView web part:

As I’ve mentioned in previous articles, it’s somewhat confusing the first time you add a list to an app you are developing - by default, there’s nothing on the page (e.g. a navigation link to the list) which tells you if the list is being provisioned successfully. However, since you’ll know the list URL, typing the address in the browser address bar shows it is there. To display items from the list, you’ll most likely want to use the XsltListView – you’ll need to add the markup to one of your pages manually. And when you do, you’ll see that displaying items from a list in your app web works just fine:

I wondered if it was possible to use this web part to show items from a list in the host web. Putting aside potential issues of including a token such as ~site in the URL (by testing with a hardcoded absolute URL), I found that it is not:

 

Page Viewer web part:

The idea of using the Page Viewer web part in an app could be quite interesting. I was intrigued to see if I could display a page from the host web within my app, and I found that it is, with caveats (below). The image below shows just that – a page from the host web. Some points:

• The “This page is allowed to be IFramed” message is just some text on the page
• The yellow “customized” message is because I manually edited the page in SPD to add the <WebPartPages:AllowFraming ID="AllowFraming" runat="server" /> tag. If this was provided by something else higher up (page layout/master page), this message would not be present

But the main caveat is probably that it all works fine with a hardcoded/absolute URL, but tokens such as ~site do not appear to work. So in real life you might struggle if you need to have any kind of relative URL here. Still, useful to know perhaps.

Result Script web part (search results):

[UPDATED AUGUST 22 2013: Scot Hillier got in touch, telling me that it IS possible to use this web part. The solution can be found at the bottom of this section.]

[Original text follows, to show errors you might run into:] The Result Script web part is effectively the modern day “search results” web part in SharePoint 2013. It can be used to display the results from a given search query – it is the web part used on the default SP2013 search results page, and very powerful it is too. It seems like this one should work in a SharePoint-hosted app, but I just couldn’t quite get there. I’m hoping I’ve overlooked something and it is actually possible, because the issue seems to simply be getting the right format of escaping quotation marks in the search query (the DataProviderJSON property). I tried various forms, but I either see this:

..or this:

Permutations of quotation marks/escape characters I tried include:

• ‘
• \”
• ””
• ..and so on

UPDATED – THE SOLUTION (thanks Scot!):

• The answer is to use XHTML encoding of the quotation marks – i.e. &quot; 

When you do this, you’ll see the Result Script web part does actually work within your app:

So, that’s good news, and means that we CAN take advantage of this web part inside an app! 

This seems a shame, because I can imagine lots of apps which might want to display search results. For these apps, a nice web part cannot be used – code will be needed, such as I show in Calling SharePoint search using REST (e.g. from JavaScript or an app).

Core Search Results web part:

Although it turns out to be possible to use the Result Script web part, if for any reason it’s not right for your circumstance there’s always the Core Search Results web part – this is the search results web part you know and love from earlier versions of SharePoint. It uses XSL (instead of JavaScript display templates) for rendering and doesn’t have the sophisticated Query Builder (which you wouldn’t be able to use within an app anyway), but works fine for showing search results on a page:

Conclusion

Some web parts can be used within an app, some cannot. In terms of other types of controls (e.g. SharePoint server controls), I found that the SafeControl settings of the web.config used by apps are more restrictive than normal – several namespaces and specific controls are marked as unsafe, so many server controls will be off-limits in an app page. Still, it’s quite possible that some can be used – more testing would be needed here.

The usual mantra should apply to developers of SharePoint apps as in general – check if there’s something in the box which can be used before you write code. It could be that something in there saves you considerable effort.

Next time - Using app parts (ClientWebPart) to bring app elements into the host web

Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps)

Before SharePoint 2013, we were all used to the idea of custom code running on SharePoint servers. However, this changes with SP2013 – Microsoft are either forcing us (Office 365) or steering us (on-premises) to run such code off the SharePoint boxes. And you can understand why – frankly Microsoft have no chance of providing a stable Office 365/SharePoint Online platform if it has everyone’s custom code running on it. So, SharePoint 2013 allows event receiver code to run on a remote server. This post looks at deploying Remote Event Receiver components to an Azure Website for a SharePoint site running in Office 365. Since RERs take the form of a provider-hosted app in SharePoint 2013, everything I write here also applies to a provider-hosted app too. Some changes would be required (to code and configuration steps) to achieve the same for an on-premises farm, unless you have configured that environment to trust ACS. Before we dive in, here’s where we are in my overall app series:

1. SharePoint 2013 apps – architecture, capability and UX considerations
2. Getting started – creating lists, content types, fields etc. within a SharePoint app (provisioning)
3. Working with data in the app web, and why you should
4. Access end-user data (in the host web) from a SharePoint 2013 app
5. Rolling out SharePoint 2013 apps to the enterprise - tenant scope and PowerShell installs
6. Azure is the new SharePoint ‘_layouts’ directory
7. “Host web apps” – provisioning files (e.g. master pages) to the host web
8. “Host web apps” – provisioning fields and content types
9. Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps) [this article]
10. Working with web parts within a SharePoint app

Why would I deploy this stuff to Azure?

I think a good option for SharePoint remote code (e.g. some provider-hosted apps, Remote Event Receivers etc.) is to host this code on the “Azure Websites” offering (*update May 2015 - Azure Websites are now known as "Azure Web Apps"*). This is compelling because it’s free, quick and easy to spin up, and you don’t need to provide any in-house servers (with the resulting high availability, scalability, backup/restore and performance work which is required). Another significant factor is that you don’t need to involve your gateway/networks team in getting the website (which hosts your provider-hosted app pages and/or WCF service for your Remote Event Receivers) published (on SSL of course) and accessible externally - e.g. through UAG, or whatever your perimeter device may be. Azure is already outside of your firewall and accessible/addressable over the internet of course, and Azure Websites have automatic SSL support on the default domain (which is *.azurewebsites.net). If you want to use a custom DNS host name instead (e.g. code.MyCompany.com) then you can do that too, with some extra steps.  In general, using Azure Websites is a great way to sidestep many of the infrastructure roadblocks which can derail you.

What about building my app/RER as an auto-hosted app?

* Update early 2015 - auto-hosted apps are no longer available in Office 365!*

It’s a fair question, since not only do auto-hosted apps for Office 365 “just work”, but in fact they actually run on Azure Websites underneath. So why would you not just do that? Well, this works great for demo and proof-of-concept code. But personally I wouldn’t feel comfortable recommending this architecture to a client for production, and I notice others feel the same way. Frankly there is too much “black box” going on with auto-hosted apps – there aren’t really any knobs and dials right now, and technical details are scarce (e.g. scale limits, scale-up possibilities). Also, anyone building apps for the Store will note that auto-hosted apps cannot be sold there, and it’s unclear if they will be in the future.
So, I prefer the more manual approach of deploying to Azure Websites myself. The benefits I get are:

• I can scale up from Azure Websites (shared) to reserved hardware (e.g. if my app is more heavily used, or uses more processor/memory than anticipated)
• I can use Azure’s AutoScale capabilities (currently in preview) to do this automatically based on rules I set (e.g. processor thresholds)
• I have better monitoring of my app
• I can make changes to the Azure pieces of my app without redeploying the SharePoint pieces
• I can publish in many different ways (e.g. FTP, WebDeploy, Continuous Deployment from TFS, git etc.)
• I can examine the files in Azure to do trouble-shooting by opening an FTP client

How does my O365/SP2013 permutation fit in with this article? What other things should I think about?

Permutation	Consideration
Office 365 + app in Azure Websites	The focus of this article. As detailed above, quite similar to auto-hosted apps but with way more control.
On-premises + app in Azure Websites	You’d need to make some changes to the code/process discussed here. Effectively you need to configure high-trust/S2S authentication instead, (rather than OAuth to ACS) and ensure you’re using the correct TokenHelper methods or equivalent custom code. An alternative to S2S could be on-the-fly auth.
Office 365 + app in on-premises server	Can be attractive because could easily integrate with other on-premises applications/data. However, usually more complex due to operational I.T. challenges listed above. Solvable with techniques like Azure Service Bus, BCS, or simply exposing your data through custom services (e.g. WCF) from on-premises to the internet/DMZ.
On-premises + app in on-premises server	Fairly simple because everything is behind your firewall. Requires S2S authentication configuration or on-the-fly auth.

What kind of code might this be?

Examples of such remote code in SharePoint 2013/Office 365 are:

• Remote event receivers
  • List events (e.g. ListAdding)
  • ListItem events
  • Web events
• App events
  • AppInstalling/AppInstalled
  • AppUpgrading/AppUpgraded
  • AppUninstalling/AppUninstalled
• Other app code – i.e. the entire set of functionality in a provider-hosted app

What you need – a summary

Although a simple F5 deployment will take care of many aspects during development, to properly package/deploy your app for "real use", here are some of the key things you need:

• A website to be created on Azure Websites
• A solution which uses a Remote Event Receiver – for the first time you do this, I recommend using something guaranteed to work rather than your own code (to avoid getting the code and/or RER declaration wrong). I used the BasicDataOperations SP2013 provider-hosted app on MSDN
• To have registered the app with AppRegNew.aspx on your SP2013 environment or Office tenancy – this creates a new App Principal “known” to the environment.
• Within the app:
  • All the URL references in the app to be updated with absolute URLs pointing to your Azure website e.g:
    • The declaration of the Remote Event Receiver
    • The app start page listed in AppManifest.xml etc.
  • The ID of the remote web application (i.e. the App Id) listed in the AppPrincipal section of the AppManifest.xml
• For the Visual Studio project which represents the remote web (rather than the app itself):
  • The web.config to be updated with the ClientId and ClientSecret (note that in Azure, it's also possible to to specify AppSettings values in the web app configuration, as an alternative to web.config. The steps below use this approach)
  • It to be published to your Azure website – there are many options for this, but I like WebDeploy (shown below)

The detailed process

Here’s a more detailed run through of the process - I’m focusing very much on the infrastructure/configuration here rather than the code, although I do call out some things:

1. Create or obtain your solution. As with any SharePoint app with remote components, in the BasicDataOperations app I’m using, there are two Visual Studio projects:
   1. One for the SharePoint app
   2. One for the remote components (i.e. this is the ASP.NET website which hosts the web service for the Remote Event Receiver)
2. Register your app with AppRegNew.aspx (i.e. the page which can be found at /_layouts/15/appregnew.aspx), and make a note of the App Id and App secret:
   
   
   
3. Create your Azure Website in the Azure Management Portal – make a note of the URL. In my case this is https://cobspdev.azurewebsites.net
4. Update the SharePoint app project so that the URL references point to this Azure site:
   1. The RER declaration should look something like this:
      
      ** N.B. My newer code samples do not show in RSS Readers - click here for full article **
   2. The AppManifest.xml should look something like this:
      ** N.B. My newer code samples do not show in RSS Readers - click here for full article **
      
      
      
      Note that Visual Studio will replace the "~remoteAppUrl" tokens when the app is packaged - a dialog box will appear asking you where the remote web app will be hosted, and the value you enter in the textbox (e.g. your URL in Azure) will be used.
      
      
5. Also ensure the SharePoint AppPrincipal section of AppManifest.xml lists a RemoteWebApplication, with the ClientId attribute set to the App Id (from the app registration):
   
   ** N.B. My newer code samples do not show in RSS Readers - click here for full article **
6. [OPTIONAL] If you want to debug your remote code, you should set up a Service Bus instance on Azure and configure your SharePoint app project with it (*update 2015 - this step is no longer necessary - remote debugging in Azure Web Apps now happens without Service Bus configuration*):
   1. Go to the “Service Bus” area within the Azure Management Portal, and create a new namespace for this app:
       
   2. Then click “Connection Information” for this namespace and view the details – copy the connection string:
      
   3. Finally, in Visual Studio go to the project properties for the app project (not the web project). Go to the SharePoint tab, and scroll to the bottom – ensure “Enable remote event debugging” is checked, and paste the Service Bus connection string into the textbox:
      
      If you need more information on this, see Update to debugging SharePoint 2013 remote events using Visual Studio 2012.
      
      
7. Publish the ASP.NET website to Azure – I’m using WebDeploy here (*update 2015 - note that things are simpler than this if you have the Azure SDK installed - Web Apps for your Azure subscription will be listed as a target once you click the "Publish..." button in VS. Note that you will need to sign-in to Azure from VS if you aren't already*):
   1. Download the Publish Profile for your Azure Website:
      
      
       
   2. Publish the app – importing the Publish Profile since this is the first time:
      
         
      
      
       
   3. Once the settings in the Publish Profile have been validated (as in the previous image), click “Publish” to deploy to your Azure site. You should then see a success message:
      
      
      The remote components are now deployed to Azure.
      
      
8. As we noted earlier, in Azure Web Apps AppSettings can either be specified in the web.config file as usual, or in the properties for the web app in Azure. Generally you'd use the former, but to illustrate the mechanism in Azure, here's what that option looks like - you'd go to the Azure portal and select your website. Click CONFIGURE, then scroll down to the AppSettings section – enter the ClientId and ClientSecret here:
   
     
9. Publish the SharePoint app to Office 365 (by first publishing to the filesystem):
   
    
   You’ll be presented with a dialog similar to the below (*update 2015 - the Client Secret is no longer specified here in later versions of the Visual Studio tools. Simply make sure it's correct in AppSettings for the app*)– ensure the URL and Client ID are correctly specified, and these values will be packaged into the app manifest properly
   
   
   
    


10. At this point, the app is ready and can be added to the App Catalog in your Office 365 tenancy. Go to the  “Apps for SharePoint” library within the associated App Catalog site, and either upload the .app file conventionally, or drag in:
    
    
11. Now the app can be added to a site:
    
    
12. Once the permission request has been accepted, the app is installed and can be run. Enter the app by clicking on it in the Site Contents page:
    
    

..and there is the BasicDataOperations MSDN sample app running in your Azure site, in all it’s ninja CSS and responsive design glory:


The point of course, is that you now have remote code running and have a location to host it.

Summary

In scenarios where your code must run “off-box” to SharePoint (such as Office 365), Azure Web Apps can provide a much easier way of doing this than with on-premises IIS servers. You can use Azure’s flexibility to scale up from the free option (not resilient) to one of the pay-for options which give a production-grade level of operations.

You have to consider if you are happy for your code to run there and who might "own" the use of Azure within your organization, and maybe some considerations such as authentication/integration with on-premises systems could rule it out for you. Otherwise, it can free you from dealing with lots of infrastructure aspects (especially getting the website published externally), and so is an incredibly useful tool in the toolbox.

SP2013 host web apps: provisioning fields and content types

Continuing my little diversion into SharePoint “host web apps” or “Full Control apps” (i.e. apps which provision to the host web) within my larger series about apps, today I want to look at provisioning fields (site columns) and content types – but provisioning them into the host web, rather than the app web. Just to explain where we are in my overall series, here’s the evolving table of contents:

1. SharePoint 2013 apps – architecture, capability and UX considerations
2. Getting started – creating lists, content types, fields etc. within a SharePoint app (provisioning)
3. Working with data in the app web, and why you should
4. Access end-user data (in the host web) from a SharePoint 2013 app
5. Rolling out SharePoint 2013 apps to the enterprise - tenant scope and PowerShell installs
6. Azure is the new SharePoint ‘_layouts’ directory
7. “Host web apps” – provisioning files (e.g. master pages) to the host web
8. “Host web apps” – provisioning fields and content types [this article]
9. Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps)
10. Working with web parts within a SharePoint app

The idea of provisioning fields and content types to the host web are important techniques for host web apps - since if we can have an app which creates fields/content types AND can provision files to the host web (using the techniques in my previous article), then we could use an app to deliver the building blocks for typical SharePoint collaboration solutions.

As ever, this will only be possible if:

• The app requests (and is granted) “Full Control” permission of the host web
  • Note this means it cannot be sold via the Office/SharePoint Store (see Validation checklist for apps for Office and SharePoint). But it can be deployed internally through the organization’s App Catalog..
• CSOM code is used for provisioning, rather than standard Feature XML

N.B. At this point, you might be interested in the “Should you do this? Deciding between host web apps/sandbox/farm..” section of the previous article.

Challenges when provisioning fields and content types to the host web

I think if we’re going to build apps like this, then we’d probably want some helper methods which can:

• Provision fields (as site columns)
• Provision content types
• Add fields to content types

As before, I’m using JSOM code in the default page of my SharePoint-hosted app – this could be adapted to something different if required. Some challenges I noticed were:

• It doesn’t seem possible to be able to create a content type with a specific content type ID with CSOM – perhaps I overlooked something, but in any case it does seem possible to obtain it with a later call
• It seems you have to re-fetch any fields and content types after initial provisioning in order to use them – you cannot use the objects returned from CSOM/JSOM methods. If you try this, it seems CSOM gives you cut-down objects which don’t have some members, and this causes errors like this:
  
  

How to: provision fields and content types to the host web

The code below is a little bit spaghetti-fied, due to it needing a chain of CSOM commands – an improvement would be to move the code over to using promises and jQuery’s deferred object. As it is, some of our methods need to be called in success callbacks since, for example, before we can add the field to the content type we need both to exist.

It also took me a while to realize that I needed a reference to the actual SP.Field to be able to use SP.FieldLinkCreationInformation, and that whilst I could probably obtain my content type’s ID after creation and then fetch it using this ID, simply iterating all the content types and finding mine resulted in the same number of CSOM calls. Anyway, here’s the code I used:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

Running the app

Once our app is built and we add it to a site, the person adding has to accept the Full Control permission request:

..and just like last time, my app has some sample UI which confirms the steps which have been taken:

 

The result

Now when we go back to the host web, we see that our field and content type have indeed been provisioned:

..and the field has been added to the content type, so it is ready to use:

Download the code

You can download the full Visual Studio project with the code I used for these two articles from here – download the code.

Speaking at SharePoint Saturday Netherlands, June 29th

Just a quick post to mention that I’m looking forward to speaking at SharePoint Saturday Netherlands next weekend. It’s always a privilege to be asked to speak at these things, and the event looks great with fellow speakers including Spence Harbar, Waldek Mastykarz, Wes Hackett, Wouter van Vugt and a whole host of other talented Dutch SharePoint folks.

My session is:

Using JavaScript templates to customise the SharePoint 2013 user interface

Developers often have a need to amend the HTML rendering of SharePoint, especially around lists and list forms, but in previous versions of SharePoint this was not always easy. Many developers resorted to using JavaScript to manipulate the page after it had loaded, even though this was potentially unreliable and unsupported. SharePoint 2013 solves this problem by providing the ability to specify a JavaScript 'display template' for several user interface elements. This session examines key developer scenarios, such as implementing custom rendering for SharePoint lists (including conditional formatting), customizing the display of the Content Search web part and providing an enhanced search 'hover panel' for specific items. Come and learn how to transform the SharePoint interface the supported way!

Look forward to seeing folks there!

SP2013 host web apps: provisioning files (e.g. master pages) to the host web

Here in early summer 2013, it’s a slightly strange time in SharePoint development land. The message is that sandbox solutions are “deprecated” in SharePoint 2013 (however you choose to interpret that) and that we should build apps where possible – however, apps currently cannot do many of the things that sandboxed solutions can. One key difference between apps and sandboxed solutions, of course, is that when you provision fields/content types/files etc. in your app they are created in the app web – not the “host web”. For many requirements, this “isolated space” idea works fine. However, when you really want your artifacts to exist in the day-to-day areas of SharePoint that users go to (i.e. the team/project/My sites that become known as host webs in the app world) then you need a different approach.

If you’re no longer comfortable using the sandbox, then what can we do? Well, one option to consider is that apps CAN sometimes “provision to the host web”. This article looks at this idea, but first here's a reminder of where we are in the overall series:

1. SharePoint 2013 apps – architecture, capability and UX considerations
2. Getting started – creating lists, content types, fields etc. within a SharePoint app (provisioning)
3. Working with data in the app web, and why you should
4. Access end-user data (in the host web) from a SharePoint 2013 app
5. Rolling out SharePoint 2013 apps to the enterprise - tenant scope and PowerShell installs
6. Azure is the new SharePoint ‘_layouts’ directory
7. “Host web apps” – provisioning files (e.g. master pages) to the host web [this article]
8. “Host web apps” – provisioning fields and content types
9. Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps)
10. Working with web parts within a SharePoint app

Anyway, apps can provision to the host web IF:

• The app requests (and is granted) “Full Control” permission of the host web
  • Note this means it cannot be sold via the Office/SharePoint Store (see Validation checklist for apps for Office and SharePoint). But it can be deployed internally through the organization’s App Catalog..
• CSOM code is used for provisioning, rather than standard Feature XML

All of a sudden, apps can now be used as the vehicle for deploying “regular” SharePoint functionality – perhaps components that are used in typical collaboration solutions. The code/solution in this article is taken from my “Deep-dive on SharePoint-hosted apps” talk at the SharePoint Evolutions Conference, and this article is the first in a series of short posts on such “host web apps” (within my wider series on SP2013 apps). But first..

Should you do this? Deciding between host web apps/sandbox/farm..

I think this approach has a place – especially if you are paranoid about the life expectancy of sandboxed solutions, or someone is insisting that the app model is used. However, my feeling is that we are effectively exploiting a loophole here. I’m sure Microsoft did not particularly intend apps to be used in this way (as evidenced by difficult CSOM-based deployment model), but, possibly because of other concerns, they have robust mechanisms in place such that it isn’t proactively blocked. Me? Well, personally I feel that if the client requirements steer you towards working in the host web, then apps aren’t the right vehicle – if you’re cloud-focused, then I believe you should be using a design centred around the sandbox. Despite the "deprecated” tag I think Microsoft will have to extend the app model before they can really remove this option. In other words, I think we should understand that we are in a transition period and new development approaches will become available – but I won’t feel bad about using the best tool for the job right now.

I should add that this might not apply to product development – an area which I currently do not have to worry about. My friend Doug Ware is building an app based around the host web, and has dug far deeper into this than me. We’ve previously debated this topic in our blog posts – you can my version (which links to his posts) at SharePoint apps: working with the app web, and why you should. You’ll see that I’m effectively softening my position slightly in this post.

How to: provision a master page (or any file) to the host web

In this example, I’m choosing to provision a master page – but the approach works for other file types too. Arguably the best approach is to provision the file, initially at least, to the app web. This works because later on we’ll need to fetch the file contents from somewhere, rather than declare the entire file contents directly in a JavaScript variable (not very practical). So, we add the file to our app and use a Module element to provision it into the app web. I found that provisioning with a .txt extension was best at this stage (more on this later):

The code

Then we need a wodge of CSOM code – specifically JSOM in my case. I’m using a SharePoint-hosted app, where the code executes on page load of the app default page. In the code we have to pull a few tricks – firstly we make a jQuery GET request to the .txt file in the app web, in order to obtain the contents. N.B. This was the reason we provisioned it with a .txt extension at first – for some file types (e.g. .master, .aspx) you might find that the file contents are not what you expect. This can happen because the page could not be executed/parsed properly by SharePoint, i.e. a runtime error occurred because you effectively browsed the master page/page layout/whatever directly. This undesired behavior goes away if you are simply requesting a .txt file.

Then we use JSOM to open a connection to the host web. We do this by passing a relative URL to the host web to the SP.ClientContext constructor, instead of asking for SP.ClientContext.get_current(), which would give us context in the app web where our page is running.

Our JSOM code then uploads to the host web (via a method which can be re-used to provision any file to any path in the host web), and just for good luck we go ahead and set the master on the host web. The code below has no external dependencies, and should work fine if you paste it into your app:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

Running the app

Once our app is built and we add it to a site, the person adding has to accept the Full Control permission request:

 

And, since the model here is that our JSOM code executes when the page is loaded, our master page is indeed provisioned when the user navigates to the app. My sample code presents show some simple UI to confirm this:

The result

Now when we go back to the host web, we see that our master page has indeed been provisioned to the Master Page Gallery:

..and because we set it to be the default master page, any branding changes in this master page (such as a red bar in my case) have been applied to the site:

So, if you really want to avoid sandboxed solutions or proactively want to use apps, then you can see that these CSOM techniques can be useful. In future posts, I'll follow up with further code for other common scenarios.

Download the code

You can download the full Visual Studio project with the code I used for these two articles (on host web apps) from here – download the code.

Azure is the new SharePoint ‘layouts’ directory

SharePoint developers have always had a need to store files in a central location that can be accessed/shared from ALL SharePoint sites. This is often needed for supporting files such as images, CSS and JavaScript - usually we don’t want such files to exist in each individual site, because when updates are needed we might then have 10,000 different places to update the file. As you can imagine, there are other reasons to avoid this duplication too.

One long-established way of working around this is to deploy such files to the server filesystem, rather than into a specific SharePoint site. Commonly, the ‘_layouts’ folder under SharePoint root directory (i.e. ‘14’ for SharePoint 2010 or ‘15’ for SharePoint 2013) was used for such shared files. A second method involved deploying files as ‘uncustomized’, or ‘GhostableInLibrary’. This works by adding a list item ‘stub’ for the file into each site or library, but since the file is also deployed to the SharePoint server’s filesystem, the file contents are pulled from there. In the content database, the SQL record literally has a pointer to the location of the physical file on the SharePoint server (again, this would be somewhere under SharePoint root directory).

Although not just related to app development, I included this article in my 10 part series on apps:

1. SharePoint 2013 apps – architecture, capability and UX considerations
2. Getting started – creating lists, content types, fields etc. within a SharePoint app (provisioning)
3. Working with data in the app web, and why you should
4. Access end-user data (in the host web) from a SharePoint 2013 app
5. Rolling out SharePoint 2013 apps to the enterprise - tenant scope and PowerShell installs
6. Azure is the new SharePoint ‘_layouts’ directory [this article]
7. “Host web apps” – provisioning files (e.g. master pages) to the host web
8. “Host web apps” – provisioning fields and content types
9. Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps)
10. Working with web parts within a SharePoint app

Cloud says no

Once you start to develop sandboxed solutions or SharePoint 2013 apps, of course the technique above cannot be used. Files cannot be provisioned to the server filesystem, since the servers might not belong to you – in the case of Office 365, the servers are run by Microsoft. So what can we do?

Well, something I’m sure I’m not the first to think of is that if the production servers are connected to the internet (and cloud services like Office 365 always are), then you can just store your images, CSS and JavaScript files on some other servers (or service), and your pages will run just fine. For what it’s worth, I’ve been doing this on my blog site for quite a while – if you go looking, you’ll see I have some CSS and JavaScript being loaded ‘across the internet’ from http://sharepointnutsandbolts.azurewebsites.net rather than http://www.sharepointnutsandbolts.com.  Like me, you might find that Azure is a good choice here – it’s a solid offering with several flavors (I use the free version), and numerous options exist for getting your files there (FTP, WebDeploy, continuous deployment from TFS/Git etc.). Of course, any similar cloud service would work fine too – as indeed might some on-premises non-SharePoint servers (rather than service) you supply. Just remember that with that latter option, you’d need to take care of high-availability, backup/restore, maybe load-balancing etc. Thanks to Azure, I can let Microsoft deal with that :)

As you can imagine, we now regain the ability to store a single instance of files, even if not on the SharePoint server itself. To illustrate, here’s what the ‘default’ and ‘centralized’ approaches look like in pictures:

Default app/sandboxed development model:

Centralized development model:

When should I be considering this?

I think this broad approach is relevant to the following scenarios:

• Developing sandboxed solutions
• Developing SharePoint-hosted apps
• Developing auto-hosted apps

The scenario I’ve left out is provider-hosted apps. This is because here your supporting images, CSS and JavaScript files are stored outside of SharePoint anyway – and by default are stored as a single instance. After all, a provider-hosted app is just a non-SharePoint website at the end of the day. Auto-hosted apps are perhaps an interesting case because the app’s supporting files are deployed to Azure, but the same deal applies – they are not shared between app instances.

If you are working in these models and not currently considering this approach, perhaps you should. Yes, it can make the initial development process more complex (since your app is effectively stored in multiple places), but rolling out updates once in production will most likely be much simpler. Even if you don’t like this idea, then at least ensure you’ve considered your update mechanisms for these files.

An interesting observation (if it hadn’t occurred to you already), is that provider-hosted apps are mainly stored outside SharePoint – including their business logic. This means, amongst other things, that the creator has the option of making fairly big changes to the app without requiring the app to be upgraded or resubmitted to the Store. Certainly worth bearing in mind.

Further considerations

I think the following also need to be considered:

• Versioning
• You’ll most likely need to provide some form of versioning of your “remotely-stored” CSS and JavaScript files, for example storing them under folders named “v1”, “v2” or “1.0.0.0”, “2.0.0.0” etc. This will allow you to deploy updates for later versions of your apps, whilst maintaining the original experience for users/sites who do not upgrade
• Remember also that if you did want to make changes to existing CSS/JS files, these will most likely be cached at the browser for existing users. So you may need to do something to bust the cache (as discussed for non-app scenarios in Avoiding bugs from cached JavaScript and CSS files in SharePoint), but remember also that changes to the URL used will require changes/upgrades to the app itself unless the page is provider-hosted
• Alternative storage flavors:
• Although I’ve mainly discussed storing files in simple Azure websites, my colleague Salvatore di Fazio has also considered the use of Azure CDN –see his post Sharepoint 2013 – How to share contents between Apps