Working with web parts within a SharePoint app

I’ve previously mentioned that “you don’t get very much” when you start creating an app which has some SharePoint-hosted components (e.g. pages). You get a bare bones ASPX page, and from there it’s down to you as the developer. Whilst playing around with apps I’ve found it interesting to experiment by adding out-of-the-box components (e.g. web parts/controls in the SharePoint namespace) to app pages. The results are quite interesting, and lead to a couple of findings which might be useful tricks in the toolbox. If nothing else, you might find this an interesting “leftfield” post about some questions you hadn’t thought to ask.

Before we dive in, here’s where we are in my overall app series:

1. SharePoint 2013 apps – architecture, capability and UX considerations
2. Getting started – creating lists, content types, fields etc. within a SharePoint app (provisioning)
3. Working with data in the app web, and why you should
4. Access end-user data (in the host web) from a SharePoint 2013 app
5. Rolling out SharePoint 2013 apps to the enterprise - tenant scope and PowerShell installs
6. Azure is the new SharePoint ‘_layouts’ directory
7. “Host web apps” – provisioning files (e.g. master pages) to the host web
8. “Host web apps” – provisioning fields and content types
9. Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps) 
10. Working with web parts within a SharePoint app [this article]

Deploying web parts within an app

If you create a SharePoint-hosted app, you might notice that the default page you get is an ASPX page rather than HTML. Also, it has several declarations at the top, allowing SharePoint controls to be used:

 

The idea here is that you might want your app to take advantage of some SharePoint building blocks (web parts/controls) rather than recreating that functionality from scratch. But since you don’t have the full page editing model in an app (it’s not SharePoint content that a contributor would edit, after all), any web parts would have to be included in the markup of the page (by the developer). I did some testing to see which web parts appear to work, and this post is really about my findings.

I expanded the markup to include some common web parts within web part zones (and added any @Register directives which I started to need). My markup was then:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

Which web parts can I (probably) use within an app?

Here’s a summary of my results:

Web part	Usable?
Content Query web part	No
Content Search web part	Yes
XsltListView web part	Yes
Core Search Results web part	Yes
Page Viewer web part	Yes
Data View web part	No
Result Script web part (for search results)	No EDIT: Yes

Content Query web part:

The CQWP appears to be a flat “no”. If you don’t have publishing enabled on the host web, you’ll get this:

This seems to be related to being unable to fetch ContentQueryMain.xsl. I notice it is a 401 rather than a 404, which is kinda interesting, but anyway if you activate publishing you then get:

So, it feels like the Content Query web part may not have received any attention to make it work within an app web. It’s worth noting here that path to the XSL references the app web (in my case, https://cob-387fbe6473fe0b.spapps.dev/Style%20Library/XSL%20Style%20Sheets/ContentQueryMain.xsl)– although it is “structurally” correct (in that the Style Library is referenced at the root), I think effectively it’s not possible to “browse” host web content on the app web domain. I even tried providing an absolute host web URL to the XSL file, but that fails also with the ‘page not found’ error.

So, perhaps the CQWP would need to be re-engineered for it to work within an app.

Content Search web part:

This web part appears to work fine, which is good news:

It’s worth knowing that Display Templates are still pulled from the host web (master page gallery), but it seems no problems arise from that.

XsltListView web part:

As I’ve mentioned in previous articles, it’s somewhat confusing the first time you add a list to an app you are developing - by default, there’s nothing on the page (e.g. a navigation link to the list) which tells you if the list is being provisioned successfully. However, since you’ll know the list URL, typing the address in the browser address bar shows it is there. To display items from the list, you’ll most likely want to use the XsltListView – you’ll need to add the markup to one of your pages manually. And when you do, you’ll see that displaying items from a list in your app web works just fine:

I wondered if it was possible to use this web part to show items from a list in the host web. Putting aside potential issues of including a token such as ~site in the URL (by testing with a hardcoded absolute URL), I found that it is not:

 

Page Viewer web part:

The idea of using the Page Viewer web part in an app could be quite interesting. I was intrigued to see if I could display a page from the host web within my app, and I found that it is, with caveats (below). The image below shows just that – a page from the host web. Some points:

• The “This page is allowed to be IFramed” message is just some text on the page
• The yellow “customized” message is because I manually edited the page in SPD to add the <WebPartPages:AllowFraming ID="AllowFraming" runat="server" /> tag. If this was provided by something else higher up (page layout/master page), this message would not be present

But the main caveat is probably that it all works fine with a hardcoded/absolute URL, but tokens such as ~site do not appear to work. So in real life you might struggle if you need to have any kind of relative URL here. Still, useful to know perhaps.

Result Script web part (search results):

[UPDATED AUGUST 22 2013: Scot Hillier got in touch, telling me that it IS possible to use this web part. The solution can be found at the bottom of this section.]

[Original text follows, to show errors you might run into:] The Result Script web part is effectively the modern day “search results” web part in SharePoint 2013. It can be used to display the results from a given search query – it is the web part used on the default SP2013 search results page, and very powerful it is too. It seems like this one should work in a SharePoint-hosted app, but I just couldn’t quite get there. I’m hoping I’ve overlooked something and it is actually possible, because the issue seems to simply be getting the right format of escaping quotation marks in the search query (the DataProviderJSON property). I tried various forms, but I either see this:

..or this:

Permutations of quotation marks/escape characters I tried include:

• ‘
• \”
• ””
• ..and so on

UPDATED – THE SOLUTION (thanks Scot!):

• The answer is to use XHTML encoding of the quotation marks – i.e. &quot; 

When you do this, you’ll see the Result Script web part does actually work within your app:

So, that’s good news, and means that we CAN take advantage of this web part inside an app! 

This seems a shame, because I can imagine lots of apps which might want to display search results. For these apps, a nice web part cannot be used – code will be needed, such as I show in Calling SharePoint search using REST (e.g. from JavaScript or an app).

Core Search Results web part:

Although it turns out to be possible to use the Result Script web part, if for any reason it’s not right for your circumstance there’s always the Core Search Results web part – this is the search results web part you know and love from earlier versions of SharePoint. It uses XSL (instead of JavaScript display templates) for rendering and doesn’t have the sophisticated Query Builder (which you wouldn’t be able to use within an app anyway), but works fine for showing search results on a page:

Conclusion

Some web parts can be used within an app, some cannot. In terms of other types of controls (e.g. SharePoint server controls), I found that the SafeControl settings of the web.config used by apps are more restrictive than normal – several namespaces and specific controls are marked as unsafe, so many server controls will be off-limits in an app page. Still, it’s quite possible that some can be used – more testing would be needed here.

The usual mantra should apply to developers of SharePoint apps as in general – check if there’s something in the box which can be used before you write code. It could be that something in there saves you considerable effort.

Next time - Using app parts (ClientWebPart) to bring app elements into the host web

Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps)

Before SharePoint 2013, we were all used to the idea of custom code running on SharePoint servers. However, this changes with SP2013 – Microsoft are either forcing us (Office 365) or steering us (on-premises) to run such code off the SharePoint boxes. And you can understand why – frankly Microsoft have no chance of providing a stable Office 365/SharePoint Online platform if it has everyone’s custom code running on it. So, SharePoint 2013 allows event receiver code to run on a remote server. This post looks at deploying Remote Event Receiver components to an Azure Website for a SharePoint site running in Office 365. Since RERs take the form of a provider-hosted app in SharePoint 2013, everything I write here also applies to a provider-hosted app too. Some changes would be required (to code and configuration steps) to achieve the same for an on-premises farm, unless you have configured that environment to trust ACS. Before we dive in, here’s where we are in my overall app series:

1. SharePoint 2013 apps – architecture, capability and UX considerations
2. Getting started – creating lists, content types, fields etc. within a SharePoint app (provisioning)
3. Working with data in the app web, and why you should
4. Access end-user data (in the host web) from a SharePoint 2013 app
5. Rolling out SharePoint 2013 apps to the enterprise - tenant scope and PowerShell installs
6. Azure is the new SharePoint ‘_layouts’ directory
7. “Host web apps” – provisioning files (e.g. master pages) to the host web
8. “Host web apps” – provisioning fields and content types
9. Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps) [this article]
10. Working with web parts within a SharePoint app

Why would I deploy this stuff to Azure?

I think a good option for SharePoint remote code (e.g. some provider-hosted apps, Remote Event Receivers etc.) is to host this code on the “Azure Websites” offering (*update May 2015 - Azure Websites are now known as "Azure Web Apps"*). This is compelling because it’s free, quick and easy to spin up, and you don’t need to provide any in-house servers (with the resulting high availability, scalability, backup/restore and performance work which is required). Another significant factor is that you don’t need to involve your gateway/networks team in getting the website (which hosts your provider-hosted app pages and/or WCF service for your Remote Event Receivers) published (on SSL of course) and accessible externally - e.g. through UAG, or whatever your perimeter device may be. Azure is already outside of your firewall and accessible/addressable over the internet of course, and Azure Websites have automatic SSL support on the default domain (which is *.azurewebsites.net). If you want to use a custom DNS host name instead (e.g. code.MyCompany.com) then you can do that too, with some extra steps.  In general, using Azure Websites is a great way to sidestep many of the infrastructure roadblocks which can derail you.

What about building my app/RER as an auto-hosted app?

* Update early 2015 - auto-hosted apps are no longer available in Office 365!*

It’s a fair question, since not only do auto-hosted apps for Office 365 “just work”, but in fact they actually run on Azure Websites underneath. So why would you not just do that? Well, this works great for demo and proof-of-concept code. But personally I wouldn’t feel comfortable recommending this architecture to a client for production, and I notice others feel the same way. Frankly there is too much “black box” going on with auto-hosted apps – there aren’t really any knobs and dials right now, and technical details are scarce (e.g. scale limits, scale-up possibilities). Also, anyone building apps for the Store will note that auto-hosted apps cannot be sold there, and it’s unclear if they will be in the future.
So, I prefer the more manual approach of deploying to Azure Websites myself. The benefits I get are:

• I can scale up from Azure Websites (shared) to reserved hardware (e.g. if my app is more heavily used, or uses more processor/memory than anticipated)
• I can use Azure’s AutoScale capabilities (currently in preview) to do this automatically based on rules I set (e.g. processor thresholds)
• I have better monitoring of my app
• I can make changes to the Azure pieces of my app without redeploying the SharePoint pieces
• I can publish in many different ways (e.g. FTP, WebDeploy, Continuous Deployment from TFS, git etc.)
• I can examine the files in Azure to do trouble-shooting by opening an FTP client

How does my O365/SP2013 permutation fit in with this article? What other things should I think about?

Permutation	Consideration
Office 365 + app in Azure Websites	The focus of this article. As detailed above, quite similar to auto-hosted apps but with way more control.
On-premises + app in Azure Websites	You’d need to make some changes to the code/process discussed here. Effectively you need to configure high-trust/S2S authentication instead, (rather than OAuth to ACS) and ensure you’re using the correct TokenHelper methods or equivalent custom code. An alternative to S2S could be on-the-fly auth.
Office 365 + app in on-premises server	Can be attractive because could easily integrate with other on-premises applications/data. However, usually more complex due to operational I.T. challenges listed above. Solvable with techniques like Azure Service Bus, BCS, or simply exposing your data through custom services (e.g. WCF) from on-premises to the internet/DMZ.
On-premises + app in on-premises server	Fairly simple because everything is behind your firewall. Requires S2S authentication configuration or on-the-fly auth.

What kind of code might this be?

Examples of such remote code in SharePoint 2013/Office 365 are:

• Remote event receivers
  • List events (e.g. ListAdding)
  • ListItem events
  • Web events
• App events
  • AppInstalling/AppInstalled
  • AppUpgrading/AppUpgraded
  • AppUninstalling/AppUninstalled
• Other app code – i.e. the entire set of functionality in a provider-hosted app

What you need – a summary

Although a simple F5 deployment will take care of many aspects during development, to properly package/deploy your app for "real use", here are some of the key things you need:

• A website to be created on Azure Websites
• A solution which uses a Remote Event Receiver – for the first time you do this, I recommend using something guaranteed to work rather than your own code (to avoid getting the code and/or RER declaration wrong). I used the BasicDataOperations SP2013 provider-hosted app on MSDN
• To have registered the app with AppRegNew.aspx on your SP2013 environment or Office tenancy – this creates a new App Principal “known” to the environment.
• Within the app:
  • All the URL references in the app to be updated with absolute URLs pointing to your Azure website e.g:
    • The declaration of the Remote Event Receiver
    • The app start page listed in AppManifest.xml etc.
  • The ID of the remote web application (i.e. the App Id) listed in the AppPrincipal section of the AppManifest.xml
• For the Visual Studio project which represents the remote web (rather than the app itself):
  • The web.config to be updated with the ClientId and ClientSecret (note that in Azure, it's also possible to to specify AppSettings values in the web app configuration, as an alternative to web.config. The steps below use this approach)
  • It to be published to your Azure website – there are many options for this, but I like WebDeploy (shown below)

The detailed process

Here’s a more detailed run through of the process - I’m focusing very much on the infrastructure/configuration here rather than the code, although I do call out some things:

1. Create or obtain your solution. As with any SharePoint app with remote components, in the BasicDataOperations app I’m using, there are two Visual Studio projects:
   1. One for the SharePoint app
   2. One for the remote components (i.e. this is the ASP.NET website which hosts the web service for the Remote Event Receiver)
2. Register your app with AppRegNew.aspx (i.e. the page which can be found at /_layouts/15/appregnew.aspx), and make a note of the App Id and App secret:
   
   
   
3. Create your Azure Website in the Azure Management Portal – make a note of the URL. In my case this is https://cobspdev.azurewebsites.net
4. Update the SharePoint app project so that the URL references point to this Azure site:
   1. The RER declaration should look something like this:
      
      ** N.B. My newer code samples do not show in RSS Readers - click here for full article **
   2. The AppManifest.xml should look something like this:
      ** N.B. My newer code samples do not show in RSS Readers - click here for full article **
      
      
      
      Note that Visual Studio will replace the "~remoteAppUrl" tokens when the app is packaged - a dialog box will appear asking you where the remote web app will be hosted, and the value you enter in the textbox (e.g. your URL in Azure) will be used.
      
      
5. Also ensure the SharePoint AppPrincipal section of AppManifest.xml lists a RemoteWebApplication, with the ClientId attribute set to the App Id (from the app registration):
   
   ** N.B. My newer code samples do not show in RSS Readers - click here for full article **
6. [OPTIONAL] If you want to debug your remote code, you should set up a Service Bus instance on Azure and configure your SharePoint app project with it (*update 2015 - this step is no longer necessary - remote debugging in Azure Web Apps now happens without Service Bus configuration*):
   1. Go to the “Service Bus” area within the Azure Management Portal, and create a new namespace for this app:
       
   2. Then click “Connection Information” for this namespace and view the details – copy the connection string:
      
   3. Finally, in Visual Studio go to the project properties for the app project (not the web project). Go to the SharePoint tab, and scroll to the bottom – ensure “Enable remote event debugging” is checked, and paste the Service Bus connection string into the textbox:
      
      If you need more information on this, see Update to debugging SharePoint 2013 remote events using Visual Studio 2012.
      
      
7. Publish the ASP.NET website to Azure – I’m using WebDeploy here (*update 2015 - note that things are simpler than this if you have the Azure SDK installed - Web Apps for your Azure subscription will be listed as a target once you click the "Publish..." button in VS. Note that you will need to sign-in to Azure from VS if you aren't already*):
   1. Download the Publish Profile for your Azure Website:
      
      
       
   2. Publish the app – importing the Publish Profile since this is the first time:
      
         
      
      
       
   3. Once the settings in the Publish Profile have been validated (as in the previous image), click “Publish” to deploy to your Azure site. You should then see a success message:
      
      
      The remote components are now deployed to Azure.
      
      
8. As we noted earlier, in Azure Web Apps AppSettings can either be specified in the web.config file as usual, or in the properties for the web app in Azure. Generally you'd use the former, but to illustrate the mechanism in Azure, here's what that option looks like - you'd go to the Azure portal and select your website. Click CONFIGURE, then scroll down to the AppSettings section – enter the ClientId and ClientSecret here:
   
     
9. Publish the SharePoint app to Office 365 (by first publishing to the filesystem):
   
    
   You’ll be presented with a dialog similar to the below (*update 2015 - the Client Secret is no longer specified here in later versions of the Visual Studio tools. Simply make sure it's correct in AppSettings for the app*)– ensure the URL and Client ID are correctly specified, and these values will be packaged into the app manifest properly
   
   
   
    


10. At this point, the app is ready and can be added to the App Catalog in your Office 365 tenancy. Go to the  “Apps for SharePoint” library within the associated App Catalog site, and either upload the .app file conventionally, or drag in:
    
    
11. Now the app can be added to a site:
    
    
12. Once the permission request has been accepted, the app is installed and can be run. Enter the app by clicking on it in the Site Contents page:
    
    

..and there is the BasicDataOperations MSDN sample app running in your Azure site, in all it’s ninja CSS and responsive design glory:


The point of course, is that you now have remote code running and have a location to host it.

Summary

In scenarios where your code must run “off-box” to SharePoint (such as Office 365), Azure Web Apps can provide a much easier way of doing this than with on-premises IIS servers. You can use Azure’s flexibility to scale up from the free option (not resilient) to one of the pay-for options which give a production-grade level of operations.

You have to consider if you are happy for your code to run there and who might "own" the use of Azure within your organization, and maybe some considerations such as authentication/integration with on-premises systems could rule it out for you. Otherwise, it can free you from dealing with lots of infrastructure aspects (especially getting the website published externally), and so is an incredibly useful tool in the toolbox.

SP2013 host web apps: provisioning fields and content types

Continuing my little diversion into SharePoint “host web apps” or “Full Control apps” (i.e. apps which provision to the host web) within my larger series about apps, today I want to look at provisioning fields (site columns) and content types – but provisioning them into the host web, rather than the app web. Just to explain where we are in my overall series, here’s the evolving table of contents:

1. SharePoint 2013 apps – architecture, capability and UX considerations
2. Getting started – creating lists, content types, fields etc. within a SharePoint app (provisioning)
3. Working with data in the app web, and why you should
4. Access end-user data (in the host web) from a SharePoint 2013 app
5. Rolling out SharePoint 2013 apps to the enterprise - tenant scope and PowerShell installs
6. Azure is the new SharePoint ‘_layouts’ directory
7. “Host web apps” – provisioning files (e.g. master pages) to the host web
8. “Host web apps” – provisioning fields and content types [this article]
9. Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps)
10. Working with web parts within a SharePoint app

The idea of provisioning fields and content types to the host web are important techniques for host web apps - since if we can have an app which creates fields/content types AND can provision files to the host web (using the techniques in my previous article), then we could use an app to deliver the building blocks for typical SharePoint collaboration solutions.

As ever, this will only be possible if:

• The app requests (and is granted) “Full Control” permission of the host web
  • Note this means it cannot be sold via the Office/SharePoint Store (see Validation checklist for apps for Office and SharePoint). But it can be deployed internally through the organization’s App Catalog..
• CSOM code is used for provisioning, rather than standard Feature XML

N.B. At this point, you might be interested in the “Should you do this? Deciding between host web apps/sandbox/farm..” section of the previous article.

Challenges when provisioning fields and content types to the host web

I think if we’re going to build apps like this, then we’d probably want some helper methods which can:

• Provision fields (as site columns)
• Provision content types
• Add fields to content types

As before, I’m using JSOM code in the default page of my SharePoint-hosted app – this could be adapted to something different if required. Some challenges I noticed were:

• It doesn’t seem possible to be able to create a content type with a specific content type ID with CSOM – perhaps I overlooked something, but in any case it does seem possible to obtain it with a later call
• It seems you have to re-fetch any fields and content types after initial provisioning in order to use them – you cannot use the objects returned from CSOM/JSOM methods. If you try this, it seems CSOM gives you cut-down objects which don’t have some members, and this causes errors like this:
  
  

How to: provision fields and content types to the host web

The code below is a little bit spaghetti-fied, due to it needing a chain of CSOM commands – an improvement would be to move the code over to using promises and jQuery’s deferred object. As it is, some of our methods need to be called in success callbacks since, for example, before we can add the field to the content type we need both to exist.

It also took me a while to realize that I needed a reference to the actual SP.Field to be able to use SP.FieldLinkCreationInformation, and that whilst I could probably obtain my content type’s ID after creation and then fetch it using this ID, simply iterating all the content types and finding mine resulted in the same number of CSOM calls. Anyway, here’s the code I used:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

Running the app

Once our app is built and we add it to a site, the person adding has to accept the Full Control permission request:

..and just like last time, my app has some sample UI which confirms the steps which have been taken:

 

The result

Now when we go back to the host web, we see that our field and content type have indeed been provisioned:

..and the field has been added to the content type, so it is ready to use:

Download the code

You can download the full Visual Studio project with the code I used for these two articles from here – download the code.

Speaking at SharePoint Saturday Netherlands, June 29th

Just a quick post to mention that I’m looking forward to speaking at SharePoint Saturday Netherlands next weekend. It’s always a privilege to be asked to speak at these things, and the event looks great with fellow speakers including Spence Harbar, Waldek Mastykarz, Wes Hackett, Wouter van Vugt and a whole host of other talented Dutch SharePoint folks.

My session is:

Using JavaScript templates to customise the SharePoint 2013 user interface

Developers often have a need to amend the HTML rendering of SharePoint, especially around lists and list forms, but in previous versions of SharePoint this was not always easy. Many developers resorted to using JavaScript to manipulate the page after it had loaded, even though this was potentially unreliable and unsupported. SharePoint 2013 solves this problem by providing the ability to specify a JavaScript 'display template' for several user interface elements. This session examines key developer scenarios, such as implementing custom rendering for SharePoint lists (including conditional formatting), customizing the display of the Content Search web part and providing an enhanced search 'hover panel' for specific items. Come and learn how to transform the SharePoint interface the supported way!

Look forward to seeing folks there!

SP2013 host web apps: provisioning files (e.g. master pages) to the host web

Here in early summer 2013, it’s a slightly strange time in SharePoint development land. The message is that sandbox solutions are “deprecated” in SharePoint 2013 (however you choose to interpret that) and that we should build apps where possible – however, apps currently cannot do many of the things that sandboxed solutions can. One key difference between apps and sandboxed solutions, of course, is that when you provision fields/content types/files etc. in your app they are created in the app web – not the “host web”. For many requirements, this “isolated space” idea works fine. However, when you really want your artifacts to exist in the day-to-day areas of SharePoint that users go to (i.e. the team/project/My sites that become known as host webs in the app world) then you need a different approach.

If you’re no longer comfortable using the sandbox, then what can we do? Well, one option to consider is that apps CAN sometimes “provision to the host web”. This article looks at this idea, but first here's a reminder of where we are in the overall series:

1. SharePoint 2013 apps – architecture, capability and UX considerations
2. Getting started – creating lists, content types, fields etc. within a SharePoint app (provisioning)
3. Working with data in the app web, and why you should
4. Access end-user data (in the host web) from a SharePoint 2013 app
5. Rolling out SharePoint 2013 apps to the enterprise - tenant scope and PowerShell installs
6. Azure is the new SharePoint ‘_layouts’ directory
7. “Host web apps” – provisioning files (e.g. master pages) to the host web [this article]
8. “Host web apps” – provisioning fields and content types
9. Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps)
10. Working with web parts within a SharePoint app

Anyway, apps can provision to the host web IF:

• The app requests (and is granted) “Full Control” permission of the host web
  • Note this means it cannot be sold via the Office/SharePoint Store (see Validation checklist for apps for Office and SharePoint). But it can be deployed internally through the organization’s App Catalog..
• CSOM code is used for provisioning, rather than standard Feature XML

All of a sudden, apps can now be used as the vehicle for deploying “regular” SharePoint functionality – perhaps components that are used in typical collaboration solutions. The code/solution in this article is taken from my “Deep-dive on SharePoint-hosted apps” talk at the SharePoint Evolutions Conference, and this article is the first in a series of short posts on such “host web apps” (within my wider series on SP2013 apps). But first..

Should you do this? Deciding between host web apps/sandbox/farm..

I think this approach has a place – especially if you are paranoid about the life expectancy of sandboxed solutions, or someone is insisting that the app model is used. However, my feeling is that we are effectively exploiting a loophole here. I’m sure Microsoft did not particularly intend apps to be used in this way (as evidenced by difficult CSOM-based deployment model), but, possibly because of other concerns, they have robust mechanisms in place such that it isn’t proactively blocked. Me? Well, personally I feel that if the client requirements steer you towards working in the host web, then apps aren’t the right vehicle – if you’re cloud-focused, then I believe you should be using a design centred around the sandbox. Despite the "deprecated” tag I think Microsoft will have to extend the app model before they can really remove this option. In other words, I think we should understand that we are in a transition period and new development approaches will become available – but I won’t feel bad about using the best tool for the job right now.

I should add that this might not apply to product development – an area which I currently do not have to worry about. My friend Doug Ware is building an app based around the host web, and has dug far deeper into this than me. We’ve previously debated this topic in our blog posts – you can my version (which links to his posts) at SharePoint apps: working with the app web, and why you should. You’ll see that I’m effectively softening my position slightly in this post.

How to: provision a master page (or any file) to the host web

In this example, I’m choosing to provision a master page – but the approach works for other file types too. Arguably the best approach is to provision the file, initially at least, to the app web. This works because later on we’ll need to fetch the file contents from somewhere, rather than declare the entire file contents directly in a JavaScript variable (not very practical). So, we add the file to our app and use a Module element to provision it into the app web. I found that provisioning with a .txt extension was best at this stage (more on this later):

The code

Then we need a wodge of CSOM code – specifically JSOM in my case. I’m using a SharePoint-hosted app, where the code executes on page load of the app default page. In the code we have to pull a few tricks – firstly we make a jQuery GET request to the .txt file in the app web, in order to obtain the contents. N.B. This was the reason we provisioned it with a .txt extension at first – for some file types (e.g. .master, .aspx) you might find that the file contents are not what you expect. This can happen because the page could not be executed/parsed properly by SharePoint, i.e. a runtime error occurred because you effectively browsed the master page/page layout/whatever directly. This undesired behavior goes away if you are simply requesting a .txt file.

Then we use JSOM to open a connection to the host web. We do this by passing a relative URL to the host web to the SP.ClientContext constructor, instead of asking for SP.ClientContext.get_current(), which would give us context in the app web where our page is running.

Our JSOM code then uploads to the host web (via a method which can be re-used to provision any file to any path in the host web), and just for good luck we go ahead and set the master on the host web. The code below has no external dependencies, and should work fine if you paste it into your app:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

Running the app

Once our app is built and we add it to a site, the person adding has to accept the Full Control permission request:

 

And, since the model here is that our JSOM code executes when the page is loaded, our master page is indeed provisioned when the user navigates to the app. My sample code presents show some simple UI to confirm this:

The result

Now when we go back to the host web, we see that our master page has indeed been provisioned to the Master Page Gallery:

..and because we set it to be the default master page, any branding changes in this master page (such as a red bar in my case) have been applied to the site:

So, if you really want to avoid sandboxed solutions or proactively want to use apps, then you can see that these CSOM techniques can be useful. In future posts, I'll follow up with further code for other common scenarios.

Download the code

You can download the full Visual Studio project with the code I used for these two articles (on host web apps) from here – download the code.

Azure is the new SharePoint ‘layouts’ directory

SharePoint developers have always had a need to store files in a central location that can be accessed/shared from ALL SharePoint sites. This is often needed for supporting files such as images, CSS and JavaScript - usually we don’t want such files to exist in each individual site, because when updates are needed we might then have 10,000 different places to update the file. As you can imagine, there are other reasons to avoid this duplication too.

One long-established way of working around this is to deploy such files to the server filesystem, rather than into a specific SharePoint site. Commonly, the ‘_layouts’ folder under SharePoint root directory (i.e. ‘14’ for SharePoint 2010 or ‘15’ for SharePoint 2013) was used for such shared files. A second method involved deploying files as ‘uncustomized’, or ‘GhostableInLibrary’. This works by adding a list item ‘stub’ for the file into each site or library, but since the file is also deployed to the SharePoint server’s filesystem, the file contents are pulled from there. In the content database, the SQL record literally has a pointer to the location of the physical file on the SharePoint server (again, this would be somewhere under SharePoint root directory).

Although not just related to app development, I included this article in my 10 part series on apps:

1. SharePoint 2013 apps – architecture, capability and UX considerations
2. Getting started – creating lists, content types, fields etc. within a SharePoint app (provisioning)
3. Working with data in the app web, and why you should
4. Access end-user data (in the host web) from a SharePoint 2013 app
5. Rolling out SharePoint 2013 apps to the enterprise - tenant scope and PowerShell installs
6. Azure is the new SharePoint ‘_layouts’ directory [this article]
7. “Host web apps” – provisioning files (e.g. master pages) to the host web
8. “Host web apps” – provisioning fields and content types
9. Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps)
10. Working with web parts within a SharePoint app

Cloud says no

Once you start to develop sandboxed solutions or SharePoint 2013 apps, of course the technique above cannot be used. Files cannot be provisioned to the server filesystem, since the servers might not belong to you – in the case of Office 365, the servers are run by Microsoft. So what can we do?

Well, something I’m sure I’m not the first to think of is that if the production servers are connected to the internet (and cloud services like Office 365 always are), then you can just store your images, CSS and JavaScript files on some other servers (or service), and your pages will run just fine. For what it’s worth, I’ve been doing this on my blog site for quite a while – if you go looking, you’ll see I have some CSS and JavaScript being loaded ‘across the internet’ from http://sharepointnutsandbolts.azurewebsites.net rather than http://www.sharepointnutsandbolts.com.  Like me, you might find that Azure is a good choice here – it’s a solid offering with several flavors (I use the free version), and numerous options exist for getting your files there (FTP, WebDeploy, continuous deployment from TFS/Git etc.). Of course, any similar cloud service would work fine too – as indeed might some on-premises non-SharePoint servers (rather than service) you supply. Just remember that with that latter option, you’d need to take care of high-availability, backup/restore, maybe load-balancing etc. Thanks to Azure, I can let Microsoft deal with that :)

As you can imagine, we now regain the ability to store a single instance of files, even if not on the SharePoint server itself. To illustrate, here’s what the ‘default’ and ‘centralized’ approaches look like in pictures:

Default app/sandboxed development model:

Centralized development model:

When should I be considering this?

I think this broad approach is relevant to the following scenarios:

• Developing sandboxed solutions
• Developing SharePoint-hosted apps
• Developing auto-hosted apps

The scenario I’ve left out is provider-hosted apps. This is because here your supporting images, CSS and JavaScript files are stored outside of SharePoint anyway – and by default are stored as a single instance. After all, a provider-hosted app is just a non-SharePoint website at the end of the day. Auto-hosted apps are perhaps an interesting case because the app’s supporting files are deployed to Azure, but the same deal applies – they are not shared between app instances.

If you are working in these models and not currently considering this approach, perhaps you should. Yes, it can make the initial development process more complex (since your app is effectively stored in multiple places), but rolling out updates once in production will most likely be much simpler. Even if you don’t like this idea, then at least ensure you’ve considered your update mechanisms for these files.

An interesting observation (if it hadn’t occurred to you already), is that provider-hosted apps are mainly stored outside SharePoint – including their business logic. This means, amongst other things, that the creator has the option of making fairly big changes to the app without requiring the app to be upgraded or resubmitted to the Store. Certainly worth bearing in mind.

Further considerations

I think the following also need to be considered:

• Versioning
• You’ll most likely need to provide some form of versioning of your “remotely-stored” CSS and JavaScript files, for example storing them under folders named “v1”, “v2” or “1.0.0.0”, “2.0.0.0” etc. This will allow you to deploy updates for later versions of your apps, whilst maintaining the original experience for users/sites who do not upgrade
• Remember also that if you did want to make changes to existing CSS/JS files, these will most likely be cached at the browser for existing users. So you may need to do something to bust the cache (as discussed for non-app scenarios in Avoiding bugs from cached JavaScript and CSS files in SharePoint), but remember also that changes to the URL used will require changes/upgrades to the app itself unless the page is provider-hosted
• Alternative storage flavors:
• Although I’ve mainly discussed storing files in simple Azure websites, my colleague Salvatore di Fazio has also considered the use of Azure CDN –see his post Sharepoint 2013 – How to share contents between Apps

Provisioning the Content Search web part with a custom Display Template

I hit an interesting issue recently, around provisioning the Content Search web part onto a page with the AllUsersWebPart element. I should say upfront that I’m not (currently) providing an awesome “here’s some code” solution to the problem, but will suggest approaches that should work fine. Mainly I want to raise awareness of the problem at this stage – if you hit it, you’re not going crazy.

Articles in this series:

• Using the Content Search web part (and understanding SP2013 search)
• Displaying the right data in the Content Search web part in SharePoint 2013
• Provisioning the Content Search web part with a custom Display Template [this article]

If you’ve worked with the Content Search web part in SharePoint 2013, you’ll know that you can provide custom JavaScript templates (known as Display Templates) to control how items are displayed. Effectively you’re specifying the HTML which is output. Once the file is in the right place, you can select it in the ‘Display Templates’ section of the Content Search web part properties, like this:

These Display Template files are expected to live in the Master Page Gallery, and if you browse there you can see all the out-of-the-box Display Templates for various bits of SharePoint (Content Search web part, search hover panel, refinement panels etc.). There’s actually a significant difference between sites that have the publishing feature activated, and sites which do not. Specifically, if you browse in a publishing site, you’ll see two sets of files – HTML and JavaScript:

..but if you look at a site without publishing, then you only see JavaScript files:

This is one aspect of the “Design Manager” support provided in publishing sites – you get a HTML and a JS file for each template, and you can simply edit HTML file. When you do this, the associated JavaScript display template (which is going to output data in the HTML format you provided) is *generated* from your HTML file. This happens through an event receiver on the Master Page Gallery, and happens fast enough for iterative development to work out fine – as a reasonably hardcore dev, doing this bit in SharePoint Designer works fine for me. However, another option (provided by this support for HTML editing) is using a HTML editor with a design surface (Dreamweaver/Expression Web/Allaire Homesite (!)/whatever) – you can map a drive in Windows Explorer to the Master Page Gallery, to facilitate saving directly to your SharePoint dev environment from your editor.

Provisioning in a Feature - deploy the JavaScript and/or the HTML file?

So, before we get to my actual issue, we have an interesting decision point if we are working against a site with publishing in the "developer/Visual Studio” way. Do we:

• Provision the HTML file in a Feature, and then “touch” the file somehow (e.g. custom event receiver) to ensure the JavaScript file gets generated?
• Provision the JavaScript file only

After playing around with both options, my strong view on this is that you should only provision the JavaScript file. By all means, use the HTML support in the core dev phase to obtain the JavaScript file – but when you’re ready to take this to other environments, package it up in Visual Studio, and include it in your farm/sandbox WSP (N.B. none of this really applies to apps). I say this because:

• You are NOT going to need those HTML files in production – they are not used at runtime. The whole thing is purely to support the design phase!
• If you need to go back and make changes, then you can either edit the JavaScript file directly (frankly no sweat for most devs) or fire up a publishing site somewhere, use the HTML support, and then drop it back into Visual Studio when done

Touching a file –> to trigger an event receiver –> to generate a file –> which you already have in the development environment == no sense to me :) However, I would say that if you *are* working in a publishing site (e.g. in SPD), always edit the HTML file if it exists – otherwise of course, your changes to the .js file will be overwritten whenever that event receiver executes.

I use XML like the following for this – I found that the key property, ‘HtmlDesignAssociated’ defaults to 0 (zero) but when deploying the .js file only, I like to set it to zero explicitly:

With that out of the way, let’s discuss the problem I see when extending the above - to have a Content Search web part automatically added to a page with AllUsersWebPart element.

Problem - custom Display Template not linked when provisioning Content Search web part using AllUsersWebPart

[Update April 30, 2013 – Ivan Neganov has found a solution to what follows, see the comments at the bottom of the article!]

Hopefully that title was descriptive enough. So the goal is to provision the Display Template in a Feature, but also make use of it – i.e. provision some kind of page which has a pre-configured Content Search web part on it, where the configuration ‘points to’ a custom Display Template. I think this scenario will be pretty common – you’d use it in many places, such as rolling up some news articles with a particular look and feel. The process I used is what any experienced SharePoint developer would probably expect:

1. Configure the CSWP with the right settings on a page
2. Export the web part to get the XML
3. Drop into an AllUsersWebPart element in a Feature
4. Check the settings all look good
5. Test

The key bit of configuration we’re interested in is this line, but it all looks good at this stage:

<property name="ItemTemplateId" type="string">~sitecollection/_catalogs/masterpage/Display Templates/Content Web Parts/Item_Picture3Lines_COB_blue.js</property>

The overall web part XML is as follows (the “ItemTemplateId” is the last property to be listed):

The result is that seemingly all aspects of CSWP configuration take effect, except the custom Display TemplateSo. all other properties are applied, so the search query works fine and shows the correct items - but the look and feel is the out-of-the-box branding supplied by the ‘Item_Picture3Lines.js’ template (i.e. not my ‘Item_Picture3Lines_COB_blue.js’ file). What’s weird is that if you then edit the page, you find that your custom template is indeed available for selection (see first image in this article), and selecting it works just fine and the branding gets applied. At first I assumed this was some sort of issue with the format, or a ‘provisioning sequence’ issue, and played around with some ideas (breaking things into multiple Features with dependencies etc.) – none worked, so it was time to break out Reflector Pro.

After a pretty forensic ‘debugging SP2013 code’ investigation (I had a colleague trace my steps independently, thanks Hrayr!), this looks like a possible bug to me. We also tried environments before and after the March 2013 Public Update, but saw the same results. Essentially the provisioning process eventually calls into some code in the ContentBySearchWebPart class itself, and in particular a method called IsTemplateLocationValid. We see that one check appears to expect the path to start with “~sitecollection/_catalogs/masterpage” (this has been stripped off by earlier SharePoint code, despite the value being specified that way in the XML), so that one fails (click to see larger image – and sorry if you’re not into the Visual Studio ‘Dark’ theme!):

..and the next check expects SPContext to be present. However, it isn’t! I’m not sure why, given the Feature is being activated through the browser in the SharePoint ‘Activate Site Features’ page in my case, but there is definitely no SPContext. So between these two checks, I’d expect one to succeed. I can’t help but think that there is currently a minor bug in SharePoint 2013 - where SPContext is expected to be present in this provisioning process, but for whatever reason it isn’t. I say this partly because I notice that even for out-of-the-box templates, the value comes through stripped of the “~sitecollection” token:

I also tried using *just* the filename:

<property name="ItemTemplateId" type="string">Item_Picture3Lines_COB_blue.js</property>

..but then the page does not load (blank screen, HTML not output properly) and ULS shows the following runtime error:

Application error when access /SitePages/CSWP_Provisioned.aspx, Error=Cannot make a cache safe URL for "item_picture3lines_cob_provisioned.js", file not found. Please verify that the file exists under the layouts directory. 
at Microsoft.SharePoint.Utilities.SPUtility.MakeBrowserCacheSafeLayoutsUrl(String name, Boolean localizable, Int32 desiredVersion)    
…

Possible solutions

Broadly I think these are the options:

1. Manually fix-up any pages where you are adding the Content Search web part using AllUsersWebPart – i.e. edit the page by hand, select the Display Template in the web part properties, then save the page.
2. On-premises only - run some server-side code (e.g. in a Feature receiver) to fix things up. Use SPLimitedWebPartManager to update the web part properties.
3. For Office 365 – run some client-side code (JSOM or CSOM) to fix up the web part.

I don’t think any of the code would be hugely challenging to write – I might do it soon if no-one else does :) But for now, I just wanted to highlight the issue. Do you see a different behavior? Please leave a comment and let me know if so!