SP2013 host web apps: provisioning files (e.g. master pages) to the host web

Here in early summer 2013, it’s a slightly strange time in SharePoint development land. The message is that sandbox solutions are “deprecated” in SharePoint 2013 (however you choose to interpret that) and that we should build apps where possible – however, apps currently cannot do many of the things that sandboxed solutions can. One key difference between apps and sandboxed solutions, of course, is that when you provision fields/content types/files etc. in your app they are created in the app web – not the “host web”. For many requirements, this “isolated space” idea works fine. However, when you really want your artifacts to exist in the day-to-day areas of SharePoint that users go to (i.e. the team/project/My sites that become known as host webs in the app world) then you need a different approach.

If you’re no longer comfortable using the sandbox, then what can we do? Well, one option to consider is that apps CAN sometimes “provision to the host web”. This article looks at this idea, but first here's a reminder of where we are in the overall series:

1. SharePoint 2013 apps – architecture, capability and UX considerations
2. Getting started – creating lists, content types, fields etc. within a SharePoint app (provisioning)
3. Working with data in the app web, and why you should
4. Access end-user data (in the host web) from a SharePoint 2013 app
5. Rolling out SharePoint 2013 apps to the enterprise - tenant scope and PowerShell installs
6. Azure is the new SharePoint ‘_layouts’ directory
7. “Host web apps” – provisioning files (e.g. master pages) to the host web [this article]
8. “Host web apps” – provisioning fields and content types
9. Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps)
10. Working with web parts within a SharePoint app

Anyway, apps can provision to the host web IF:

• The app requests (and is granted) “Full Control” permission of the host web
  • Note this means it cannot be sold via the Office/SharePoint Store (see Validation checklist for apps for Office and SharePoint). But it can be deployed internally through the organization’s App Catalog..
• CSOM code is used for provisioning, rather than standard Feature XML

All of a sudden, apps can now be used as the vehicle for deploying “regular” SharePoint functionality – perhaps components that are used in typical collaboration solutions. The code/solution in this article is taken from my “Deep-dive on SharePoint-hosted apps” talk at the SharePoint Evolutions Conference, and this article is the first in a series of short posts on such “host web apps” (within my wider series on SP2013 apps). But first..

Should you do this? Deciding between host web apps/sandbox/farm..

I think this approach has a place – especially if you are paranoid about the life expectancy of sandboxed solutions, or someone is insisting that the app model is used. However, my feeling is that we are effectively exploiting a loophole here. I’m sure Microsoft did not particularly intend apps to be used in this way (as evidenced by difficult CSOM-based deployment model), but, possibly because of other concerns, they have robust mechanisms in place such that it isn’t proactively blocked. Me? Well, personally I feel that if the client requirements steer you towards working in the host web, then apps aren’t the right vehicle – if you’re cloud-focused, then I believe you should be using a design centred around the sandbox. Despite the "deprecated” tag I think Microsoft will have to extend the app model before they can really remove this option. In other words, I think we should understand that we are in a transition period and new development approaches will become available – but I won’t feel bad about using the best tool for the job right now.

I should add that this might not apply to product development – an area which I currently do not have to worry about. My friend Doug Ware is building an app based around the host web, and has dug far deeper into this than me. We’ve previously debated this topic in our blog posts – you can my version (which links to his posts) at SharePoint apps: working with the app web, and why you should. You’ll see that I’m effectively softening my position slightly in this post.

How to: provision a master page (or any file) to the host web

In this example, I’m choosing to provision a master page – but the approach works for other file types too. Arguably the best approach is to provision the file, initially at least, to the app web. This works because later on we’ll need to fetch the file contents from somewhere, rather than declare the entire file contents directly in a JavaScript variable (not very practical). So, we add the file to our app and use a Module element to provision it into the app web. I found that provisioning with a .txt extension was best at this stage (more on this later):

The code

Then we need a wodge of CSOM code – specifically JSOM in my case. I’m using a SharePoint-hosted app, where the code executes on page load of the app default page. In the code we have to pull a few tricks – firstly we make a jQuery GET request to the .txt file in the app web, in order to obtain the contents. N.B. This was the reason we provisioned it with a .txt extension at first – for some file types (e.g. .master, .aspx) you might find that the file contents are not what you expect. This can happen because the page could not be executed/parsed properly by SharePoint, i.e. a runtime error occurred because you effectively browsed the master page/page layout/whatever directly. This undesired behavior goes away if you are simply requesting a .txt file.

Then we use JSOM to open a connection to the host web. We do this by passing a relative URL to the host web to the SP.ClientContext constructor, instead of asking for SP.ClientContext.get_current(), which would give us context in the app web where our page is running.

Our JSOM code then uploads to the host web (via a method which can be re-used to provision any file to any path in the host web), and just for good luck we go ahead and set the master on the host web. The code below has no external dependencies, and should work fine if you paste it into your app:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

Running the app

Once our app is built and we add it to a site, the person adding has to accept the Full Control permission request:

 

And, since the model here is that our JSOM code executes when the page is loaded, our master page is indeed provisioned when the user navigates to the app. My sample code presents show some simple UI to confirm this:

The result

Now when we go back to the host web, we see that our master page has indeed been provisioned to the Master Page Gallery:

..and because we set it to be the default master page, any branding changes in this master page (such as a red bar in my case) have been applied to the site:

So, if you really want to avoid sandboxed solutions or proactively want to use apps, then you can see that these CSOM techniques can be useful. In future posts, I'll follow up with further code for other common scenarios.

Download the code

You can download the full Visual Studio project with the code I used for these two articles (on host web apps) from here – download the code.

Azure is the new SharePoint ‘layouts’ directory

SharePoint developers have always had a need to store files in a central location that can be accessed/shared from ALL SharePoint sites. This is often needed for supporting files such as images, CSS and JavaScript - usually we don’t want such files to exist in each individual site, because when updates are needed we might then have 10,000 different places to update the file. As you can imagine, there are other reasons to avoid this duplication too.

One long-established way of working around this is to deploy such files to the server filesystem, rather than into a specific SharePoint site. Commonly, the ‘_layouts’ folder under SharePoint root directory (i.e. ‘14’ for SharePoint 2010 or ‘15’ for SharePoint 2013) was used for such shared files. A second method involved deploying files as ‘uncustomized’, or ‘GhostableInLibrary’. This works by adding a list item ‘stub’ for the file into each site or library, but since the file is also deployed to the SharePoint server’s filesystem, the file contents are pulled from there. In the content database, the SQL record literally has a pointer to the location of the physical file on the SharePoint server (again, this would be somewhere under SharePoint root directory).

Although not just related to app development, I included this article in my 10 part series on apps:

1. SharePoint 2013 apps – architecture, capability and UX considerations
2. Getting started – creating lists, content types, fields etc. within a SharePoint app (provisioning)
3. Working with data in the app web, and why you should
4. Access end-user data (in the host web) from a SharePoint 2013 app
5. Rolling out SharePoint 2013 apps to the enterprise - tenant scope and PowerShell installs
6. Azure is the new SharePoint ‘_layouts’ directory [this article]
7. “Host web apps” – provisioning files (e.g. master pages) to the host web
8. “Host web apps” – provisioning fields and content types
9. Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps)
10. Working with web parts within a SharePoint app

Cloud says no

Once you start to develop sandboxed solutions or SharePoint 2013 apps, of course the technique above cannot be used. Files cannot be provisioned to the server filesystem, since the servers might not belong to you – in the case of Office 365, the servers are run by Microsoft. So what can we do?

Well, something I’m sure I’m not the first to think of is that if the production servers are connected to the internet (and cloud services like Office 365 always are), then you can just store your images, CSS and JavaScript files on some other servers (or service), and your pages will run just fine. For what it’s worth, I’ve been doing this on my blog site for quite a while – if you go looking, you’ll see I have some CSS and JavaScript being loaded ‘across the internet’ from http://sharepointnutsandbolts.azurewebsites.net rather than http://www.sharepointnutsandbolts.com.  Like me, you might find that Azure is a good choice here – it’s a solid offering with several flavors (I use the free version), and numerous options exist for getting your files there (FTP, WebDeploy, continuous deployment from TFS/Git etc.). Of course, any similar cloud service would work fine too – as indeed might some on-premises non-SharePoint servers (rather than service) you supply. Just remember that with that latter option, you’d need to take care of high-availability, backup/restore, maybe load-balancing etc. Thanks to Azure, I can let Microsoft deal with that :)

As you can imagine, we now regain the ability to store a single instance of files, even if not on the SharePoint server itself. To illustrate, here’s what the ‘default’ and ‘centralized’ approaches look like in pictures:

Default app/sandboxed development model:

Centralized development model:

When should I be considering this?

I think this broad approach is relevant to the following scenarios:

• Developing sandboxed solutions
• Developing SharePoint-hosted apps
• Developing auto-hosted apps

The scenario I’ve left out is provider-hosted apps. This is because here your supporting images, CSS and JavaScript files are stored outside of SharePoint anyway – and by default are stored as a single instance. After all, a provider-hosted app is just a non-SharePoint website at the end of the day. Auto-hosted apps are perhaps an interesting case because the app’s supporting files are deployed to Azure, but the same deal applies – they are not shared between app instances.

If you are working in these models and not currently considering this approach, perhaps you should. Yes, it can make the initial development process more complex (since your app is effectively stored in multiple places), but rolling out updates once in production will most likely be much simpler. Even if you don’t like this idea, then at least ensure you’ve considered your update mechanisms for these files.

An interesting observation (if it hadn’t occurred to you already), is that provider-hosted apps are mainly stored outside SharePoint – including their business logic. This means, amongst other things, that the creator has the option of making fairly big changes to the app without requiring the app to be upgraded or resubmitted to the Store. Certainly worth bearing in mind.

Further considerations

I think the following also need to be considered:

• Versioning
• You’ll most likely need to provide some form of versioning of your “remotely-stored” CSS and JavaScript files, for example storing them under folders named “v1”, “v2” or “1.0.0.0”, “2.0.0.0” etc. This will allow you to deploy updates for later versions of your apps, whilst maintaining the original experience for users/sites who do not upgrade
• Remember also that if you did want to make changes to existing CSS/JS files, these will most likely be cached at the browser for existing users. So you may need to do something to bust the cache (as discussed for non-app scenarios in Avoiding bugs from cached JavaScript and CSS files in SharePoint), but remember also that changes to the URL used will require changes/upgrades to the app itself unless the page is provider-hosted
• Alternative storage flavors:
• Although I’ve mainly discussed storing files in simple Azure websites, my colleague Salvatore di Fazio has also considered the use of Azure CDN –see his post Sharepoint 2013 – How to share contents between Apps

Provisioning the Content Search web part with a custom Display Template

I hit an interesting issue recently, around provisioning the Content Search web part onto a page with the AllUsersWebPart element. I should say upfront that I’m not (currently) providing an awesome “here’s some code” solution to the problem, but will suggest approaches that should work fine. Mainly I want to raise awareness of the problem at this stage – if you hit it, you’re not going crazy.

Articles in this series:

• Using the Content Search web part (and understanding SP2013 search)
• Displaying the right data in the Content Search web part in SharePoint 2013
• Provisioning the Content Search web part with a custom Display Template [this article]

If you’ve worked with the Content Search web part in SharePoint 2013, you’ll know that you can provide custom JavaScript templates (known as Display Templates) to control how items are displayed. Effectively you’re specifying the HTML which is output. Once the file is in the right place, you can select it in the ‘Display Templates’ section of the Content Search web part properties, like this:

These Display Template files are expected to live in the Master Page Gallery, and if you browse there you can see all the out-of-the-box Display Templates for various bits of SharePoint (Content Search web part, search hover panel, refinement panels etc.). There’s actually a significant difference between sites that have the publishing feature activated, and sites which do not. Specifically, if you browse in a publishing site, you’ll see two sets of files – HTML and JavaScript:

..but if you look at a site without publishing, then you only see JavaScript files:

This is one aspect of the “Design Manager” support provided in publishing sites – you get a HTML and a JS file for each template, and you can simply edit HTML file. When you do this, the associated JavaScript display template (which is going to output data in the HTML format you provided) is *generated* from your HTML file. This happens through an event receiver on the Master Page Gallery, and happens fast enough for iterative development to work out fine – as a reasonably hardcore dev, doing this bit in SharePoint Designer works fine for me. However, another option (provided by this support for HTML editing) is using a HTML editor with a design surface (Dreamweaver/Expression Web/Allaire Homesite (!)/whatever) – you can map a drive in Windows Explorer to the Master Page Gallery, to facilitate saving directly to your SharePoint dev environment from your editor.

Provisioning in a Feature - deploy the JavaScript and/or the HTML file?

So, before we get to my actual issue, we have an interesting decision point if we are working against a site with publishing in the "developer/Visual Studio” way. Do we:

• Provision the HTML file in a Feature, and then “touch” the file somehow (e.g. custom event receiver) to ensure the JavaScript file gets generated?
• Provision the JavaScript file only

After playing around with both options, my strong view on this is that you should only provision the JavaScript file. By all means, use the HTML support in the core dev phase to obtain the JavaScript file – but when you’re ready to take this to other environments, package it up in Visual Studio, and include it in your farm/sandbox WSP (N.B. none of this really applies to apps). I say this because:

• You are NOT going to need those HTML files in production – they are not used at runtime. The whole thing is purely to support the design phase!
• If you need to go back and make changes, then you can either edit the JavaScript file directly (frankly no sweat for most devs) or fire up a publishing site somewhere, use the HTML support, and then drop it back into Visual Studio when done

Touching a file –> to trigger an event receiver –> to generate a file –> which you already have in the development environment == no sense to me :) However, I would say that if you *are* working in a publishing site (e.g. in SPD), always edit the HTML file if it exists – otherwise of course, your changes to the .js file will be overwritten whenever that event receiver executes.

I use XML like the following for this – I found that the key property, ‘HtmlDesignAssociated’ defaults to 0 (zero) but when deploying the .js file only, I like to set it to zero explicitly:

With that out of the way, let’s discuss the problem I see when extending the above - to have a Content Search web part automatically added to a page with AllUsersWebPart element.

Problem - custom Display Template not linked when provisioning Content Search web part using AllUsersWebPart

[Update April 30, 2013 – Ivan Neganov has found a solution to what follows, see the comments at the bottom of the article!]

Hopefully that title was descriptive enough. So the goal is to provision the Display Template in a Feature, but also make use of it – i.e. provision some kind of page which has a pre-configured Content Search web part on it, where the configuration ‘points to’ a custom Display Template. I think this scenario will be pretty common – you’d use it in many places, such as rolling up some news articles with a particular look and feel. The process I used is what any experienced SharePoint developer would probably expect:

1. Configure the CSWP with the right settings on a page
2. Export the web part to get the XML
3. Drop into an AllUsersWebPart element in a Feature
4. Check the settings all look good
5. Test

The key bit of configuration we’re interested in is this line, but it all looks good at this stage:

<property name="ItemTemplateId" type="string">~sitecollection/_catalogs/masterpage/Display Templates/Content Web Parts/Item_Picture3Lines_COB_blue.js</property>

The overall web part XML is as follows (the “ItemTemplateId” is the last property to be listed):

The result is that seemingly all aspects of CSWP configuration take effect, except the custom Display TemplateSo. all other properties are applied, so the search query works fine and shows the correct items - but the look and feel is the out-of-the-box branding supplied by the ‘Item_Picture3Lines.js’ template (i.e. not my ‘Item_Picture3Lines_COB_blue.js’ file). What’s weird is that if you then edit the page, you find that your custom template is indeed available for selection (see first image in this article), and selecting it works just fine and the branding gets applied. At first I assumed this was some sort of issue with the format, or a ‘provisioning sequence’ issue, and played around with some ideas (breaking things into multiple Features with dependencies etc.) – none worked, so it was time to break out Reflector Pro.

After a pretty forensic ‘debugging SP2013 code’ investigation (I had a colleague trace my steps independently, thanks Hrayr!), this looks like a possible bug to me. We also tried environments before and after the March 2013 Public Update, but saw the same results. Essentially the provisioning process eventually calls into some code in the ContentBySearchWebPart class itself, and in particular a method called IsTemplateLocationValid. We see that one check appears to expect the path to start with “~sitecollection/_catalogs/masterpage” (this has been stripped off by earlier SharePoint code, despite the value being specified that way in the XML), so that one fails (click to see larger image – and sorry if you’re not into the Visual Studio ‘Dark’ theme!):

..and the next check expects SPContext to be present. However, it isn’t! I’m not sure why, given the Feature is being activated through the browser in the SharePoint ‘Activate Site Features’ page in my case, but there is definitely no SPContext. So between these two checks, I’d expect one to succeed. I can’t help but think that there is currently a minor bug in SharePoint 2013 - where SPContext is expected to be present in this provisioning process, but for whatever reason it isn’t. I say this partly because I notice that even for out-of-the-box templates, the value comes through stripped of the “~sitecollection” token:

I also tried using *just* the filename:

<property name="ItemTemplateId" type="string">Item_Picture3Lines_COB_blue.js</property>

..but then the page does not load (blank screen, HTML not output properly) and ULS shows the following runtime error:

Application error when access /SitePages/CSWP_Provisioned.aspx, Error=Cannot make a cache safe URL for "item_picture3lines_cob_provisioned.js", file not found. Please verify that the file exists under the layouts directory. 
at Microsoft.SharePoint.Utilities.SPUtility.MakeBrowserCacheSafeLayoutsUrl(String name, Boolean localizable, Int32 desiredVersion)    
…

Possible solutions

Broadly I think these are the options:

1. Manually fix-up any pages where you are adding the Content Search web part using AllUsersWebPart – i.e. edit the page by hand, select the Display Template in the web part properties, then save the page.
2. On-premises only - run some server-side code (e.g. in a Feature receiver) to fix things up. Use SPLimitedWebPartManager to update the web part properties.
3. For Office 365 – run some client-side code (JSOM or CSOM) to fix up the web part.

I don’t think any of the code would be hugely challenging to write – I might do it soon if no-one else does :) But for now, I just wanted to highlight the issue. Do you see a different behavior? Please leave a comment and let me know if so!

Slide decks for “SharePoint-hosted apps deep-dive” and “Customising SP2013 with JavaScript” published

A brief post to mention that I’ve now published my slide decks for the two talks I delivered at the SharePoint Evolutions Conference in April 2013. I had a great time at the conference (as always!), and it was awesome to have so many people come to the sessions – thanks to all who came, I appreciate it. Just for a bit of atmosphere, here’s the attendee perspective in one of these conference talks (not mine – I was at the back, taking the photo):

As always, technical talks are made richer by the demos and the decks alone do not convey this – I’ve added some screenshots for one talk (the JavaScript session), but I haven’t for the apps talk as I’m not sure they would add as much. Anyway, here’s the summary and links:

Customizing the SharePoint 2013 user interface with JavaScript

Covers several approaches for user interface customization in SP2013 - using JSLink to customize a list and/or view, creating custom Display Templates for the Content Search web part, and different approaches for customizing the search hover panel.

Slideshare link - http://www.slideshare.net/chrisobrien/customizing-the-sharepoint-2013-user-interface-with-javascript-chris-obrien

Deep dive into SharePoint 2013 hosted apps

Covers key aspects of SharePoint 2013 apps, with a focus on SharePoint-hosted apps. Includes detail on app parts, using web parts within an app, configuring SSL, troubleshooting apps and possible reasons to move away from a SharePoint-hosted app to a cloud app. Also covers "high-privilege" apps which provision to the host web.

Slideshare link - http://www.slideshare.net/chrisobrien/deep-dive-into-sharepoint-2013-hosted-apps-chris-obrien

 

Other notes

In upcoming posts I’ll be publishing more detail on this content, including the source code I used for my demos. Stay tuned.

Rolling out SharePoint 2013 apps to the enterprise - tenant scope and PowerShell installs

Let’s say you’ve decided to make some custom functionality available to your users as a SharePoint 2013 app. You’ve decided that this will be published to the appropriate internal SharePoint App Catalog (as opposed to being available in the public SharePoint Store). In a large company, it’s probably not realistic to expect all users to install such internal apps, even if we want to make the functionality available to everyone.

The answer *might* be to use one of the options for deploying the app at “tenant scope”. This works for both on-premises deployments and Office 365/SharePoint Online. This article looks at these options, but just as a reminder of where we are in the overall series, here's the table of contents:

1. SharePoint 2013 apps – architecture, capability and UX considerations
2. Getting started – creating lists, content types, fields etc. within a SharePoint app (provisioning)
3. Working with data in the app web, and why you should
4. Access end-user data (in the host web) from a SharePoint 2013 app
5. Rolling out SharePoint 2013 apps to the enterprise - tenant scope and PowerShell installs [this article]
6. Azure is the new SharePoint ‘_layouts’ directory
7. “Host web apps” – provisioning files (e.g. master pages) to the host web
8. “Host web apps” – provisioning fields and content types
9. Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps)
10. Working with web parts within a SharePoint app

Microsoft even provide a nice admin interface for tenant-scope installs, which offers the following deployment options:

• Deployment to named site collections
• Deployment to all sites under a particular managed path
• Deployment to all sites created from a certain site template

I said *might*. When deploying apps in this way, there is a crucial detail to be aware of – something which could either be highly-desirable or a deal-breaker. Essentially, all installed instances of the app *share* a single app web. This means any lists or libraries you provision as part of your app, are shared amongst all the webs where the app is installed. As I say, this could be exactly how you want your app to operate, or completely against it’s design.

The process for using this model is as follows:

1. Upload app to App Catalog.
2. Install the app to the App Catalog site collection – yes, for this scenario we actually have to install the app there (for reasons which will become clear later).
3. Go to the Site Contents page and find the app. On the callout menu here (and only here), you’ll have an action labelled ‘DEPLOYMENT’:
   
    
4. Click the ‘DEPLOYMENT’ button to go to the admin page.

On this page, I see the 3 options for deployment – individual site collections:

..or deploying to all sites under a Managed Path, or all sites created from a certain template:

Once you’ve selected appropriate options and hit OK, after a few minutes time you’ll find your app has landed in the target locations:

As you can probably guess, it’s a timer job which performs the deployment. This is the App Installation Service timer job, set to run every 5 mins by default.

Important aspects of tenant-scoped app installations

As I highlighted before, all app instances share a single app web. The way this works is that the instance which is installed to the App Catalog site is used for all instances – effectively the user is always redirected to this URL on the app domain, regardless of which site they entered the app from:

There are also important by-products of this:

• Users in sites where the app is installed cannot remove the app – regardless of their permissions there
  • This makes sense because they would effectively be uninstalling the single instance used by everyone
• Individual “usages” of the app do not get reported by Get-SPAppInstance – only the instance in the App Catalog site gets returned (N.B. applicable to on-premises SP2013 only)

Any changes to the app (e.g. app upgrades, app removal) need to be managed at the instance which is installed to the App Catalog site.

But I want something else - to roll out my app in bulk, but it NOT to be “single instance”

Then in this case, PowerShell is your friend – this is a two step process to install a new app:

• Use Import-SPAppPackage to initially install the app to the site collection, then..
• Use Install-SPApp to install to individual webs

For a full list of cmdlets around SharePoint 2013 apps, see App Management Service cmdlets in SharePoint 2013 on TechNet. Corey Roth also has a post on some of these cmdlets.

If you’re on Office 365 (with it’s limited set of PowerShell cmdlets, as of April 2013), you’ll need to accomplish the same thing using one of the client APIs. I’m starting to form the view that .NET CSOM code “wrapped” with PowerShell is a good way to do this – since it can then be integrated with TFS automated builds etc.

Happy app installing!

The SharePoint Evolution Conference 2013 is near – details on my talks

I’m a bit late in posting this, but if you’re still considering whether to attend this conference (London, April 15-17th 2013) then I highly recommend it. As usual, the speaker line-up looks awesome and after missing out last year (due to moving house) I’m really looking forward to it.

I’m giving two talks centred around SharePoint 2013:

DEV 203 - Deep dive into SharePoint-hosted apps (Monday, 1:30pm)

Even though apps can be outside of SharePoint, often it makes sense to leverage what's inside. This session takes a close look at how to use lists, content types and pages within a SharePoint-hosted app - whether there are additional remote components or not. We'll also look at advanced approaches for using a custom site definition, and explain why some web parts can be used and some cannot. In addition to working with the app web, we'll discuss pros and cons of "high-privilege" apps which provision into the host web - potentially bringing collaboration-based solutions closer to the app world.

P&M 306 - Using JavaScript templates to customise the SharePoint user interface (Tuesday, 9am)

Developers often have a need to amend the HTML rendering of SharePoint, especially around lists and list forms, but in previous versions of SharePoint this was not always easy. Many developers resorted to using JavaScript to manipulate the page after it had loaded, even though this was potentially unreliable and unsupported. SharePoint 2013 solves this problem by providing the ability to specify a JavaScript 'template override' for several user interface elements. Even better, these can be applied globally or locally as needed. Come and learn how to transform the SharePoint interface the supported way!

If you need more information on the conference, point your browser to http://www.sharepointevolutionconference.com

Displaying the right data in the Content Search web part in SharePoint 2013

Articles in this series:

• Using the Content Search web part (and understanding SP2013 search)
• Displaying the right data in the Content Search web part in SharePoint 2013 (this article)
• Provisioning the Content Search web part with a custom Display Template

So, let’s say you’re working with the Content Search web part in SP2013 and you’ve got it displaying the correct items. (If not, see my big post on Using the Content Search web part (and understanding search). In the case of this article, I’m searching for items I’ve created and tagged with “SharePoint”. Whatever the query, you’ll most likely be confronted by something like this:

Your client will be thrilled with something so aesthetically pleasing! By the way, the above search is for “documents in my SharePoint farm tagged with SharePoint” -  having a thumbnail image shown might not really be needed for documents, but just for fun we’ll talk about both options. In a later post, I’ll cover using the CSWP to roll up some news pages (publishing pages) – these do tend to benefit from thumbnails, but the good news is there’s less work to get them there.

Anyway, if we break down the issues we’d like to fix, we would say:

• An image is rendered for each item, but one is not found (so a default image is used)
• The fields may not be correct – certainly for the 2nd and 3rd items, a fairly useless value is being fetched from the Office document properties. This is being displayed in the absence of a ‘Description’ field on the list item (N.B. this is actually a great ‘fallback’ feature of SP2013 search – Office doc properties get pulled into the property bag of the list item, and indexed by search if no other values take precedence [via Managed Property mappings]), but in this case, it’s some irrelevant data about who created the original PowerPoint template for the conference I presented at:
   
• The overall formatting (HTML and CSS, especially colors) do not match our desired rendering

There are several settings in the CSWP web part properties at work here:

Thing	Screenshot	Notes
Control Display Template		This determines the overall style of the presentation – it’s effectively the “outer HTML” that is output. I’d say List will be the most commonly-used value, but it’s great to see List with Paging come for free. Slideshow is an image rotator.
Item Display Template		This dictates the style applied to each item displayed i.e. “the HTML for each item”. Later when we create a new Item Display Template, we’ll see this become available for selection in this dropdown.
Property Mappings		These are very much like ‘slots’ in the Content Query web part – they are a feature of the Item Display Template which is selected (specified in JavaScript). Item templates use generic  properties (e.g. ‘Picture URL’, ‘Line 1’ etc.), and you can ‘map’ some specific fields which exist on your list item. The image shows the default mappings. This means that if you only need to get a different field displayed in an existing slot, you can do this without modifying XSL (CQWP) or JavaScript (CSWP). If you needed 10 slots rather than 5, editing the Item Display Template’s JavaScript controls this. Note that can specify multiple values specified by a semi-colon (as shown).

 

Choosing the Item Display Template

As you change the selected Item Display Template in the dropdown, the Property Mappings section in the CSWP tool pane will change accordingly, allowing the mapping for each slot. Once configured appropriately (see next section), “Picture on left, 3 lines on right” can have 3 lines of information displayed:

Whereas “Two lines” of course, will give you no picture and just two lines:

We’d probably say that “Two lines” works just fine for documents – unless you do happen to want thumbnails (e.g. for some key policies/procedures/CAD files/PDFs/whatever). Whilst thumbnails kinda happen by default for publishing pages, the process for other items is useful for illustrating the ‘Property Mappings’ aspect – first let’s cover the possibilities for working with these mappings. 

Deep dive - getting the right data coming through

If you haven’t guessed already, those property mappings such as ‘PublishingImage’, ‘Path’, ‘Title’ and ‘Description’ above are Managed Properties in the search service app. If you are rolling up content from multiple locations (Content by *Search* remember!), consider that these mappings allow you to show results from across your farm/tenant  even if you don’t have standardized content types with the same fields everywhere. For example, perhaps ‘Description’ could really come from a custom field named ‘Project Description’ in one place but ‘Task Description’ for other items.

The use of Managed Properties means you might need to go down a different route to get your desired values coming through:

Option 1 – no work needed, the CSWP automatically displayed my data where I want it!

Awesome! You’ll see this for the ‘Title’ field and some others (e.g. roll-up images in publishing pages). This is happening because for this ‘slot’ in the selected item display template, the default Property Mapping contains a Managed Property which points to the field on the list item/content type which holds the data. Perfect. If this isn’t happening and you think it should be, check that you have populated some content and it has been successfully crawled – you won’t get anywhere until it has.

Option 2 – use an existing Managed Property and site column, but edit the CSWP properties for a particular slot (to point to this Managed Property). You would do this when the CSWP isn’t looking at this Managed Property by default  for this slot:

It’s a big old dropdown, but all you have to do is select the existing Managed Property you want:

Additionally you can add to the semi-colon separated list of Managed Properties, by typing into the box.

Option 3 - use a new site column but add it to an existing Managed Property:

Rare maybe, but you might want to do this if you want the same mapping to be use in different places e.g. multiple CSWP instances, or CSWP and the search results page etc. The process would be:

1. Add a new column to your list/library/content type. Populate some content and perform a full/incremental crawl.
   1. Find an existing, somewhat generic, Managed Property which represents the thing you are working with – this could be something like ‘Title’ or ‘Description’. Edit the Managed Property so that the corresponding Crawled Property for your field (e.g. ‘ows_COBDescription’ for a field named ‘COBDescription’) is added to the list of Crawled Properties. 
2. Ensure your Crawled Property is higher in the mappings list than others (if using the ‘Include content from  the first crawled property that is not empty..’ option. As an illustration:
          
   

Option 4 – use an entirely custom Managed Property and site column:

This might be the best approach if there are no appropriate fields in existence, or you really don’t want to pollute any other CSWP instances and/or the search results page with these settings. I used this approach to add thumbnails to my documents when using “Picture on left, 3 lines on right”. The process would be:

1. Add a new column to your list/library/content type. Populate some content and perform a full/incremental crawl.
2. Create a new Managed Property, and map it to the corresponding Crawled Property for your field (e.g. ‘ows_COBThumbnailUrl’ for a field named ‘COBThumbnailUrl’).
3. Edit the Content Search web part properties, so that the Property Mappings for the desired slot contain the name of the Managed Property – here I’ve added my Managed Property (‘COBThumbnailUrlManaged’) to the ‘Picture URL’ slot:
   
    

In general, the deciding factors will be things like:

• Do I want this change to take effect everywhere? (editing an existing Managed Property [i.e. adding a new Crawled Property to it] could mean that other Content Search web parts/search results show data from an instance of this column)
• Is there an existing field and/or Managed Property which could represent this data?

“CAD files/documents with thumbnails (in team sites)” example

Recently I had to implement thumbnails for CAD files for a client demo – to appear in both CSWP results and the search hover panel (I’ll talk about customizing the search hover panel another day). The thumbnails aren’t automatically generated, but could be in the future. At it’s core, there’s an additional column which contains a hyperlink to the thumbnail image (e.g. in a Site Assets library).

I chose option 3 above in this case – since I don’t need to ensure I’m scoping only to one CSWP instance or anything.

Here are the specific changes I made to get my thumbnails and descriptions to come through to the CSWP:

• Added two columns to my content type, to contain the thumbnail URL and description
• Populated these fields for each item (for the thumbnails, as mentioned the images themselves live in the local Site Assets library)
• Item thumbnail:
• I created a site column (‘COBThumbnailUrl’), and mapped this to the ‘PictureThumbnailURL’ Managed Property
• Item description (for display in the ‘Line 2’ value):
• In this case I re-used the OOTB ‘Description’ Managed Property, but added a new column ‘COBDescription’ and included this in the Crawled Properties mapped to that Managed Property
• (N.B. There are a bunch of OOTB fields named ‘Description’ in a team site, but most are in the ‘Hidden’ group and didn’t actually seem suitable for re-use)

Once the thumbnail is against that file, it’s fairly easy to surface it in search hover panels (as well as CSWP rollups):

“Publishing pages with thumbnails” example

If the items you’re rolling up are publishing pages (e.g. news pages), then things are slightly simpler – this is because the default mappings of, say, “Picture on left, 3 lines on right” work better. Assuming you’ve configured the query correctly and the pages have a roll-up image set (e.g. a rendition of an existing image), then you’ll probably get thumbnails but no descriptions by default:

In this case, you might notice that my thumbnails aren’t high quality. This is because they are simply generated using the SP2013 publishing framework’s image renditions capability (i.e. a 100 x 100 version of the much larger image displayed on the page itself) – if I produced special images for thumbnails, the result would be higher quality (but more work).

We have no item descriptions so far, only titles. However, if we edit the Property Mappings for “Line 2” and “Line 3”, we can start to bring in other data – here I’m selecting the article byline and also the property which lists which terms this item has been tagged with:

 
..and now we get the those details displayed by the Display Template:

Next considerations – modifying the HTML/CSS/JS, and packaging/deployment

So far, we haven’t worried about changing the actual HTML which is output. My next articles look at this, for both publishing and non-publishing scenarios (the process is slightly different).

There are also a couple of considerations for working the “developer way” and provisioning these artifacts through a WSP. I’ll also discuss these next time.

SharePoint/TFS 2012 ALM – updated SharePoint CI scripts and assembly versioning

Over the past couple of years I’ve been a big advocate of continuous integration (automated builds) for SharePoint projects, and I’ve previously published some resources to help people get started with this. Specifically I have:

• A custom TFS build workflow – this is for use with TFS Build. It integrates with some PowerShell scripts I also supply, so that WSPs can be deployed automatically as part of the build. This is my “SharePoint CI starter pack”.
• Assembly versioning workflow activity – this is an optional custom workflow activity which will increment the version numbers of your assemblies as they are built in the right way for SharePoint. To simplify things I supply one workflow with this activity dropped in and configured, and one without.

Because Microsoft made some changes between TFS 2010 and TFS 2012, these things do not “just work” for TFS 2012 unfortunately. In fact, migrating a build workflow from TFS 2010 to TFS 2012 is a bit of a nightmare. But, I’ve now updated these with new releases for TFS 2012 so things should be a bit easier for if you work with this stuff.

A while back someone pinged me on Twitter asking for an update, and then some Microsoft folks got in touch to say “can we take ownership of your stuff, or could you update it?” and then last week I was in meetings about a TFS 2012 upgrade at my place of work (Content and Code), and since our larger SharePoint projects use this stuff I knew I had to bump this up the priority list :)

Where to find this stuff

I have two projects on Codeplex:

• SharePoint CI starter pack – http://sharepointci.codeplex.com
• TFS simple assembly versioning – http://tfssimpleversioning.codeplex.com

Both new releases are labelled “for TFS 2012”, and instructions can be found on each site. Briefly, you’ll need to download both if you want to perform versioning within your automated builds – if not, you can just use the CI starter pack, selecting the “no versioning” workflow in there.

Release notes

I’ve added release notes on Codeplex, but just to summarize here:-

• TFS simple assembly versioning
• This is a brand new workflow activity (with a new name - COB.CI.Workflow.AssemblyVersioning), due to the incompatibility between TFS 2010 and TFS 2012. As I say above, you can either drop this into your own (custom) TFS build workflow to have your assemblies versioning automatically, or use my workflows from my CI starter pack.
• This has been fairly well tested and is robust.
• SharePoint CI starter pack
• As before, this contains 2 workflows (with versioning/without versioning) and a set of PowerShell scripts which deploy WSPs etc. If you’re an existing user, you should add the 2 workflow XAML files to TFS, and then edit your existing build definition(s) to use one of these workflows.
• The TFS/PowerShell integration to deploy WSPs as part of the build has been well tested. However, at the time of writing I haven’t yet tested including automated tests (unit tests/UI tests) into the build – at this point, I just want to get something out there quickly to support any TFS upgrades of existing users.

Incompatibility issues around TFS 2010/TFS 2012

For completeness, I just wanted to show the errors you’ll see if you upgrade to TFS 2012 and are using my resources (i.e. attempting to use TFS 2010 workflow stuff in TFS 2012). In fact, I think you can expect to see errors if you open any build workflow after the upgrade – whether you use my stuff or not. You’ll see something like:

• System.IO.FileNotFoundException: Could not load file or assembly ‘Microsoft.Team.Foundation.Build.Client, PublicKeyToken=b03f5f7f11d50a3a’ or one of its dependencies.
• System.IO.FileNotFoundException: Could not load file or assembly ‘Microsoft.Team.Foundation.Build.Workflow, PublicKeyToken=b03f5f7f11d50a3a’ or one of its dependencies.
• Various other error messages all related to versioning.

Yes, it seems many TFS 2010 build workflows appear to have version numbers stamped into the XAML. I understand this is due to an unfortunate bug in VS 2010, specifically Team Explorer - this KB article has more details and a hotfix. However, chances are that (like mine) your workflows already have ended up with the version numbers in them by now. So, it’s certainly more painful than it should be.

You have two options to make your build workflow run in TFS 2012:

• Rebuild it from scratch in VS 2012/TFS 2012 – that’s what I did in the end with my CI starter pack workflows
• Attempt to clean the XAML:
• I tried this by hand several times and failed. The workflow failed to display in the designer, and there is precious little diagnostic information.
• Another option could be Jason Prickett’s “XAML cleaner” tool to remove the versioned namespaces in the file. I tried this, but for some reason it didn’t give me work on my workflows – you might have more luck.

The Jason Prickett post linked above also has other variations of error messages you might see.

Related error messages you might see if you use my resources:

Again for completeness, here are some other errors you might encounter if you’re using my resources but are “doing it wrong” according to TFS 2012. This is what you might see if you open a workflow with my (TFS 2010) assembly versioning activity after upgrading to TFS 2012/VS 2012:

You’ll need to fix the workflow to use the latest version of the activity if so.

And here’s what you’ll see if you try to drop my old (TFS 2010 version) assembly versioning activity into a TFS 2012 workflow:

Conclusion

Upgrading to TFS 2012 brings some pain if you do automated builds, whether in relation to SharePoint or not. For any SharePoint folks who use my resources, I’ve upgraded these to hopefully save you some pain. Please leave a comment on the Codeplex site if you hit any issues with them.

Using JSLink to change the UI of a SharePoint list/view

Changing the user interface of certain bits of SharePoint has always been somewhat challenging – at least, if we want to avoid the SharePoint Designer route and produce something which is packaged, deployable as a WSP and repeatable across many sites. Not to mention source-controlled. A common area of customization is the user interface around SharePoint lists - when it comes to this, there are a few “flavours” of common requests:

• Custom list forms – e.g. a more “designed” form for adding and/or editing items
• Custom behaviour for a certain field - e.g. a “UK address” field which allows postcode lookup
• Custom rendering of a list in display mode (e.g. the ‘All items’ view) – a good example of this is conditional formatting e.g. “Add a red background to the row if the value for ‘Days overdue’ is over 5”

As you might guess from the article title, it’s the third scenario that this article focuses on, for SharePoint 2013 specifically.

Earlier versions of SharePoint gave us a couple of approaches for fulfilling some of these requirements. We could create a custom field control (item number 2 above) or modify the XSLT of an XsltListView perhaps (number 3 above) – a method that was new in SP2010 of course, since we previously had CAML rendering for list views (yuck). SharePoint 2013 brings a new JavaScript-based approach for modifying the UI of fields and lists, and many more things too. You’ll often see this referred to as “JSLink”, because many things in SP2013 have a new JSLink property which you use to point to your JavaScript file. Here’s a list of SharePoint objects which you can modify in this way:

Lots of interesting possibilities there – all the field types are obviously represented, but binding UI changes to a:

• content type
• form
• view
• list view web part instance (rather than to the list/view itself)

..could all be extremely useful.

[As an aside, if you came to this article looking for a solution to the 1st scenario above (modifying forms), you’ll probably be interested that SPForm has a JSLink property. However I imagine there’s still a place for providing entirely custom forms as opposed to making relatively small changes with JavaScript. The old approaches still apply here - modifying list forms is easy in SPD, but requires a bit more thought in the Visual Studio world. In that case, we’d probably want to NOT edit the original .aspx, but instead provide a different .aspx file and update the SPList.NewFormUrl and/or SPList.EditFormUrl properties to point to it (e.g. in a Feature receiver).]

Implementing changes to list rendering with JSLink

For a recent SharePoint talk, I wanted a fairly dramatic example of changing the UI of a list (when looking at list items). So, my example showed changing the list from this:

..to this:

There are some slightly nasty scrollbars in these images, but that’s just because I’ve reduced the size of the window for my screenshots - normally the accordion looks great. As you might imagine, I get a nice sliding “expand and contract” experience when I select different items:

If you’ve worked with it before, you’ll immediately recognise this as the accordion from jQuery UI. In many ways, it’s a nice example, since although YOU are unlikely to need/want to use jQuery accordion for your “customizing a list” needs, it uses custom CSS, images and JavaScript to provide rendering which you probably ARE going to need. So, I’ll walk through most aspects of this process, but as you’ll see only around 20% of the work relates to the JSLink stuff – the rest is just scaffolding. We’ll cover the JSLink specifics first.

The full Visual Studio project can be downloaded at the end of this article.

The important bit (part 1)– how to hook up your custom rendering with JSLink (e.g. call into jQuery UI accordion)

The first thing to say is that when we want to change the rendering of a SharePoint list, it’s actually individual views (SPView) we’ll be working with. The SPList class does NOT have a JSLink property, which makes sense given this is all about presentation. Getting SharePoint to “see” your custom rendering will probably depend on what you’re doing:

Scenario	Approach
Creating a new list	Specify the path to your .js file in the schema.xml file for the list (specifically within the declaration for the view)
Modifying an existing list	In code, update the SPLink property (i.e. SPView.JSLink) to provide the path to your .js file (using PowerShell/server-side/client-side API as appropriate)

This bit only ensures your .js file is referenced by the page. You also need to make sure it has the right contents to register your display templates – we’ll come to that in a second.

In my case, I’m creating a new list – it’s worth noting that I’m assigning a list type ID of “11000” – we’ll use this later with respect to JSLink:

When developing for SharePoint 2013 onwards, when a list is created in Visual Studio the JSLink element for any views will contain “clienttemplates.js”:

..but we should change that to our custom JavaScript file which has our display template implementation (you’ll see me deploy this later):

Our file will then be loaded when this view is requested (i.e. SharePoint will add the .js file to the page). But that’s not enough – we now have to think about what JavaScript is needed to actually register the templates. This is done by specifying two properties of the list views to match:

• OPTIONAL - the BaseViewID property (e.g. BaseViewID=1 for a standard view like “All items”, but you could use another ID for a custom view)
• The ListTemplateType property (for a new list, you’ll be specifying a unique integer value e.g. 10000)

So, it’s usually a combination of THREE controls overall which dictate how rendering is applied to a list view – the JSLink path, BaseViewID and ListTemplateType. Although BaseViewID appears to be optional, it seems sensible to set it to avoid unforeseen problems with Explorer View/Datasheet View etc. So for a given list, if you wanted one view to have custom rendering and one to have default rendering (even though they share a BaseViewID), simply ensure the JSLink property for the default one is NOT set to your custom .js file.

I also note that it appears possible to specify multiple values in a JSLink path – I haven’t tried this, but I see things like <JSLink>mquery.js|contentfollowing.js</JSLink> (note the pipe character) within out-of-the-box files under the 15 folder.

The important bit (part 2)– what your JavaScript should look like:

Here’s my full AccordionListView.js file specified in the JSLink property for my view – I supply a header and footer and then a JavaScript method (function pointer) to execute for each list item. Notice some context gets passed to this function, including details of the list item:

As you might be able to infer, the jQuery UI accordion expects a H3 and div element for each item – so that’s what my template does, in addition to actually calling the accordion() method.

An issue with JSLink and the Minimal Download Strategy (MDS)?

Continuing my recent tradition of discovering slightly strange behaviour around what I’m writing about, this week is no exception. In my testing, I noticed that if a list has multiple views and the user switches between them, the client templates specified by JSLink do not get applied if MDS is enabled (even though the debugger shows they are called). I’m hoping that I’m doing something wrong, but I can’t rule out a bug in SharePoint 2013’s MDS framework at this stage.

If anyone sees this/has any info, please leave a comment :)

Integrating CSS/JS (such as jQuery UI) into the solution

OK, so if all you wanted to know about was the JSLink aspect, we’re now done covering that. Hopefully that was useful. The remaining bits of this article will cover the “scaffolding” aspects of my particular jQuery UI accordion example – integrating jQuery/jQuery UI, and so on.

My first step was to go to the jQuery UI Download Builder and download the files (plus jQuery too if you don’t have it already). If you haven’t done this before, you can basically configure some options (e.g. styles, behaviors, jQuery UI widgets to use) to build a package for download – this means you get a smaller package than if you were opted for all styles and all jQuery UI components. You’ll therefore have lower page weight and better performance, than if you were using the full set. However, I had an issue where my solution would only work with the full jQuery UI file rather than the one I “built” – if this was production code I’d stop and resolve this, but for demoware I was OK with it.

When you get your download and crack it open, you’ll have some CSS and JavaScript files:

I chose a grey-ish theme called “overcast”, and within the CSS folder come a stack of images – we’ll need to integrate all of these files into our VS project:

In my project, I created a “Site Assets” folder to house my supporting images, CSS and JavaScript – since I’m working in a sandboxed solution, I need these to go into the content database and in my case the contents get deployed to the SharePoint library of the same name. After copy/pasting these files into my Visual Studio project, I get this:

..and, 0f course, Visual Studio is kind enough to detect the new files and generates my elements.xml file accordingly:

The next step is to ensure the site we’re deploying to references these files – we’ll deal with JavaScript first and then CSS.

Add jQuery/jQuery UI to the page (in this case, every page in the web)

Here we need to ensure jQuery and jQuery UI are added to pages in the site. Since we’re not deploying a custom master page (and don’t want to use the Content Editor web part to add JS to the page), we’ll need to use either CustomAction + ScriptLink or a delegate control going into ‘AdditionalPageHead’. In my case I’m choosing the former so that my solution works in the cloud (as a sandboxed solution):

Notice that I’m referencing a non-minified version of jQuery UI – that was the issue I said earlier that I would resolve for production use. The next step is CSS.

Ensure the jQuery UI branding/CSS is applied

Since we’re not using a custom master page, we’re just using the AlternateCssUrl property of the web to ensure our custom CSS file is referenced by our pages:

Our site should now receive all branding and JavaScript dependencies – so if we were to deploy a static page with the correct HTML for a jQuery UI accordion, then it should work fine. However, we want ours to serve as the UI for a SharePoint list, so we’ll now create that list. In the final step we’ll do the actual JSLink work – this will change the rendering of the list at run-time.

Create the list we will use with JSLink

Nothing special here and I won’t go into every step – but it is worth mentioning that creating a list via Visual Studio 2012 is much easier than before:

..then add whatever columns your list needs (here I’m adding them direct to the list rather than to a content type):

As we mentioned earlier, we are allocating a new list type ID  (the value used was 11000) for the underlying template for this list – this is important since, in this “new list” scenario, we’re using a combination of this and the BaseViewID for the "All items” view to hang our JSLink customizations off:

Finally I add some dummy data (some old blog articles) to my list so each time I deploy/test, I don’t have to manually add list items:

Download link

You can download my Visual Studio project for this sample here.

Conclusion

SharePoint 2013 brings some great possibilities for customizing the user interface, and JSLink/client-side rendering is central to much of this. The great news is that it’s all cloud-friendly, so these techniques can be used in Office 365. The solution above was implemented as a sandboxed solution, and the only code was really to set the AlternateCssUrl of the site. As a result, it would be fairly easy to move all this to a SharePoint app if needed.

Hopefully this has been useful in understanding how JSLink/client rendering might be a valuable tool.