SharePoint developers have always had a need to store files in a central location that can be accessed/shared from ALL SharePoint sites. This is often needed for supporting files such as images, CSS and JavaScript - usually we don’t want such files to exist in each individual site, because when updates are needed we might then have 10,000 different places to update the file. As you can imagine, there are other reasons to avoid this duplication too.
One long-established way of working around this is to deploy such files to the server filesystem, rather than into a specific SharePoint site. Commonly, the ‘_layouts’ folder under SharePoint root directory (i.e. ‘14’ for SharePoint 2010 or ‘15’ for SharePoint 2013) was used for such shared files. A second method involved deploying files as ‘uncustomized’, or ‘GhostableInLibrary’. This works by adding a list item ‘stub’ for the file into each site or library, but since the file is also deployed to the SharePoint server’s filesystem, the file contents are pulled from there. In the content database, the SQL record literally has a pointer to the location of the physical file on the SharePoint server (again, this would be somewhere under SharePoint root directory).
Cloud says no
Once you start to develop sandboxed solutions or SharePoint 2013 apps, of course the technique above cannot be used. Files cannot be provisioned to the server filesystem, since the servers might not belong to you – in the case of Office 365, the servers are run by Microsoft. So what can we do?
Well, something I’m sure I’m not the first to think of is that if the production servers are connected to the internet (and cloud services like Office 365 always are), then you can just store your images, CSS and JavaScript files on some other servers (or service), and your pages will run just fine. For what it’s worth, I’ve been doing this on my blog site for quite a while – if you go looking, you’ll see I have some CSS and JavaScript being loaded ‘across the internet’ from http://sharepointnutsandbolts.azurewebsites.net rather than http://www.sharepointnutsandbolts.com. Like me, you might find that Azure is a good choice here – it’s a solid offering with several flavors (I use the free version), and numerous options exist for getting your files there (FTP, WebDeploy, continuous deployment from TFS/Git etc.). Of course, any similar cloud service would work fine too – as indeed might some on-premises non-SharePoint servers (rather than service) you supply. Just remember that with that latter option, you’d need to take care of high-availability, backup/restore, maybe load-balancing etc. Thanks to Azure, I can let Microsoft deal with that :)
As you can imagine, we now regain the ability to store a single instance of files, even if not on the SharePoint server itself. To illustrate, here’s what the ‘default’ and ‘centralized’ approaches look like in pictures:
Default app/sandboxed development model:
Centralized development model:
When should I be considering this?
I think this broad approach is relevant to the following scenarios:
Developing sandboxed solutions
Developing SharePoint-hosted apps
Developing auto-hosted apps
The scenario I’ve left out is provider-hosted apps. This is because here your supporting images, CSS and JavaScript files are stored outside of SharePoint anyway – and by default are stored as a single instance. After all, a provider-hosted app is just a non-SharePoint website at the end of the day. Auto-hosted apps are perhaps an interesting case because the app’s supporting files are deployed to Azure, but the same deal applies – they are not shared between app instances.
If you are working in these models and not currently considering this approach, perhaps you should. Yes, it can make the initial development process more complex (since your app is effectively stored in multiple places), but rolling out updates once in production will most likely be much simpler. Even if you don’t like this idea, then at least ensure you’ve considered your update mechanisms for these files.
An interesting observation (if it hadn’t occurred to you already), is that provider-hosted apps are mainly stored outside SharePoint – including their business logic. This means, amongst other things, that the creator has the option of making fairly big changes to the app without requiring the app to be upgraded or resubmitted to the Store. Certainly worth bearing in mind.
Further considerations
I think the following also need to be considered:
Versioning
You’ll most likely need to provide some form of versioning of your “remotely-stored” CSS and JavaScript files, for example storing them under folders named “v1”, “v2” or “1.0.0.0”, “2.0.0.0” etc. This will allow you to deploy updates for later versions of your apps, whilst maintaining the original experience for users/sites who do not upgrade
Remember also that if you did want to make changes to existing CSS/JS files, these will most likely be cached at the browser for existing users. So you may need to do something to bust the cache (as discussed for non-app scenarios in Avoiding bugs from cached JavaScript and CSS files in SharePoint), but remember also that changes to the URL used will require changes/upgrades to the app itself unless the page is provider-hosted
Alternative storage flavors:
Although I’ve mainly discussed storing files in simple Azure websites, my colleague Salvatore di Fazio has also considered the use of Azure CDN –see his post Sharepoint 2013 – How to share contents between Apps
I hit an interesting issue recently, around provisioning the Content Search web part onto a page with the AllUsersWebPart element. I should say upfront that I’m not (currently) providing an awesome “here’s some code” solution to the problem, but will suggest approaches that should work fine. Mainly I want to raise awareness of the problem at this stage – if you hit it, you’re not going crazy.
If you’ve worked with the Content Search web part in SharePoint 2013, you’ll know that you can provide custom JavaScript templates (known as Display Templates) to control how items are displayed. Effectively you’re specifying the HTML which is output. Once the file is in the right place, you can select it in the ‘Display Templates’ section of the Content Search web part properties, like this:
These Display Template files are expected to live in the Master Page Gallery, and if you browse there you can see all the out-of-the-box Display Templates for various bits of SharePoint (Content Search web part, search hover panel, refinement panels etc.). There’s actually a significant difference between sites that have the publishing feature activated, and sites which do not. Specifically, if you browse in a publishing site, you’ll see two sets of files – HTML and JavaScript:
..but if you look at a site without publishing, then you only see JavaScript files:
This is one aspect of the “Design Manager” support provided in publishing sites – you get a HTML and a JS file for each template, and you can simply edit HTML file. When you do this, the associated JavaScript display template (which is going to output data in the HTML format you provided) is *generated* from your HTML file. This happens through an event receiver on the Master Page Gallery, and happens fast enough for iterative development to work out fine – as a reasonably hardcore dev, doing this bit in SharePoint Designer works fine for me. However, another option (provided by this support for HTML editing) is using a HTML editor with a design surface (Dreamweaver/Expression Web/Allaire Homesite (!)/whatever) – you can map a drive in Windows Explorer to the Master Page Gallery, to facilitate saving directly to your SharePoint dev environment from your editor.
Provisioning in a Feature - deploy the JavaScript and/or the HTML file?
So, before we get to my actual issue, we have an interesting decision point if we are working against a site with publishing in the "developer/Visual Studio” way. Do we:
Provision the HTML file in a Feature, and then “touch” the file somehow (e.g. custom event receiver) to ensure the JavaScript file gets generated?
Provision the JavaScript file only
After playing around with both options, my strong view on this is that you should only provision the JavaScript file. By all means, use the HTML support in the core dev phase to obtain the JavaScript file – but when you’re ready to take this to other environments, package it up in Visual Studio, and include it in your farm/sandbox WSP (N.B. none of this really applies to apps). I say this because:
You are NOT going to need those HTML files in production – they are not used at runtime. The whole thing is purely to support the design phase!
If you need to go back and make changes, then you can either edit the JavaScript file directly (frankly no sweat for most devs) or fire up a publishing site somewhere, use the HTML support, and then drop it back into Visual Studio when done
Touching a file –> to trigger an event receiver –> to generate a file –> which you already have in the development environment == no sense to me :) However, I would say that if you *are* working in a publishing site (e.g. in SPD), always edit the HTML file if it exists – otherwise of course, your changes to the .js file will be overwritten whenever that event receiver executes.
I use XML like the following for this – I found that the key property, ‘HtmlDesignAssociated’ defaults to 0 (zero) but when deploying the .js file only, I like to set it to zero explicitly:
With that out of the way, let’s discuss the problem I see when extending the above - to have a Content Search web part automatically added to a page with AllUsersWebPart element.
Problem - custom Display Template not linked when provisioning Content Search web part using AllUsersWebPart
[Update April 30, 2013 – Ivan Neganov has found a solution to what follows, see the comments at the bottom of the article!]
Hopefully that title was descriptive enough. So the goal is to provision the Display Template in a Feature, but also make use of it – i.e. provision some kind of page which has a pre-configured Content Search web part on it, where the configuration ‘points to’ a custom Display Template. I think this scenario will be pretty common – you’d use it in many places, such as rolling up some news articles with a particular look and feel. The process I used is what any experienced SharePoint developer would probably expect:
Configure the CSWP with the right settings on a page
Export the web part to get the XML
Drop into an AllUsersWebPart element in a Feature
Check the settings all look good
Test
The key bit of configuration we’re interested in is this line, but it all looks good at this stage:
<property name="ItemTemplateId" type="string">~sitecollection/_catalogs/masterpage/Display Templates/Content Web Parts/Item_Picture3Lines_COB_blue.js</property>
The overall web part XML is as follows (the “ItemTemplateId” is the last property to be listed):
The result is that seemingly all aspects of CSWP configuration take effect, except the custom Display TemplateSo. all other properties are applied, so the search query works fine and shows the correct items - but the look and feel is the out-of-the-box branding supplied by the ‘Item_Picture3Lines.js’ template (i.e. not my ‘Item_Picture3Lines_COB_blue.js’ file). What’s weird is that if you then edit the page, you find that your custom template is indeed available for selection (see first image in this article), and selecting it works just fine and the branding gets applied. At first I assumed this was some sort of issue with the format, or a ‘provisioning sequence’ issue, and played around with some ideas (breaking things into multiple Features with dependencies etc.) – none worked, so it was time to break out Reflector Pro.
After a pretty forensic ‘debugging SP2013 code’ investigation (I had a colleague trace my steps independently, thanks Hrayr!), this looks like a possible bug to me. We also tried environments before and after the March 2013 Public Update, but saw the same results. Essentially the provisioning process eventually calls into some code in the ContentBySearchWebPart class itself, and in particular a method called IsTemplateLocationValid. We see that one check appears to expect the path to start with “~sitecollection/_catalogs/masterpage” (this has been stripped off by earlier SharePoint code, despite the value being specified that way in the XML), so that one fails (click to see larger image – and sorry if you’re not into the Visual Studio ‘Dark’ theme!):
..and the next check expects SPContext to be present. However, it isn’t! I’m not sure why, given the Feature is being activated through the browser in the SharePoint ‘Activate Site Features’ page in my case, but there is definitely no SPContext. So between these two checks, I’d expect one to succeed. I can’t help but think that there is currently a minor bug in SharePoint 2013 - where SPContext is expected to be present in this provisioning process, but for whatever reason it isn’t. I say this partly because I notice that even for out-of-the-box templates, the value comes through stripped of the “~sitecollection” token:
..but then the page does not load (blank screen, HTML not output properly) and ULS shows the following runtime error:
Application error when access /SitePages/CSWP_Provisioned.aspx, Error=Cannot make a cache safe URL for "item_picture3lines_cob_provisioned.js", file not found. Please verify that the file exists under the layouts directory. at Microsoft.SharePoint.Utilities.SPUtility.MakeBrowserCacheSafeLayoutsUrl(String name, Boolean localizable, Int32 desiredVersion) …
Possible solutions
Broadly I think these are the options:
Manually fix-up any pages where you are adding the Content Search web part using AllUsersWebPart – i.e. edit the page by hand, select the Display Template in the web part properties, then save the page.
On-premises only - run some server-side code (e.g. in a Feature receiver) to fix things up. Use SPLimitedWebPartManager to update the web part properties.
For Office 365 – run some client-side code (JSOM or CSOM) to fix up the web part.
I don’t think any of the code would be hugely challenging to write – I might do it soon if no-one else does :) But for now, I just wanted to highlight the issue. Do you see a different behavior? Please leave a comment and let me know if so!
A brief post to mention that I’ve now published my slide decks for the two talks I delivered at the SharePoint Evolutions Conference in April 2013. I had a great time at the conference (as always!), and it was awesome to have so many people come to the sessions – thanks to all who came, I appreciate it. Just for a bit of atmosphere, here’s the attendee perspective in one of these conference talks (not mine – I was at the back, taking the photo):
As always, technical talks are made richer by the demos and the decks alone do not convey this – I’ve added some screenshots for one talk (the JavaScript session), but I haven’t for the apps talk as I’m not sure they would add as much. Anyway, here’s the summary and links:
Customizing the SharePoint 2013 user interface with JavaScript
Covers several approaches for user interface customization in SP2013 - using JSLink to customize a list and/or view, creating custom Display Templates for the Content Search web part, and different approaches for customizing the search hover panel.
Covers key aspects of SharePoint 2013 apps, with a focus on SharePoint-hosted apps. Includes detail on app parts, using web parts within an app, configuring SSL, troubleshooting apps and possible reasons to move away from a SharePoint-hosted app to a cloud app. Also covers "high-privilege" apps which provision to the host web.
Let’s say you’ve decided to make some custom functionality available to your users as a SharePoint 2013 app. You’ve decided that this will be published to the appropriate internal SharePoint App Catalog (as opposed to being available in the public SharePoint Store). In a large company, it’s probably not realistic to expect all users to install such internal apps, even if we want to make the functionality available to everyone.
The answer *might* be to use one of the options for deploying the app at “tenant scope”. This works for both on-premises deployments and Office 365/SharePoint Online. Microsoft even provide a nice admin interface for tenant-scope installs, which offers the following deployment options:
Deployment to named site collections
Deployment to all sites under a particular managed path
Deployment to all sites created from a certain site template
I said *might*. When deploying apps in this way, there is a crucial detail to be aware of – something which could either be highly-desirable or a deal-breaker. Essentially, all installed instances of the app *share* a single app web. This means any lists or libraries you provision as part of your app, are shared amongst all the webs where the app is installed. As I say, this could be exactly how you want your app to operate, or completely against it’s design.
The process for using this model is as follows:
Upload app to App Catalog.
Install the app to the App Catalog site collection – yes, for this scenario we actually have to install the app there (for reasons which will become clear later).
Go to the Site Contents page and find the app. On the callout menu here (and only here), you’ll have an action labelled ‘DEPLOYMENT’:
Click the ‘DEPLOYMENT’ button to go to the admin page.
On this page, I see the 3 options for deployment – individual site collections:
..or deploying to all sites under a Managed Path, or all sites created from a certain template:
Once you’ve selected appropriate options and hit OK, after a few minutes time you’ll find your app has landed in the target locations:
As you can probably guess, it’s a timer job which performs the deployment. This is the App Installation Service timer job, set to run every 5 mins by default.
Important aspects of tenant-scoped app installations
As I highlighted before, all app instances share a single app web. The way this works is that the instance which is installed to the App Catalog site is used for all instances – effectively the user is always redirected to this URL on the app domain, regardless of which site they entered the app from:
There are also important by-products of this:
Users in sites where the app is installed cannot remove the app – regardless of their permissions there
This makes sense because they would effectively be uninstalling the single instance used by everyone
Individual “usages” of the app do not get reported by Get-SPAppInstance – only the instance in the App Catalog site gets returned (N.B. applicable to on-premises SP2013 only)
Any changes to the app (e.g. app upgrades, app removal) need to be managed at the instance which is installed to the App Catalog site.
But I want something else - to roll out my app in bulk, but it NOT to be “single instance”
Then in this case, PowerShell is your friend – this is a two step process to install a new app:
Use Import-SPAppPackage to initially install the app to the site collection, then..
If you’re on Office 365 (with it’s limited set of PowerShell cmdlets, as of April 2013), you’ll need to accomplish the same thing using one of the client APIs. I’m starting to form the view that .NET CSOM code “wrapped” with PowerShell is a good way to do this – since it can then be integrated with TFS automated builds etc.
I’m a bit late in posting this, but if you’re still considering whether to attend this conference (London, April 15-17th 2013) then I highly recommend it. As usual, the speaker line-up looks awesome and after missing out last year (due to moving house) I’m really looking forward to it.
I’m giving two talks centred around SharePoint 2013:
DEV 203 - Deep dive into SharePoint-hosted apps (Monday, 1:30pm)
Even though apps can be outside of SharePoint, often it makes sense to leverage what's inside. This session takes a close look at how to use lists, content types and pages within a SharePoint-hosted app - whether there are additional remote components or not. We'll also look at advanced approaches for using a custom site definition, and explain why some web parts can be used and some cannot. In addition to working with the app web, we'll discuss pros and cons of "high-privilege" apps which provision into the host web - potentially bringing collaboration-based solutions closer to the app world.
P&M 306 - Using JavaScript templates to customise the SharePoint user interface (Tuesday, 9am)
Developers often have a need to amend the HTML rendering of SharePoint, especially around lists and list forms, but in previous versions of SharePoint this was not always easy. Many developers resorted to using JavaScript to manipulate the page after it had loaded, even though this was potentially unreliable and unsupported. SharePoint 2013 solves this problem by providing the ability to specify a JavaScript 'template override' for several user interface elements. Even better, these can be applied globally or locally as needed. Come and learn how to transform the SharePoint interface the supported way!
Displaying the right data in the Content Search web part in SharePoint 2013 (this article)
Customizing Display Templates – modifying the HTML/CSS output by the Content Search web part
So, let’s say you’re working with the Content Search web part in SP2013 and you’ve got it displaying the correct items. (If not, see my big post on Using the Content Search web part (and understanding search). In the case of this article, I’m searching for items I’ve created and tagged with “SharePoint”. Whatever the query, you’ll most likely be confronted by something like this:
Your client will be thrilled with something so aesthetically pleasing! By the way, the above search is for “documents in my SharePoint farm tagged with SharePoint” - having a thumbnail image shown might not really be needed for documents, but just for fun we’ll talk about both options. In a later post, I’ll cover using the CSWP to roll up some news pages (publishing pages) – these do tend to benefit from thumbnails, but the good news is there’s less work to get them there.
Anyway, if we break down the issues we’d like to fix, we would say:
An image is rendered for each item, but one is not found (so a default image is used)
The fields may not be correct – certainly for the 2nd and 3rd items, a fairly useless value is being fetched from the Office document properties. This is being displayed in the absence of a ‘Description’ field on the list item (N.B. this is actually a great ‘fallback’ feature of SP2013 search – Office doc properties get pulled into the property bag of the list item, and indexed by search if no other values take precedence [via Managed Property mappings]), but in this case, it’s some irrelevant data about who created the original PowerPoint template for the conference I presented at:
The overall formatting (HTML and CSS, especially colors) do not match our desired rendering
There are several settings in the CSWP web part properties at work here:
Thing
Screenshot
Notes
Control Display Template
This determines the overall style of the presentation – it’s effectively the “outer HTML” that is output.
I’d say List will be the most commonly-used value, but it’s great to see List with Paging come for free. Slideshow is an image rotator.
Item Display Template
This dictates the style applied to each item displayed i.e. “the HTML for each item”. Later when we create a new Item Display Template, we’ll see this become available for selection in this dropdown.
Property Mappings
These are very much like ‘slots’ in the Content Query web part – they are a feature of the Item Display Template which is selected (specified in JavaScript). Item templates use generic properties (e.g. ‘Picture URL’, ‘Line 1’ etc.), and you can ‘map’ some specific fields which exist on your list item. The image shows the default mappings. This means that if you only need to get a different field displayed in an existing slot, you can do thiswithout modifying XSL (CQWP) or JavaScript (CSWP).
If you needed 10 slots rather than 5, editing the Item Display Template’s JavaScript controls this.
Note that can specify multiple values specified by a semi-colon (as shown).
Choosing the Item Display Template
As you change the selected Item Display Template in the dropdown, the Property Mappings section in the CSWP tool pane will change accordingly, allowing the mapping for each slot. Once configured appropriately (see next section), “Picture on left, 3 lines on right” can have 3 lines of information displayed:
Whereas “Two lines” of course, will give you no picture and just two lines:
We’d probably say that “Two lines” works just fine for documents – unless you do happen to want thumbnails (e.g. for some key policies/procedures/CAD files/PDFs/whatever). Whilst thumbnails kinda happen by default for publishing pages, the process for other items is useful for illustrating the ‘Property Mappings’ aspect – first let’s cover the possibilities for working with these mappings.
Deep dive - getting the right data coming through
If you haven’t guessed already, those property mappings such as ‘PublishingImage’, ‘Path’, ‘Title’ and ‘Description’ above are Managed Properties in the search service app. If you are rolling up content from multiple locations (Content by *Search* remember!), consider that these mappings allow you to show results from across your farm/tenant even if you don’t have standardized content types with the same fields everywhere. For example, perhaps ‘Description’ could really come from a custom field named ‘Project Description’ in one place but ‘Task Description’ for other items.
The use of Managed Properties means you might need to go down a different route to get your desired values coming through:
Option 1 – no work needed, the CSWP automatically displayed my data where I want it!
Awesome! You’ll see this for the ‘Title’ field and some others (e.g. roll-up images in publishing pages). This is happening because for this ‘slot’ in the selected item display template, the default Property Mapping contains a Managed Property which points to the field on the list item/content type which holds the data. Perfect. If this isn’t happening and you think it should be, check that you have populated some content and it has been successfully crawled – you won’t get anywhere until it has.
Option 2 – use an existing Managed Property and site column, but edit the CSWP properties for a particular slot (to point to this Managed Property). You would do this when the CSWP isn’t looking at this Managed Property by default for this slot:
It’s a big old dropdown, but all you have to do is select the existing Managed Property you want:
Additionally you can add to the semi-colon separated list of Managed Properties, by typing into the box.
Option 3 - use a new site column but add it to an existing Managed Property:
Rare maybe, but you might want to do this if you want the same mapping to be use in different places e.g. multiple CSWP instances, or CSWP and the search results page etc. The process would be:
Add a new column to your list/library/content type. Populate some content and perform a full/incremental crawl.
Find an existing, somewhat generic, Managed Property which represents the thing you are working with – this could be something like ‘Title’ or ‘Description’. Edit the Managed Property so that the corresponding Crawled Property for your field (e.g. ‘ows_COBDescription’ for a field named ‘COBDescription’) is added to the list of Crawled Properties.
Ensure your Crawled Property is higher in the mappings list than others (if using the ‘Include content from the first crawled property that is not empty..’ option. As an illustration:
Option 4 – use an entirely custom Managed Property and site column:
This might be the best approach if there are no appropriate fields in existence, or you really don’t want to pollute any other CSWP instances and/or the search results page with these settings. I used this approach to add thumbnails to my documents when using “Picture on left, 3 lines on right”. The process would be:
Add a new column to your list/library/content type. Populate some content and perform a full/incremental crawl.
Create a new Managed Property, and map it to the corresponding Crawled Property for your field (e.g. ‘ows_COBThumbnailUrl’ for a field named ‘COBThumbnailUrl’).
Edit the Content Search web part properties, so that the Property Mappings for the desired slot contain the name of the Managed Property – here I’ve added my Managed Property (‘COBThumbnailUrlManaged’) to the ‘Picture URL’ slot:
In general, the deciding factors will be things like:
Do I want this change to take effect everywhere? (editing an existing Managed Property [i.e. adding a new Crawled Property to it] could mean that other Content Search web parts/search results show data from an instance of this column)
Is there an existing field and/or Managed Property which could represent this data?
“CAD files/documents with thumbnails (in team sites)” example
Recently I had to implement thumbnails for CAD files for a client demo – to appear in both CSWP results and the search hover panel (I’ll talk about customizing the search hover panel another day). The thumbnails aren’t automatically generated, but could be in the future. At it’s core, there’s an additional column which contains a hyperlink to the thumbnail image (e.g. in a Site Assets library).
I chose option 3 above in this case – since I don’t need to ensure I’m scoping only to one CSWP instance or anything.
Here are the specific changes I made to get my thumbnails and descriptions to come through to the CSWP:
Added two columns to my content type, to contain the thumbnail URL and description
Populated these fields for each item (for the thumbnails, as mentioned the images themselves live in the local Site Assets library)
Item thumbnail:
I created a site column (‘COBThumbnailUrl’), and mapped this to the ‘PictureThumbnailURL’ Managed Property
Item description (for display in the ‘Line 2’ value):
In this case I re-used the OOTB ‘Description’ Managed Property, but added a new column ‘COBDescription’ and included this in the Crawled Properties mapped to that Managed Property
(N.B. There are a bunch of OOTB fields named ‘Description’ in a team site, but most are in the ‘Hidden’ group and didn’t actually seem suitable for re-use)
Once the thumbnail is against that file, it’s fairly easy to surface it in search hover panels (as well as CSWP rollups):
“Publishing pages with thumbnails” example
If the items you’re rolling up are publishing pages (e.g. news pages), then things are slightly simpler – this is because the default mappings of, say, “Picture on left, 3 lines on right” work better. Assuming you’ve configured the query correctly and the pages have a roll-up image set (e.g. a rendition of an existing image), then you’ll probably get thumbnails but no descriptions by default:
In this case, you might notice that my thumbnails aren’t high quality. This is because they are simply generated using the SP2013 publishing framework’s image renditions capability (i.e. a 100 x 100 version of the much larger image displayed on the page itself) – if I produced special images for thumbnails, the result would be higher quality (but more work).
We have no item descriptions so far, only titles. However, if we edit the Property Mappings for “Line 2” and “Line 3”, we can start to bring in other data – here I’m selecting the article byline and also the property which lists which terms this item has been tagged with:
..and now we get the those details displayed by the Display Template:
Next considerations – modifying the HTML/CSS/JS, and packaging/deployment
So far, we haven’t worried about changing the actual HTML which is output. My next articles look at this, for both publishing and non-publishing scenarios (the process is slightly different).
There are also a couple of considerations for working the “developer way” and provisioning these artifacts through a WSP. I’ll also discuss these next time.
Over the past couple of years I’ve been a big advocate of continuous integration (automated builds) for SharePoint projects, and I’ve previously published some resources to help people get started with this. Specifically I have:
A custom TFS build workflow – this is for use with TFS Build. It integrates with some PowerShell scripts I also supply, so that WSPs can be deployed automatically as part of the build. This is my “SharePoint CI starter pack”.
Assembly versioning workflow activity – this is an optional custom workflow activity which will increment the version numbers of your assemblies as they are built in the right way for SharePoint. To simplify things I supply one workflow with this activity dropped in and configured, and one without.
Because Microsoft made some changes between TFS 2010 and TFS 2012, these things do not “just work” for TFS 2012 unfortunately. In fact, migrating a build workflow from TFS 2010 to TFS 2012 is a bit of a nightmare. But, I’ve now updated these with new releases for TFS 2012 so things should be a bit easier for if you work with this stuff.
A while back someone pinged me on Twitter asking for an update, and then some Microsoft folks got in touch to say “can we take ownership of your stuff, or could you update it?” and then last week I was in meetings about a TFS 2012 upgrade at my place of work (Content and Code), and since our larger SharePoint projects use this stuff I knew I had to bump this up the priority list :)
Both new releases are labelled “for TFS 2012”, and instructions can be found on each site. Briefly, you’ll need to download both if you want to perform versioning within your automated builds – if not, you can just use the CI starter pack, selecting the “no versioning” workflow in there.
Release notes
I’ve added release notes on Codeplex, but just to summarize here:-
TFS simple assembly versioning
This is a brand new workflow activity (with a new name - COB.CI.Workflow.AssemblyVersioning), due to the incompatibility between TFS 2010 and TFS 2012. As I say above, you can either drop this into your own (custom) TFS build workflow to have your assemblies versioning automatically, or use my workflows from my CI starter pack.
This has been fairly well tested and is robust.
SharePoint CI starter pack
As before, this contains 2 workflows (with versioning/without versioning) and a set of PowerShell scripts which deploy WSPs etc. If you’re an existing user, you should add the 2 workflow XAML files to TFS, and then edit your existing build definition(s) to use one of these workflows.
The TFS/PowerShell integration to deploy WSPs as part of the build has been well tested. However, at the time of writing I haven’t yet tested including automated tests (unit tests/UI tests) into the build – at this point, I just want to get something out there quickly to support any TFS upgrades of existing users.
Incompatibility issues around TFS 2010/TFS 2012
For completeness, I just wanted to show the errors you’ll see if you upgrade to TFS 2012 and are using my resources (i.e. attempting to use TFS 2010 workflow stuff in TFS 2012). In fact, I think you can expect to see errors if you open any build workflow after the upgrade – whether you use my stuff or not. You’ll see something like:
System.IO.FileNotFoundException: Could not load file or assembly ‘Microsoft.Team.Foundation.Build.Client, PublicKeyToken=b03f5f7f11d50a3a’ or one of its dependencies.
System.IO.FileNotFoundException: Could not load file or assembly ‘Microsoft.Team.Foundation.Build.Workflow, PublicKeyToken=b03f5f7f11d50a3a’ or one of its dependencies.
Various other error messages all related to versioning.
Yes, it seems many TFS 2010 build workflows appear to have version numbers stamped into the XAML. I understand this is due to an unfortunate bug in VS 2010, specifically Team Explorer - this KB article has more details and a hotfix. However, chances are that (like mine) your workflows already have ended up with the version numbers in them by now. So, it’s certainly more painful than it should be.
You have two options to make your build workflow run in TFS 2012:
Rebuild it from scratch in VS 2012/TFS 2012 – that’s what I did in the end with my CI starter pack workflows
Attempt to clean the XAML:
I tried this by hand several times and failed. The workflow failed to display in the designer, and there is precious little diagnostic information.
The Jason Prickett post linked above also has other variations of error messages you might see.
Related error messages you might see if you use my resources:
Again for completeness, here are some other errors you might encounter if you’re using my resources but are “doing it wrong” according to TFS 2012. This is what you might see if you open a workflow with my (TFS 2010) assembly versioning activity after upgrading to TFS 2012/VS 2012:
You’ll need to fix the workflow to use the latest version of the activity if so.
And here’s what you’ll see if you try to drop my old (TFS 2010 version) assembly versioning activity into a TFS 2012 workflow:
Conclusion
Upgrading to TFS 2012 brings some pain if you do automated builds, whether in relation to SharePoint or not. For any SharePoint folks who use my resources, I’ve upgraded these to hopefully save you some pain. Please leave a comment on the Codeplex site if you hit any issues with them.
Changing the user interface of certain bits of SharePoint has always been somewhat challenging – at least, if we want to avoid the SharePoint Designer route and produce something which is packaged, deployable as a WSP and repeatable across many sites. Not to mention source-controlled. A common area of customization is the user interface around SharePoint lists - when it comes to this, there are a few “flavours” of common requests:
Custom list forms – e.g. a more “designed” form for adding and/or editing items
Custom behaviour for a certain field - e.g. a “UK address” field which allows postcode lookup
Custom rendering of a list in display mode (e.g. the ‘All items’ view) – a good example of this is conditional formatting e.g. “Add a red background to the row if the value for ‘Days overdue’ is over 5”
As you might guess from the article title, it’s the third scenario that this article focuses on, for SharePoint 2013 specifically.
Earlier versions of SharePoint gave us a couple of approaches for fulfilling some of these requirements. We could create a custom field control (item number 2 above) or modify the XSLT of an XsltListView perhaps (number 3 above) – a method that was new in SP2010 of course, since we previously had CAML rendering for list views (yuck). SharePoint 2013 brings a new JavaScript-based approach for modifying the UI of fields and lists, and many more things too. You’ll often see this referred to as “JSLink”, because many things in SP2013 have a new JSLink property which you use to point to your JavaScript file. Here’s a list of SharePoint objects which you can modify in this way:
Lots of interesting possibilities there – all the field types are obviously represented, but binding UI changes to a:
content type
form
view
list view web part instance (rather than to the list/view itself)
..could all be extremely useful.
[As an aside, if you came to this article looking for a solution to the 1st scenario above (modifying forms), you’ll probably be interested that SPForm has a JSLink property. However I imagine there’s still a place for providing entirely custom forms as opposed to making relatively small changes with JavaScript. The old approaches still apply here - modifying list forms is easy in SPD, but requires a bit more thought in the Visual Studio world. In that case, we’d probably want to NOT edit the original .aspx, but instead provide a different .aspx file and update the SPList.NewFormUrl and/or SPList.EditFormUrl properties to point to it (e.g. in a Feature receiver).]
Implementing changes to list rendering with JSLink
For a recent SharePoint talk, I wanted a fairly dramatic example of changing the UI of a list (when looking at list items). So, my example showed changing the list from this:
..to this:
There are some slightly nasty scrollbars in these images, but that’s just because I’ve reduced the size of the window for my screenshots - normally the accordion looks great. As you might imagine, I get a nice sliding “expand and contract” experience when I select different items:
If you’ve worked with it before, you’ll immediately recognise this as the accordion from jQuery UI. In many ways, it’s a nice example, since although YOU are unlikely to need/want to use jQuery accordion for your “customizing a list” needs, it uses custom CSS, images and JavaScript to provide rendering which you probably ARE going to need. So, I’ll walk through most aspects of this process, but as you’ll see only around 20% of the work relates to the JSLink stuff – the rest is just scaffolding. We’ll cover the JSLink specifics first.
The full Visual Studio project can be downloaded at the end of this article.
The important bit (part 1)– how to hook up your custom rendering with JSLink (e.g. call into jQuery UI accordion)
The first thing to say is that when we want to change the rendering of a SharePoint list, it’s actually individual views (SPView) we’ll be working with. The SPList class does NOT have a JSLink property, which makes sense given this is all about presentation. Getting SharePoint to “see” your custom rendering will probably depend on what you’re doing:
Scenario
Approach
Creating a new list
Specify the path to your .js file in the schema.xml file for the list (specifically within the declaration for the view)
Modifying an existing list
In code, update the SPLink property (i.e. SPView.JSLink) to provide the path to your .js file (using PowerShell/server-side/client-side API as appropriate)
This bit only ensures your .js file is referenced by the page. You also need to make sure it has the right contents to register your display templates – we’ll come to that in a second.
In my case, I’m creating a new list – it’s worth noting that I’m assigning a list type ID of “11000” – we’ll use this later with respect to JSLink:
When developing for SharePoint 2013 onwards, when a list is created in Visual Studio the JSLink element for any views will contain “clienttemplates.js”:
..but we should change that to our custom JavaScript file which has our display template implementation (you’ll see me deploy this later):
Our file will then be loaded when this view is requested (i.e. SharePoint will add the .js file to the page). But that’s not enough – we now have to think about what JavaScript is needed to actually register the templates. This is done by specifying two properties of the list views to match:
OPTIONAL - the BaseViewID property (e.g. BaseViewID=1 for a standard view like “All items”, but you could use another ID for a custom view)
The ListTemplateType property (for a new list, you’ll be specifying a unique integer value e.g. 10000)
So, it’s usually a combination of THREE controls overall which dictate how rendering is applied to a list view – the JSLink path, BaseViewID and ListTemplateType. Although BaseViewID appears to be optional, it seems sensible to set it to avoid unforeseen problems with Explorer View/Datasheet View etc. So for a given list, if you wanted one view to have custom rendering and one to have default rendering (even though they share a BaseViewID), simply ensure the JSLink property for the default one is NOT set to your custom .js file.
I also note that it appears possible to specify multiple values in a JSLink path – I haven’t tried this, but I see things like <JSLink>mquery.js|contentfollowing.js</JSLink> (note the pipe character) within out-of-the-box files under the 15 folder.
The important bit (part 2)– what your JavaScript should look like:
Here’s my full AccordionListView.js file specified in the JSLink property for my view – I supply a header and footer and then a JavaScript method (function pointer) to execute for each list item. Notice some context gets passed to this function, including details of the list item:
As you might be able to infer, the jQuery UI accordion expects a H3 and div element for each item – so that’s what my template does, in addition to actually calling the accordion() method.
An issue with JSLink and the Minimal Download Strategy (MDS)?
Continuing my recent tradition of discovering slightly strange behaviour around what I’m writing about, this week is no exception. In my testing, I noticed that if a list has multiple views and the user switches between them, the client templates specified by JSLink do not get applied if MDS is enabled (even though the debugger shows they are called). I’m hoping that I’m doing something wrong, but I can’t rule out a bug in SharePoint 2013’s MDS framework at this stage.
If anyone sees this/has any info, please leave a comment :)
Integrating CSS/JS (such as jQuery UI) into the solution
OK, so if all you wanted to know about was the JSLink aspect, we’re now done covering that. Hopefully that was useful. The remaining bits of this article will cover the “scaffolding” aspects of my particular jQuery UI accordion example – integrating jQuery/jQuery UI, and so on.
My first step was to go to the jQuery UI Download Builder and download the files (plus jQuery too if you don’t have it already). If you haven’t done this before, you can basically configure some options (e.g. styles, behaviors, jQuery UI widgets to use) to build a package for download – this means you get a smaller package than if you were opted for all styles and all jQuery UI components. You’ll therefore have lower page weight and better performance, than if you were using the full set. However, I had an issue where my solution would only work with the full jQuery UI file rather than the one I “built” – if this was production code I’d stop and resolve this, but for demoware I was OK with it.
When you get your download and crack it open, you’ll have some CSS and JavaScript files:
I chose a grey-ish theme called “overcast”, and within the CSS folder come a stack of images – we’ll need to integrate all of these files into our VS project:
In my project, I created a “Site Assets” folder to house my supporting images, CSS and JavaScript – since I’m working in a sandboxed solution, I need these to go into the content database and in my case the contents get deployed to the SharePoint library of the same name. After copy/pasting these files into my Visual Studio project, I get this:
..and, 0f course, Visual Studio is kind enough to detect the new files and generates my elements.xml file accordingly:
The next step is to ensure the site we’re deploying to references these files – we’ll deal with JavaScript first and then CSS.
Add jQuery/jQuery UI to the page (in this case, every page in the web)
Here we need to ensure jQuery and jQuery UI are added to pages in the site. Since we’re not deploying a custom master page (and don’t want to use the Content Editor web part to add JS to the page), we’ll need to use either CustomAction + ScriptLink or a delegate control going into ‘AdditionalPageHead’. In my case I’m choosing the former so that my solution works in the cloud (as a sandboxed solution):
Notice that I’m referencing a non-minified version of jQuery UI – that was the issue I said earlier that I would resolve for production use. The next step is CSS.
Ensure the jQuery UI branding/CSS is applied
Since we’re not using a custom master page, we’re just using the AlternateCssUrl property of the web to ensure our custom CSS file is referenced by our pages:
Our site should now receive all branding and JavaScript dependencies – so if we were to deploy a static page with the correct HTML for a jQuery UI accordion, then it should work fine. However, we want ours to serve as the UI for a SharePoint list, so we’ll now create that list. In the final step we’ll do the actual JSLink work – this will change the rendering of the list at run-time.
Create the list we will use with JSLink
Nothing special here and I won’t go into every step – but it is worth mentioning that creating a list via Visual Studio 2012 is much easier than before:
..then add whatever columns your list needs (here I’m adding them direct to the list rather than to a content type):
As we mentioned earlier, we are allocating a new list type ID (the value used was 11000) for the underlying template for this list – this is important since, in this “new list” scenario, we’re using a combination of this and the BaseViewID for the "All items” view to hang our JSLink customizations off:
Finally I add some dummy data (some old blog articles) to my list so each time I deploy/test, I don’t have to manually add list items:
SharePoint 2013 brings some great possibilities for customizing the user interface, and JSLink/client-side rendering is central to much of this. The great news is that it’s all cloud-friendly, so these techniques can be used in Office 365. The solution above was implemented as a sandboxed solution, and the only code was really to set the AlternateCssUrl of the site. As a result, it would be fairly easy to move all this to a SharePoint app if needed.
Hopefully this has been useful in understanding how JSLink/client rendering might be a valuable tool.
Sidenote - although this article focuses on search, I’ve tried to generalize slightly so that it may also be useful for:
Understanding the pattern for working with *any* SharePoint 2013 REST API (e.g. social, querying lists/libraries etc.) – the URL used for the REST call is the main piece that would change
Seeing (broadly) how to process REST API results formatted in the two available formats - JSON or XML
Understand how to call a REST API from both a SharePoint-hosted app (JavaScript) and a cloud app (provider-hosted or auto-hosted – C#)
Briefly discuss using CSOM to call search instead of REST
Often it’s valuable to leverage SharePoint services such as search, user profiles, Managed Metadata and so on from an app. With search specifically, an example I’ve been giving recently is my ‘learner’ time-tracking app, which displays a ‘Related content’ bar with links to documents and other things which relate to completing timesheets:
My search query is somewhat simplistic (I’m currently just searching for any documents and pages which contain the term ‘timesheets’), but it’s easy to imagine that with some refinement we could return only highly-relevant items. A nice approach in real-life could be to have a configurable value to allow each client (who installs the app) to edit/store the search query that works best for them. Regardless, I think the main point is that search can be a valuable ‘bridge’ between an app and the rest of the SharePoint environment. In my example, consider that:
As the app vendor, I have no idea what your organization’s timesheet policy/guidelines are
If I can bring in some of your content around this, users will have more success with my app, and that’s good for everyone
In terms of implementation, I’ll now talk about the basics of accessing search via REST, and also how to do this from an app.
Broadly, the process is:
Identify (and test) the REST URL to use
Deal with app authentication if necessary (i.e. you’re calling from outside SharePoint, e.g. a cloud app)
Add a Permission Request to your app, so that a request to use search is successful
Develop the success handler code to process the results
Accessing search via REST (_api)
The beauty of REST is that it’s just a URL, and you can test/refine it in the browser before writing a single line of code. From SharePoint 2013, such a URL takes the format:
[site]/_api/[name of SP API]/[name of operation]?[parameters]
So in the case of search, a basic query can be executed with the following:
[site]/_api/search/query?querytext='timesheets'
If such a URL is entered into the browser address bar, then you should see an XML response like this – you’ll find some interesting properties, like those I’ve highlighted in the images:
Common operations
So far we’ve just looked at executing a basic query where the query text is specified. Here are a couple of other common operations you might want to do:
Operation
Sample REST URL
Use a search Result Source (i.e. a scope)
/_api/search/query?querytext='search term'&sourceid='B09A7990-05EA-4AF9-81EF-EDFAB16C4E31' (this example is to search the ‘People’ result source)
For a more comprehensive reference, the best source I can find is the SharePoint 2013 Search REST API post in the ‘Search Space’ blog by ‘nadeemis’ on MSDN – kudos to the author.
Working with the data
As with other SharePoint REST operations, you can choose to have the data in XML or JSON – and JSON might be preferable if you are in JavaScript. As I’ve mentioned before, this is accomplished with the “Accept” header – specify one of the following:
Your code which processes the results will need to deal with JSON/XML as appropriate. In either case, you’ll usually have some drilling down to do to get the valuable data – for example, the main result rows are in a property called jsonObject.d.query.PrimaryQueryResult.RelevantResults.Table.Rows.results (where jsonObject is the object which was parsed from the JSON string). And when you’re iterating through the results, if you want the URL for the item, then you can get that from the cell at index 6 with result.Cells.results[6].Value. The title is at index 3, so result.Cells.results[3].Value. These are basically just the things you work out with a JavaScript debugger, or from a code sample like mine below.
Calling the search REST service from the JavaScript Object Model (JSOM)
In this sample, I have a web page which has a DIV waiting to output the search results:
<div id="related-content-results"></div>
If I’m in a page running on SharePoint (including a SharePoint-hosted app), I can use SP.RequestExecutor (as opposed to a jQuery.get() or jQuery.getJson() for instance) and this simplifies things by meaning I don’t have to worry about adding an OAuth header to my request. So, this code would look like:
The URL we’re using is the app web, rather than needing to reference the host web somehow – this is fine for search (but isn’t for accessing data in the host web, for example)
We need something to iterate over the result items within the JSON – jQuery’s each() operator is a good choice here
There are some ‘magic numbers’ in processing the JSON results to get to the values e.g. result.Cells.results[6] to get to the URL of the item. That’s just a reflection of the data structure SharePoint returns, and you’ll see the same if you’re working with XML instead
Calling the search REST service from a cloud app (provider-hosted/auto-hosted) in .NET CSOM
Accessing the search REST API (just like any SP2013 REST API) in other contexts can be a bit more involved. The main consideration is that if your page isn’t on the SharePoint server, you’ll need to deal with OAuth and adding the ‘Authorization’ token to the REST request. If you’re in C# (i.e. the .NET CSOM), then the TokenHelper class automatically added to your Visual Studio project simplifies this – you’ll need to use a HttpWebRequest object (or similar) to call the REST URL, and then process the response. Since I’m on the server this time and have access to .NET, XML could be a better choice here:
And just for clarity, here’s my ASPX page which renders this – as you can see, I’m using an ASP.NET ListView control to put the results on the page:
Sidenote: Other contexts – calling SP2013 REST APIs using jQuery in a cloud app
As I worked through these samples, I realized that another useful approach would be calling the REST API client-side from a cloud app (in an AJAX way). In this case, we’d probably be coding in jQuery, and would need to deal with obtaining an OAuth token and adding it to the web request – but in this case, we don’t have TokenHelper to support us. Sure, you could perhaps wrap TokenHelper in a WCF service and call that from jQuery, but that would mean an inefficient process of multiple web requests (firstly to the service, then the exchange to get the OAuth token, and finally the call to the SharePoint REST API).
Far better would be to eliminate the extra call and just directly obtain the OAuth token and call the REST URL directly from jQuery – but some work is needed to facilitate this. I’m sure someone will look at this soon as I expect it to be a common pattern for SharePoint 2013 app development.
In order to be able to call the search REST API from an app (of any hosting type), the app needs to be granted permission to use the search service – this doesn’t happen by default. Like any permission which relates to “core SharePoint” rather than the app itself, the developer must specify the requirement in the form of a Permission Request, and the administrator installing the app must agree to granting this permission at install time.
So, the developer needs to add the following to the appmanifest.xml file:
Note that the value specified for the Right attribute gives a very descriptive string reminding you what’s being asked for – although most actions within an app run “as the app’s identity” (i.e. as the ‘app principal’), the search will instead be performed as the user (with the associated security-trimming).
In the Visual Studio designer for the appmanifest.xml file, this looks like this:
And, of course, when someone is installing (or upgrading) the app, they’ll see search listed as one of the permissions that the app requires – and can make an informed decision on whether they are OK with this:
An issue for on-premises deployments?
So everything above works great for Office 365 environments. However, I see something weird when I deploy an app which uses search to my local developer environment. I haven’t yet tested the app against another on-premises environment, but I’d be interested to hear if anyone else sees this – the app in fact cannot be added, and I see a message saying “Sorry, only tenant administrators can add or give access to this app”:
As a concept, I understand “tenant administrator” to apply to Office 365 environments but not on-premises
So, does anyone else see this? Am I missing something? Please let me know in the comments.
Finally, it’s worth pausing to consider – that fetching content via search across an entire SharePoint environment is a fairly elevated level of permission. The app is basically being given access to content created within your business, and anything in the search index is available. Clearly an app could potentially do some ‘interesting’ things with this, so it might be prudent to do some evaluation of such apps before blindly installing.
Conclusion
Calling into search from an app can be valuable, and here we’ve looked at how to do it from both a SharePoint-hosted app (in JavaScript) and a cloud app (in C#). The template used here for making such REST calls can be applied to many other operations – ultimately the pattern is to identify the URL, deal with authentication if outside SharePoint, ensure a permission request exists, then drill down in the JSON/XML response to get to the valuable data.
I'm an independent SharePoint/.Net consultant in London, UK, currently acting as Head of Development at Content and Code, a UK-based SharePoint specialist.
I've worked with Microsoft technologies for around 12 years and am MCSD.Net and SharePoint certified. My blog focuses on the architecture/development aspects of working with SharePoint.
