Using the Content Search web part (and understanding SP2013 search)

Articles in this series:

• Using the Content Search web part (and understanding SP2013 search) [this article]
• Displaying the right data in the Content Search web part in SharePoint 2013
• Provisioning the Content Search web part with a custom Display Template

Meeting client requirements with SharePoint often involves aggregating items somehow – often we want to display things like “all the overdue tasks across all finance sites”, or “navigation links to all of the subsites of this area” or “related items (e.g. tagged with the same term)” and so on. In SharePoint 2010 there have been two main ways of accomplishing this:

• Content Query web part
• Custom solution built on SPSiteDataQuery (site collection-scoped), SPQuery (list-scoped) or search API

To a lesser extent, using the search web parts as part of a custom solution may also have been an option. Regardless, it was common to need custom code to meet such requirements. Maybe we needed to add paging to the results, or we needed to use some value obtained dynamically through code (e.g. from the current site/current page/current user/something else) – several Codeplex solutions arose from this gap, and lots of lines of code were written.

SharePoint 2013 presents the Content Search web part as a new option – it’s capabilities mean that simply using the web part (with some front-end work to meet look and feel requirements) will meet many needs, without use of custom code. If you’re a developer, the following screenshot should give you a clue as to why code won’t be required too often (with one of my favorite options highlighted):

It’s incredibly powerful, and it’s a good idea to understand what it can do.

Understanding the deal with search-based solutions

As the name suggests, the Content Search web part is powered by SharePoint’s search function. As such, there are the following considerations:

• The CSWP can be configured to “see” items anywhere in SharePoint (potential advantage)
  • In contrast, the CQWP and related SPSiteDataQuery can only search within the current site collection - the site collection “boundary” is a factor
• Results shown are not guaranteed to be 100% up-to-date (potential disadvantage)  
  • Since a search crawl has to run before any content changes will be shown in search results (remember this can include titles, summaries, images and so on for pages/documents), if a user creates/edits an item it will not be shown immediately. This can be a critical point.
  • Furthermore, my understanding from a FAST engineer is that in SharePoint 2013 there is no longer any means of pushing a document directly into the search index – in previous FAST incarnations including FAST for SharePoint 2010, there were options such as docpush.exe for “proactively” add an item to the index, rather than waiting for the next search crawl.
  • That said, it should be possible to obtain much lower indexing latencies in SharePoint 2013 via the “Continuous Crawl'” capability. In most deployments, my guess would be that changes would be reflected within a few minutes at most if this is enabled (where previously you may have had an incremental crawl scheduled every 15, 30 or 60 minutes for a SharePoint sites content source.

Summary - if the functionality you are creating needs fully up-to-date results (e.g. a user has created/edited something and it needs to be *immediately* reflected in the site) then you will probably need to stick with the original approaches (i.e. a query-based rather than search-based solution).

Terminology – new concepts in SharePoint 2013 search

So if we’re going to build solutions built on SP2013 search, we need to have a basic understanding of some concepts – we’ll run into these time and time again:

Concept	My quick definition
Result Source	Like a search ‘scope’ in SP2007/SP2010, but on steroids. Rules are specified to say what the scope consists of – e.g. DOCUMENTS in my TEAM SITES area (constraining on content type and path in this example). Created centrally, or at the web level. Result Sources can be used in just about any search-related functionality, including the Content Search web part.
Query Rule	Like a ‘best bet’ on steroids. Ability to do specially formatted results at top of results list (e.g. Promoted Result) for highly-recommended content. In addition to Promoted Result, we can also do a Result Block (example could be a block of 5 image results within main list of text links). Another option is to Change the Ranked Results – i.e. put something at the top, promote or demote something by 1-10 (previously known as a ‘boost’ in FAST)  LOTS of flexibility in matching the user’s query, including regular expressions and matching terms in the Managed Metadata store.
Display Templates	A Display Template is a JavaScript template (similar to jQuery templates) which controls formatting – in the case of the CSWP, this effectively replaces the use of XSL for look and feel. There is a separate template to pick for the overall control and formatting of an individual item. The .js files for the templates are stored in the ‘Content Web Parts’ subfolder of the Master Page Gallery.  Side note – in the context of a search results page (rather than CSWP), a Display Template is associated with a Result Type (e.g. Word doc, wiki page, PowerPoint file etc.) and so we have granular control over how each is displayed (and when). Extremely cool.

So, lots of flexibility in the search infrastructure. Let's see some of this in the context of the Content Search web part.

Configuring the Content Search web part

There are two main aspects to this:

• Displaying the right items (Search Criteria)
• Look and feel (Display Templates)

In terms of the search criteria, there is *enormous* flexibility in what the CSWP - and the underlying search capability - can do. For one thing, it’s possible to either directly configure the query entirely in the properties of this web part instance (e.g. show me all documents which meet criteria X), and/or start from a pre-existing Result Source to do some of the filtering. Combining the approaches will be fairly common – an example could be “search only on wiki pages” (an OOTB Result Source) but only show items tagged with X (this defined directly in the CSWP properties).

Interestingly, configuring a centralized Result Source and a Content Search web part on a page are very similar, even though it would seem some sort of “reusable scope” and a web part are very different things in SharePoint. The overlap comes because underneath both there is a search query which does the work of isolating the desired results - indeed, as we'll see later the same “Query Builder” UI is used in both places (with a couple of minor differences). So, if you’ve learnt how to configure a CSWP you’ve essentially also learned how to create  a custom Result Source.

Configuring the web part

The first thing to understand is that the Content Search web part appears in different guises in the web part gallery. The ‘main’ web part is in the ‘Content Rollup’ category:

But there are also many pre-configured versions available, each of which finds a specific type of content. This is great for end-users who don’t necessarily think in terms of needing a ‘Content Search’ web part: 

And just to prove the point, the web parts above correspond to the following .webpart definition files in the Web Part Gallery:

Once the web part has been added to the page, it can be configured by it’s tool pane. The main configuration item is the query to use, and this can be started by clicking the ‘Change query’ button:

 
This opens the "’Build Your Query” dialog – this has tabs labeled BASICS, REFINERS, SORTING, SETTINGS and TEST. This thing is known (unsurprisingly) as the Query Builder – what you might not realize, is that it’s used in several places in SharePoint 2013:

• Configuring a Content Search web part (obviously)
• Creating a Result Source (specifically in the Query Transform section)
• Configuring a Search Results web part

There are some differences – for example, when configuring a Search Results web part there is no SORTING tab because this will be handled in the Result Source or the query. I’m going to talk about things from the perspective of the Content Search web part, but will call out any differences for the other usages – so hopefully by learning the CSWP, you also get to learn 75% of the search infrastructure.

BASICS tab – Quick Mode

Although the first tab is labeled ‘BASICS’, I’d say it’s actually the most involved - this is where the query itself is configured, and there is a ‘Quick Mode’ and ‘Advanced Mode’. You’ll also notice that - and let me just say I’d personally be willing to give the Product Manager for this feature A BIG HUG for this - that there’s a “live” results preview pane, permanently visible on the right-hand side of the Query Builder. This shows the first 10 results which would display from running the currently configured search against the current index, without the need to save the web part after each change:

Note that if you create your own query, then this preview pane is only able to show results when you are on the TEST tab. And we’ll talk about that towards the end.

Let’s now walk through the various configuration steps in here.

Select a query

In Quick Mode, the dropdown contains the Result Sources (see my definition above if you’ve forgotten already :)) which come out-of-the-box with SharePoint 2013 – one of these may provide a good foundation for what you need:

 
As you select a Result Source from the dropdown, other options may become available lower down. So if I want to find items matching a specific content type, I get this:

 
In fact, this option to restrict by content type appears for many of the pre-defined Result Sources, not just “Items matching a content type” – which makes sense, because it’s a common thing to include as a filter. Similarly, “Items matching a tag” and several other queries give this interface for selecting a tag to filter on:

And, happy days, if I specify the tag by typing one I get auto-complete to help me pick the term – this is a fully-fledged Managed Metadata input field. Consequently there’s also full validation of the terms you type-in (though this takes a few seconds to show), so if an author accidentally enters something which isn’t a known term, he/she should spot the mistake immediately:

Consider also that those middle options of using the navigation term associated with the current page is exactly what’s needed to build many types of ‘related items’ functionality – again, no code needed now.

Restrict results by app

In the next section, I can restrict the scope of the results to a particular location (e.g. the current site). This enables me to get something like the Content Query web part behavior of only searching within the current site collection if needed – because although we now have the power, it won’t always make sense to go across the entire farm :)

Add additional filters

In the next section I can supplement the query with any valid query text, e.g. a property filter. In this example, I’m adding a filter to only present items which were created by the current user:

Sort results

When we scope our query to a pre-defined Result Source (as we are here in the CSWP ‘Quick Mode’), then sorting is usually pre-defined at that level. The CSWP does give us the opportunity to override sorting based on based on some popularity ranking models (around most viewed/most clicked) instead though – expect proper wording to appear in this dropdown in the RTM version, but you get the idea: 

 
So what happens if none of the options presented so far do what you want? An example could be wanting to use an existing Result Source (e.g. ‘wiki pages’) but sort on Last Modified in descending order. Obviously the dropdown above does not allow that. We could create a custom Result Source and implement the query/sorting there, but that only really makes sense if we expect it to be re-used in multiple places.

In these cases, we can click into Advanced Mode (still on the BASICS tab).

BASICS tab – Advanced Mode

In Advanced Mode you basically get to specify the full query text yourself. In my mind, this is like building a solution with the search API in SP2007/SP2010 – I saw many custom solutions (and built several myself) which used the FullTextSqlQuery or KeywordQuery classes to find the right items. SharePoint 2013 makes it much easier to have this full control whilst still piggybacking onto the out-of-the-box web parts – meaning less work and more productivity.

When switching to the Advanced Mode, a couple of things become available:

• A SORTING tab (details later)
• Controls to help you build the query (which you’d previously do essentially by hand in earlier versions), with ‘Keyword filter’ and ‘Property filter’ options. These can be combined as you like, and the resulting query text appears in the textbox at the bottom:

 

Avoid custom code by using tokens

There are many tokens which can be used when building a query in this way – often you might want to pass something into the query, such as a URL (querystring) parameter, the value in a particular field on the page, and so on. Being able to do this unlocks a huge range of possibilities for building solutions. This is where the first image in this article comes from – here’s a reminder:

 

In summary, when using the Advanced Mode of the query builder you should be able to target just about any content in your SharePoint environment.

SORTING tab (Advanced Mode only)

In SharePoint 2010 Enterprise Search, you could only sort by relevance/rank (the normal search engine approach) or date. FAST for SharePoint 2010 had more options (you could sort by a Managed Property). In SharePoint 2013, frankly the sort options alone are enough to blow your mind :)  If you don’t need anything specific around sorting then you can skip this bit, but if you do then here are your options:

First you can sort by way more things than just rank and date:

One thing to note there – I’m unclear as to what makes it into that ‘Sort by’ list and what does not. It’s not Managed Properties as far as I can tell, so although the list is long many options may not be hugely useful. Still, better than before.

Usefully, you can now do multi-level sorting (sort by this, then by that). The ‘Add sort level’ link in the image above adds another row, allowing me to do things like sorting by URL depth (so items higher up in the site hierarchy show at the top), and then by rank (that makes sense, because there’ll be lots of items at the same URL depth so I do need two levels of sorting):

Note that effectively what I’m doing here is building some sort of custom ranking model. This works great if I need something very specific on sorting, but also note SharePoint 2013 comes with several ranking models – the next section allows me to pick from these if I’ve left the ‘Sort by’ dropdown on ‘Rank’, unlike in the image above. This is because all these options are effectively different forms of rank – most are around People Search or popularity:

And for those occasions when the client is telling you that his/her strategic document really has to be on page 1 of the results (but not a Promoted Result/best bet), you have ‘Dynamic ordering’ – you can boost/demote results, including the option to promote to the top:

REFINERS tab

In the context of search, refiners are usually the links on the search engine’s results page (typically in the left nav) which allow the user to further filter the results. So if I do a search for “meeting minutes” and get lots of results, it would be nice to be able to filter by, say:

• Date range
• SharePoint site (since minutes might be stored in individual project sites)
• Author
• ..and so on

However, in the context of the Content Search web part, refiners actually allow you to do this filtering as part of the initial query. The REFINERS tab is effectively a convenience to you, the person configuring the web part – what happens is that a search is performed whilst in edit mode, and all relevant refiners (e.g. managed properties) are presented as available refiners. These can be selected and moved over to the right-hand list:

The effect of this is that a further filter is added to my query. In the example above, this may be easier than using a Property Filter on the BASICS tab – since there I have little support, I just select the property and type the value:

In the REFINERS tab, SharePoint is doing the search for me (as it’s configured so far), and only coming back with values which have been found in the returned results.

SETTINGS tab

The SETTINGS tab controls some high-level options for running the search:

Query rules

Since these can be defined at the parent site or search service, it could be the case that your CSWP gets affected by one of these. As the radio button shows, this can be overridden, but consider that some types of Query Rules may not have an effect anyway - as a reminder (from the table at the beginning), a Query Rule can either:

• Add a promoted result
• Add a result block
• Change the ranked results somehow (by modifying the query)

Out of these 3 actions, 1.5 of them could affect the results of a ‘default’ CSWP. This can be summarized:

Query Rule Action	Will affect CSWP results?
Add a promoted result	Not by default. When a search runs in SharePoint, multiple result sets are returned (e.g. ‘main results’, ‘best bet results’ and so on - in SP2013, the real names for these are ‘RelevantResults’, ‘SpecialTermResults’, ‘PersonalFavoriteResults’ and ‘RefinementResults’.). Although a CSWP can be configured to show any table, the default is ‘RelevantResults’ – and a promoted result gets added to ‘SpecialTermResults’.
Add a result block	Yes if result block is configured to show ‘ranked within core results’ (the default), rather than ‘shown above core results’.
Change ranked results	Yes.

For completeness, here’s the place in the CSWP where you select which search result set to use (e.g. if you want to switch from the default of ‘RelevantResults’:

Options in the Results Table dropdown (shown to the left):

URL rewriting

This one is fairly simple – if results are being returned from a catalog which is using “friendly” URLs, then the CSWP can override this to use the original URLs. It may not always make sense to use rewritten URLs in aggregations outside of the catalog pages, especially if you’ve implemented anything funky there.

Loading behavior

This is useful – specify whether the CSWP web part instance should load in the main page load (default) or in an AJAX manner after the main page has finished. Considering that a CSWP could either be the centerpiece of your landing page or merely some page footer navigation, it’s nice to be able to prioritize in this way.

Priority

Similarly, we can actually specify High, Medium or Low priority for each CSWP instance we use – great for the different usages we will have, although as per the description, note this only has any effect if the search service is overloaded.

TEST tab

The TEST tab is hugely useful – it provides you the ability:

• To see the underlying query text (in Keyword Query Language [KQL]) which has been generated (though it must be edited in other tabs)
• To see the preview when you are defining a query yourself (the preview pane will be empty on other tabs in this scenario)

Which is all great, but at first glance it’s easy to miss some extra functionality – if the ‘Show more’ link is clicked, other information becomes visible including details on any refiners and Query Rules which have been applied. So below I can see that a custom Query Rule I created has indeed been used, so there’s no guesswork on (for example) whether a certain item is actually being promoted or not:

Sidenote - listing items from ONE site/list/library with the Content Search web part

Worthy of a quick note - if all you need to do is roll-up content from one list/library, then you *can* do this with the CSWP – in the query, simply restrict the search using PATH:[URL to document library]. The Query Builder UI helps you do this by providing the ‘Restrict by app’ area: 

N.B. note that one potential gotcha here can be that you need ‘HTTP’ if your sites are browsed on HTTPS but crawled on HTTP (as in my case).

If you do want to filter by site/list/library, consider of course that the good ol’ Content Query web part will work just fine here, *and* you’ll get instant changes as content is changed. What you won’t have, is the Content Search Web Part’s ability to automatically use tokens in the query (e.g. value of current navigation category, value from current user’s profile etc.)

Summary

The Content Search web part is a great tool in the SharePoint consultant’s box of tricks. Configuration may prove quite simple for some scenarios, but there is also huge amount of flexibility and so a certain degree of complexity comes with that. Many advanced scenarios which make use SP2013 search capabilities (such as Result Sources, Query Rules, promoted results and so on) will be possible – knowing the details will help you identify whether the CSWP can be the answer to a particular problem or not.

In this post, we looked simply at “displaying the right items” i.e. the search query aspect. In other posts, I’ll talk about:

• Displaying items
  • Displaying the right data in the Content Search web part

SP2010 Continuous Integration – pt 6: Integrating SPDisposeCheck and other tools into the build

Tools such as SPDisposeCheck are great for developers, but having to remember to run them isn’t! This is yet another reason why having an automated build is a good idea – you get to offload the responsibility for running such tools to the computer, and we all know computers do not forget. It seems a while now since I wrote this article, but the MSDN SharePoint Developer blog finally has my step-by-step guide on adding tools into a TFS automated build. The process I describe works for SPDisposeCheck, but also any other tool that can be called from the command-line.

See Integrating Additional Tools in a SharePoint Continuous Integration Build

Here’s an idea of what your team would see in the build report each morning (after an overnight build has run):

This concludes the article series on Continuous Integration for SharePoint projects. The full list is:

1. Benefits of Continuous Integration
2. TFS 2010 Build installation/config (Kirk Evans)
3. Creating your first TFS build process for SharePoint projects
4. Implementing assembly versioning
5. Using PowerShell to deploy WSPs from build output
6. Running coded UI tests as part of a build
7. Integrating Additional Tools in a SharePoint Continuous Integration Build

SharePoint apps: working with the app web, and why you should

Disclaimer: Just because I’m writing about apps, doesn’t mean I think everything in SP2013 should be developed as an app! I’m focusing on apps simply because it’s a deep area with lots to learn.

1. SharePoint 2013 apps – architecture, capability and UX considerations
2. Getting started – creating lists, content types, fields etc. within a SharePoint app (provisioning)
3. Working with data in the app web, and why you should [this article]
4. Access end-user data (in the host web) from a SharePoint 2013 app
5. Rolling out SharePoint 2013 apps to the enterprise - tenant scope and PowerShell installs
6. Azure is the new SharePoint ‘_layouts’ directory
7. “Host web apps” – provisioning files (e.g. master pages) to the host web
8. ““Host web apps” – provisioning fields and content types
9. Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps)
10. Working with web parts within a SharePoint app

The app web is the default

You might already have come across the idea that SharePoint 2013 apps which host SharePoint components (e.g. lists, content types etc.) put these into an isolated area called the app web. If not, my previous article Creating lists, content types and fields within a SharePoint app (provisioning) provides some details, including some explanation around the diagram below, which summarises an app I’m building:

The main thing to consider about the app web is that it is the default – if you write some Client Object Model (CSOM) code (remember that SharePoint server-side code is forbidden within an app), perhaps to find a list and retrieve the items, in the standard app arrangement it is the app web you are working with. This is because, by default, you’ll be in a page within the app web – meaning your context is the app web, and that’s what is reflected in the ClientContext object you’ll be working with in your code. What happens is that when you go through an app’s “front door” in the Site Contents area, you are redirected to a page in the app web – and the ClientContext object in the CSOM is simply doing what it always does i.e. obtaining the current web as per the URL in the browser address bar.

Consequently, there is nothing special needed to ‘get a reference’ to the app web in your code. As a separate topic, the same cannot be said of the “host web” (the “real” SharePoint site where the app was installed) – in a best practice configuration which has the app domain on a completely separate URL domain to team sites/My Sites etc. (e.g. “http://myCorp-apps.com” vs. “http://myCorp.com” ), you’ll need to use the SP.RequestExecutor cross-domain library which Microsoft supply to access the host web. More on this in the "Working with data in the host web” article later in this series.

Getting the URLs for the app web and host web (within an app)

Within an app, the developer needs to know the URL for the app web and the host web where the app is installed.  The relevant URLs for the app web and the host web are passed as querystring parameters in the URL through to the first page load of your app, when the user arrives there from the app entry point in the Site Contents area. In fact, any time the user hits the home page of your app (e.g. by clicking the breadcrumb link within the app itself), you’ll see them in the browser address bar. So, it’s simply a matter of reading these URL  parameters – there are lots of ways to do this in JavaScript, I use the following jQuery plugin but anything similar will be fine; simply ensure it’s in a JavaScript file referenced by your app pages:

// jQuery plugin for fetching querystring parameters..

jQuery.extend({

    getUrlVars: function () {

        var vars = [], hash;

        var hashes = window.location.href.slice(window.location.href.indexOf('?') + 1).split('&');

        for (var i = 0; i < hashes.length; i++) {

            hash = hashes[i].split('=');

            vars.push(hash[0]);

            vars[hash[0]] = hash[1];

        }

        return vars;

    },

    getUrlVar: function (name) {

        return jQuery.getUrlVars()[name];

    }

});

Then to grab the app web/host web URLs, you would do:

// retrieve passed app web/host web URLs..

hostweburl = decodeURIComponent($.getUrlVar("SPHostUrl"));

appweburl = decodeURIComponent($.getUrlVar("SPAppWebUrl"));

BEWARE: bug in SharePoint 2013 Preview around the host web URL

One thing I notice in the current build (Preview), is that the host web URL only comes through on the first-time page load (app entry) from the host web. When the user is navigating around any pages you create in the app web, if they click on the breadcrumb link to go to the app’s default page, the URL parameter is present but has an incorrect value. It now has the URL  of the app web, instead of the host web. Consequently if you’re trying to look up something in the host web, this code will fail at this point. I’m assuming this has to be a bug, and will file it on Connect very soon (it hadn’t already been logged when I checked).

As a temporary workaround, I’m persisting the value in a cookie on first-time entry, then fetching it from there as needed. This works fine, I’d be happy to share my code if useful to anyone (just leave me a comment below), but it’s not particularly complex and I’m sure you can imagine what it does.

Common tasks with the app web

So if it’s just standard Client Object Model code and nothing special is required to ‘get a reference’ to the app web, then standard JavaScript CSOM code samples are all you need to help you along. As such, a previous article of mine has some basic JavaScript CSOM examples that I often find myself referring to when I can’t remember syntax and whatnot:

• Fetching the title of the current web
• Fetching items from a list/library
• Fetching list items with query (type-ahead)
• Add new list items to a list

Other good sources are:

• MSDN: SharePoint Client Object Model Data Retrieval (skip to the JavaScript parts)
• Vesa Juvonen: How to – SharePoint 2010 – JS client object model and UI advancements

Remember that the JavaScript version of the Client Object Model (which I’m discussing here) is what you’d use in a web page, but if you were in server-side code of some sort (Windows desktop app/Windows Phone app, WCF service, PowerShell script, console app etc.) then you could use the .NET Managed Client Object Model and code in C# for example.

Using the app web vs. the host web

So, working with the app web is pretty simple. In security terms, the app principal (the identity of the app – this is a fully-fledged security identity) has Full Control permission of the app web, so you can accomplish anything you need to, so long as it can be done with client-side code. However, as I pointed out last time, the app web is pretty sparse by default – there’s no real navigation, many of the SharePoint system pages cannot be used (e.g. the Site Contents page, /_layouts/viewlsts.aspx) and obviously there’s a separation between the app and the end-user’s collaboration stuff (lists and libraries) which is back in the host web. Obviously you could customise heavily and maybe even get to something like that feels like a team site, but it would be quite a lot of work – it’s not permitted to actually use a template such as a team site (STS#0) for an app web. Here’s a reminder of what that default user experience looks like:

 

If this isolation of the app doesn’t really suit the requirements, an alternative arrangement could be for the app to provision it’s lists/pages/etc. into the host web. To be clear, this is actually isn’t possible in many scenarios for two main reasons:

• Security – the app cannot mess around with the host web; by default it has very limited read permissions only. This, of course, is the whole point of the isolation and a key benefit of apps for whoever has to plan, maintain and upgrade the SharePoint deployment
• Provisioning model – if you add a Feature to your Visual Studio project to create, for example, a list – then this list gets created in the app web. It is simply not possible to use traditional provisioning approaches against the host web from within an app

However, there is a combination of circumstances where it is possible:

• The app comes with a permission request for Full Control to the host web (which must be granted by the user installing the app for the app to install)
• Provisioning is implemented via the Client Object Model (CSOM)

These constraints mean it might not always be possible to use this approach. For example, Microsoft is unlikely to ever approve an app which requires Full Control of the host web for use in Office 365/SharePoint Online. Similarly, SharePoint administrators deciding whether a publicly-available app is safe to be added their organisation’s App Catalog would most likely take the same view. BUT, it might make sense to fulfil a particular client requirement – if a client wants a certain set of functionality and wants it implemented as an app, then this might be the way.

Doug Ware, a fellow SharePoint MVP and true “developer’s developer”, is doing a lot of thought leadership in this area. The links at the bottom of this article are to his posts, which I think are required reading if you believe this approach is relevant to you. To build an app of this type, it quickly becomes obvious that a library of CSOM provisioning methods (e.g. createField(), createContentType(), createList(), addListItem(), addFile() and so on) is necessary, and Doug is starting a blog series on this topic.

However, I don’t believe this approach is a panacea for challenges around the app model. Here’s why…

…implementing an app in the host web might not be a good idea

So we’ve already covered the constraints which could mean it’s not possible in the first place (the need for Full Control permissions in the host web). I also think that:

• The host web approach partly defeats the purpose of apps (a very specific piece of functionality, which runs isolated from the SharePoint environment so there’s no negative impact to platform stability, any future upgrades and user data) 
  • If the app is just for one client, I feel like the benefits of apps are eroded so much you may as well build a farm solution. It will be way quicker, no matter how good you are with JavaScript and the CSOM.
• For many requirements, putting the artifacts in the app web should actually work just fine. If you need to store some list/library data and provide some pages for a front-end, then the fact that this is somewhat separate from the end user’s core data doesn’t really matter.
  • For example, in the somewhat contrived time-tracking app that I’m building, so long as the user interface makes it quick and easy to see how many hours the current user has logged this week, and what his/her target is - I don’t think he/she would care whether the data was coming from a list in the app or host web.
• I’m convinced Microsoft don’t want you to build apps in the host web. I’m mainly basing this on the following observations:
  • The provisioning model is all around the app web
  • It’s not possible to set the app’s start page to be a page in the host web. Notably, all the URL tokens which can be used at the beginning of the start page URL are for the app web (or remote app web in the case of an externally-hosted app). See the MSDN page URL strings and tokens in apps for SharePoint and note the “This token cannot be used in the StartPage element of an app manifest…” statements next to each token which refers to a host web item
  • No SDK samples use this approach
  • No MSDN documentation discusses it (at the time of writing at least)
  • Many of their design goals of the app web would not be met (security, reduction in upgrade complexity for customized environments, etc.)

Effectively, I almost see the idea of an app requesting Full Control permission and then provisioning there as a LOOPHOLE.

Summary

So does that mean that we should never use this approach (and that Doug is wrong)? Do I think that we have to build solutions exactly as Microsoft envisage? Hell no! For starters, Microsoft say that every customization should be built as an app in SharePoint 2013, and as my disclaimer at the beginning of these articles says, I personally don’t agree with that. 

But I do think you should think long and hard before using the “exploitable loophole” of building in the host web. Too many devs fail to see the full Total Cost of Ownership of their solutions - around maintainability (and skills/experience/capability required), long-term stability, performance, re-work required during upgrade and so on - because they’ve long since moved onto the next project/client. I’ve seen extremely impressive solutions (created by very talented devs) slide from being absolutely loved to being a big organizational headache within months. Of course, the original dev wasn’t around to see this – if he was, he probably could have managed perceptions, minimized the problem and perhaps learnt a lesson - but he wasn’t. It’s hard to accept, but some of my solutions have probably gone in this direction too. Now I’m not suggesting Doug is guilty of this – I’m sure someone of that brainpower is able to look after his SharePoint clients’ best interests very well – but there is a  long-term impact around SharePoint dev work which implementers should consider. And I feel that if the app model is suitable in the first place, then implementing an app in the host web (rather than the app web) may be part of the problem, rather than part of the solution.

All this isn’t to say it never makes sense though – as I said earlier, some situations where the SharePoint moving pieces really have to sit next to the other lists/libraries/pages the end user has would be examples. I think Doug is actually saying as much when he talks about Building Traditional SharePoint Collaboration Solutions with the App Model – it’s just that I might be more optimistic that the isolated model works for more scenarios than he is. Time will tell.

Further reading:

• Doug Ware: Building Traditional SharePoint Collaboration Solutions with the App Model
• Doug Ware: Provisioning with the Client Object Model Part 1 – Introduction

Create lists, content types, fields etc. within a SharePoint 2013 app

Disclaimer: Just because I’m writing about apps, doesn’t mean I think everything in SP2013 should be developed as an app! I’m focusing on apps simply because it’s a deep area with lots to learn.

Recently I’ve been building a SharePoint 2013 app to help me learn the framework – specifically an on-premises app which has some SharePoint components, rather than a purely remote app. This series is based on some scenarios which I think are key building blocks for apps – for a given set of requirements, you’ll likely need to do one or more of these things. Frankly,you might need to do 100 other things too, but these are certainly common. Here’s a rough outline for the series (which I’ll probably tweak, and will definitely add to):

1. SharePoint 2013 apps – architecture, capability and UX considerations
2. Getting started – creating lists, content types, fields etc. within a SharePoint app (provisioning) [this article]
3. Working with data in the app web, and why you should
4. Access end-user data (in the host web) from a SharePoint 2013 app
5. Rolling out SharePoint 2013 apps to the enterprise - tenant scope and PowerShell installs
6. Azure is the new SharePoint ‘_layouts’ directory
7. “Host web apps” – provisioning files (e.g. master pages) to the host web
8. “Host web apps” – provisioning fields and content types
9. Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps)
10. Working with web parts within a SharePoint app

You’ll notice that I’m skipping setting up a SharePoint 2013 dev environment for developing apps. I think other folks have done a good job here – here are some good sources:

• How to: Set up an on-premises development environment for apps for SharePoint (MSDN)
• Setting up your App domain for SharePoint 2013 (Mirjam van Olst)

Before we start – understanding the ‘app web’ and security model

If you’re getting into developing apps for SharePoint, then I’m assuming you know something about the separation of apps and host webs. In a nutshell, apps are kept separate from ‘real’ end-user SharePoint sites – any lists and document libraries created by the app, any files and web pages, all live in a separate web application to the host site. When a user clicks on the app, they leave the host site behind and get redirected to a web which was created when the app was installed for this site. Effectively an ‘app web’ gets created for every site the app is installed to, in a structure which mirrors the host sites themselves.

Security is the primary driver behind this architecture – the app itself has Full Control over this specially-created site, but no rights (by default) to host sites. Clearly if a malicious app could delete end-user data, that would be A Very Bad Thing. Microsoft would not want that to happen, and so we have this separation. Additionally, it also serves to provide protection against cross-site scripting (IF the web application created for apps does not use the same domain or a subdomain – it should not), and allows SharePoint to identify the caller for Client Object Model calls.

UPDATE 17 Sept 2012: Actually there are circumstances where the developer *can* choose to have the lists/files/web pages etc. created in the host web rather than the app web. See the second half of the next article in this series Working with the app web, and why you should for discussion of this.

If you need more detail this host/app web split, then Ted Pattison has a nice post (in a great series) at SharePoint apps are isolated with respect to request processing and storage. You might not need to understand everything about this right now, but you shouldn’t ignore it.

Getting started– creating your first SharePoint-hosted app

Nothing too complex here – just fire up Visual Studio 2012 and create a project from the ‘App for SharePoint 2013’ project template:

You’ll then be asked to specify some settings for the app – here we need to tell Visual Studio which local site we’ll be developing against/deploying to, and that we are creating a SharePoint-hosted app:

The project is created, and you’ll see that a bunch of files are added automatically – this is effectively a “Hello World” sample which is useful to get you started. jQuery is referenced, a Default.aspx page is added to the app, and this references a CSS file (App.css) and a JavaScript file (App.js) – the JavaScript file has some CSOM code to fetch the current user’s name and put it in a DIV on the page:

If you hit F5 to deploy this, you’ll see that the result is a pretty empty page which shows your app’s title (not yet converted to a user-friendly name in my case), and after a second or so for the AJAX call to complete, shows your username:

At this point, it’s interesting to consider what this sample page does NOT have:

• Any form of left-navigation/quick launch
• Top navigation (although the more global strip in the chrome across the top exists)
• Breadcrumb (although note the single link back to the host web, ‘Team 2’)
• A Site Actions menu
• A link to a Site Contents page (N.B. this page cannot be viewed in an app – if you try, you’ll get something like “The endpoint /team2/cobsharepointappsmyfirstapp/_layouts/15/viewlsts.aspx is not accessible in the context of a SharePoint App.”

So, it’s a pretty “bare bones” starting point. Now let’s take this towards something a bit more like a real-life app.

The concept - my "learner” time-recording sample app

To help us understand the app framework, let’s say we have to build some kind of time-recording app. Lots of us work in utilization-focused consulting organisations, so the concept of a timesheet is usually all too familiar.  The app I’m going to show won’t win any awards for functionality or design – and it definitely doesn’t provide enough end-user value to consider submitting it to the app store :) It’s purely to help me learn about apps, and I’d rather explore different areas even if the design is impractical or doesn’t really make sense.

Here’s what it currently looks like, though I’ll probably add bits to it over the course of this article series:

 

 

App implementation

In terms of artifacts, some fundamental pieces of the app include:

• Projects list
• Logged time list
  • Uses TrackedTime content type – this defines 4 fields, including a lookup to Projects and a Person/Group field for employee
• Utilisation targets (per employee) list
• Default page
• “My time summary” page – this is effectively a navigation-free version of Default.aspx
• App part (ClientWebPart) to display “My time summary” page in host web

As I discussed in SharePoint 2013 apps – architecture, capability and UX considerations, it’s important to understand (or remember) is that with a couple of exceptions, a SharePoint-hosted app can only provision things into the automatically-created “app web”. This lives under the “apps” web application which must be created in an on-premises scenario. The diagram below tries to illustrate the breakdown between host site and app web – importantly, note that the “Utilisation targets” list in the host web was manually created outside of the app – purely because I wanted my app to include the scenario of working with data in the host web. In other words, you cannot usually get your app to provision a list in the host web. The only exception to this is if the app has Full Control to the host web, and some CSOM code runs to create this list - see Working with the app web, and why you should for discussion of this scenario. Note you can click on the image below to see a larger version:

 

Provisioning lists, content types, site columns etc. within a SharePoint-hosted app

So the good news here is that there isn’t a huge amount of change if you’ve done these commonplace tasks in SharePoint 2010/Visual Studio 2010. That said, there are a couple of things which can throw you off track if you weren’t expecting them. The main difference of course, is that you’re not creating things in the actual site itself – and in the initial stages it’s tricky to know whether your artifacts are being provisioning successfully.

I’m not going to show the beginning-to-end process of deploying the above artifacts, but let’s look the outline. Depending on how you like to work, you might want to start with things like:

• Rename ‘Feature1’ which was automatically added to your VS project, add a description etc.
• Create some folders in your project if you prefer more structure than the default

Creating fields and content types

I created a TrackedTime content type with 4 fields. For my Projects list I didn’t actually define a custom content type – a simple list purely with a Title field sufficed.

1. Right-click on the project in Solution Explorer, then select Add > New Item…
2. From the Office/SharePoint category, select Site Column and name the item accordingly:
   
    
   
3. Visual Studio will add some default XML for you. To define the 4 fields needed by my TrackedTime content type, I ended up with the following XML – note the lookup to the Projects list which is accomplished purely with declarative XML courtesy of the “List=’Lists/Projects’” on the 3rd field:

      1: <?xml version="1.0" encoding="utf-8"?>

      2: <Elements xmlns="http://schemas.microsoft.com/sharepoint/">

      3:   <Field

      4:        ID="{ac8a5625-8769-4ed5-bfce-5ced1bfda792}"

      5:        Name="TimeTrackingEmployee"

      6:        DisplayName="Employee"

      7:        Type="User"

      8:        List="UserInfo"

      9:        Required="TRUE"

     10:        ShowField="ImnName" 

     11:        UserSelectionMode="PeopleOnly"

     12:        UserSelectionScope="0" 

     13:        Group="COB time-tracking site columns" />

     14:   <Field

     15:        ID="{CA358CA2-ED71-49D9-A05F-51C92B504062}"

     16:        Name="TimeTrackingDate"

     17:        DisplayName="Date"

     18:        Type="DateTime"

     19:        Format="DateOnly"

     20:        Required="TRUE"

     21:        Group="COB time-tracking site columns" />

     22:   <Field

     23:        ID="{ED1B7F0F-CACA-45A1-A0DD-8D0E6E604C5C}"

     24:        Name="TimeTrackingProjectName"

     25:        DisplayName="Project/task"

     26:        Type="Lookup"

     27:        List="Lists/Projects"

     28:        ShowField="Title"

     29:        Required="TRUE"

     30:        Group="COB time-tracking site columns" />

     31:   <Field

     32:        ID="{62D635BE-6481-4E2A-A6B6-4CDA40EAFB00}"

     33:        Name="TimeTrackingDuration"

     34:        DisplayName="Duration (hours)"

     35:        Type="Number"

     36:        Decimals="1"

     37:        Required="TRUE"

     38:        Group="COB time-tracking site columns" />

     39: </Elements>

4. To make a content type using these fields, I then selected Add > New Item… and selected Content Type and named appropriately:
   
   
   
    
5. The next step is to add the fields we created to the content type. Visual Studio 2012 has a cool new Content Type Designer which appears here – simply start typing the name of the field, and VS scans fields from the SharePoint site it is connected to PLUS fields defined in the VS solution:
   
   
   Any fields added here will then be referenced as FieldRef elements in the content type XML behind the designer.
   
   
6. Now we’re ready to provision a list based on this content type/set of fields. The initial steps are the same as you might be used to, but again we have a handy VS designer to help. Firstly we need to do an Add > New Item… and select List, then name appropriately:
   
   
    
7. Now we see the new List Designer – if we wanted to add fields directly to the list rather than add a content type, we could use the main area of the Columns tab, but usually we’ll want to hit the ‘Content Types’ button:
    
8. In here, we can type the name of a content type already in the SharePoint site or within our VS solution:
   
   
   
9. Usually we’d also want to delete the Item and Folder content types, to leave just our custom content type with the fields:
   
   
   
10. The ‘Views’ tab within the list designer can then be used to add any custom views – in my case, I added one called ‘By employee’:
    
     
    N.B.  Unfortunately the designer doesn’t support all the things you might want to do, so in my case I had edit the XML by hand (the horror!) to actually implement the grouping. Still, this is all WAY better than we had before with SharePoint 2010 development.
    
    
11. The final step is to ensure we’re happy with the top-level properties for the list, such as the title and URL. Take note of the URL the list will be provisioned to, you’ll need this soon:
    
     
12. If we’re happy with the list, we can then test our app by hitting F5. When we click through to the app from the Site Contents page, we’ll be back here:
    
    
    At this point, the big question is…

“Where’s my list?!”

Of course, since there’s no default navigation in an app, it’s not immediately obvious if the list has been provisioned or not. However, if you’ve done the right things you’ll find your list is actually there. It’s tempting of course to try typing /_layouts/viewlsts.aspx in the URL bar, but as I mentioned earlier that’s not a page which can be used in an app.

Instead, just paste the URL of the list directly into the browser address bar after the app URL and you should be taken to your newly-provisioned list:

If you do not wish the list to be “hidden” (i.e. you want users to use it as a regular list), your next step will be to think about navigation – and what you’ll add to that default page so that users can get to the list.

Summary

So we now have a list which our app could use to store data. We could either allow end-users to access the list through the UI, use CSOM to read/write to it, or a combination of both. If you’ve created lists, content types etc. from Visual Studio in SharePoint 2010, you should find the process almost exactly the same – the main differences are the new VS designers and the fact that we’re provisioning to the app web.

Next time we’ll look at working with data in the app web.