Thursday 5 December 2013

Using CSOM in PowerShell scripts with Office 365

In my recent “Office 365 developer decisions, tips and tricks” talk I mentioned that we’d been doing a lot of “PowerShell with CSOM” work, and this was enabling us to run scripts against SharePoint Online in the same way that we are used to (for on-premises SharePoint). This is becoming a popular approach, but since I got questions on it I thought it worth writing about.

When we first started to work with Office 365, I remember being quite concerned at the lack of PowerShell cmdlets – basically all the commands we’re used to using do not exist there. Here’s a gratuitous graph to illustrate the point:

image

So yes, nearly 800 PowerShell commands in SP2013 (up from around 530 in SP2010) down to a measly 30 in SharePoint Online. And those 30 mainly cover basic operations with sites, users and permissions – no scripting of, say, Managed Metadata, user profiles, search and so on. It’s true to say that some of these things are now available down at site-collection scope (needed, of course, when you don’t have a true “Central Admin” site but there are still “tenant-level” settings that you want to use script for rather than make manual changes through the UI.

So what’s a poor developer/administrator to do?

The answer is to write PowerShell as you always did, but embed CSOM code in there. More examples later, but here’s a small illustration:

# get the site collection scoped Features collections (e.g. to activate one) – not showing how to obtain $clientContext here..
$siteFeatures = $clientContext.Site.Features
$clientContext.Load($siteFeatures)
$clientContext.ExecuteQuery()

So we’re using the .NET CSOM, but instead of C# we are using PowerShell’s ability to call any .NET object (indeed, nearly every script will use PowerShell’s New-Object command). All the things we came to love about PowerShell are back on the table:

  • Scripts can be easily amended, no need to recompile (or open Visual Studio)
  • We can debug with PowerGui or PowerShell ISE
  • We can leverage other things PowerShell is good at e.g. easily reading from XML files, using other PowerShell modules and other APIs (including .NET) etc.

Of course, we can only perform operations where the method exists in the .NET CSOM – that’s the boundary of what we can do.

Getting started

Step 1 – understand the landscape

The first thing to understand is that there are actually 3 different approaches for scripting against Office 365/SharePoint Online, depending on what you need to do. It might just be me, but I think that when you start it’s easy to get confused between them, or not fully appreciate that they all exist. The 3 approaches I’m thinking of are:

  • SharePoint Online cmdlets
  • MSOL cmdlets
  • PowerShell + CSOM

This post focuses on the last flavor. I also wrote a short companion post about the overall landscape and with some details/examples on the other flavors, at Using SharePoint Online and MSOL cmdlets in PowerShell with Office 365

Step 2 – prepare the machine you will run scripts against SharePoint Online

Option 1 - if you will NOT run scripts from a SP2013 box (e.g. a SP2013 VM):

You need to obtain the SharePoint DLLs which comprise the .NET CSOM, and copy them to a folder on your machine – your scripts will reference these DLLs.

  1. Go to any SharePoint 2013 server, and copy any DLL
  2. which starts with Microsoft.SharePoint.Client*.dll from the C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI folder.
  3. Store them in a folder on your machine e.g. C:\Lib – make a note of this location.

CSOM DLLs

Option 2 - if you WILL run scripts from a SP2013 box (e.g. a SP2013 VM):

In this case, there is no need to copy the DLLs – your scripts will reference them in the original SharePoint install location (C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI).

The top of your script – referencing DLLs and authentication

Each .ps1 file which calls the SharePoint CSOM needs to deal with two things before you can use the API – loading the CSOM types, and authenticating/obtaining a ClientContext object. So, you’ll need this at the top of your script:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

In the scripts which follow, we’ll include this “top of script” stuff by dot-sourcing TopOfScript.ps1 in every script below – you could follow a similar approach (perhaps with a different name!) or simply paste that stuff into every script you create. If you enter a valid set of credentials and URL, running the script above should see you ready to rumble:

PS CSOM got context

Script examples

Activating a Feature in SPO

Something you might want to do at some point is enable or disable a Feature using script. The script below, like the others that follow it, all reference my TopOfScript.ps1 script above:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

PS CSOM activate feature

Enable side-loading (for app deployment)

Along very similar lines (because it also involves activating a Feature), is the idea of enabling “side-loading” on a site. By default, if you’re developing a SharePoint app it can only be F5 deployed from Visual Studio to a site created from the Developer Site template, but by enabling “side-loading” you can do it on (say) a team site too. Since the Feature isn’t visible (in the UI), you’ll need a script like this:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

PS CSOM enable side loading

Iterating webs

Sometimes you might want to loop through all the webs in a site collection, or underneath a particular web:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

PS CSOM iterate webs

(Worth noting that you also see SharePoint-hosted app webs also in the image above, since these are just subwebs (albeit ones which get accessed on the app domain URL rather than the actual host site’s web application URL).

Iterating webs, then lists, and updating a property on each list

Or how about extending the sample above to not only iterate webs, but also the lists in each - the property I'm updating on each list is the EnableVersioning property, but you easily use any other property or method in the same way:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

 PS CSOM iterate lists enable versioning

Import search schema XML

In SharePoint 2013 and Office 365, many aspects of search configuration (such as Managed Properties and Crawled Properties, Query Rules, Result Sources and Result Types) can be exported and importing between environments as an XML file. The sample below shows the import operation handled with PS + CSOM: 

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

PS CSOM import search schema

Summary

As you can hopefully see, there’s lots you can accomplish with the PowerShell and CSOM combination. Anything that can be done with CSOM API can be wrapped into a script, and you can build up a library of useful PowerShell snippets just like the old days. There are some interesting things that you CANNOT do with CSOM (such as automating the process of uploading/deploying a sandboxed WSP to Office 365), but there ARE approaches for solving even these problems, and I’ll most likely cover this (and our experiences) in future posts.

A final idea on the PowerShell + CSOM front is the idea that you can have “hybrid” scripts which can deal with both SharePoint Online and on-premises SharePoint. For example, on my current project everything we build must be deployable to both SPO and on-premises, and our scripts take a “DeploymentTarget” parameter where the values can be “Online” or “OnPremises”. There are some differences (i.e. branching) in the scripts, but for many operations the same commands can be run.

Related post - Using SharePoint Online and MSO cmdlets in PowerShell with Office 365

14 comments:

Bruce said...

Great stuff! Thanks so much for sharing Chris. It's getting to the point that I start coming to your blog first when I have a question, rather than googling it... not sure there can be higher praise for a tech blog :-)

GuyD said...

Thanks for the great article. One question: "You need to obtain the SharePoint DLLs which comprise the .NET CSOM" is this the same as the Client SDK for SharePoint 2013, that is part of the installation of Visual Studio 2013 Pro?

Chris O'Brien said...

@GuyD,

Yes indeed, and thanks for pointing this out - I'd forgotten about this option. The Client SDK is effectively a nice "packaged" set of these DLLs with an installer. You can either get it with Visual Studio or as a standalone/redistributable installer.

Thanks,

Chris.

Mark said...

A massive thanks for this post. It been a massive help to me.
I am trying to set the amount of major versions at the same time but it says the majorversionlimit doesn't exit for Sharepoint online. You don't happen to know what the property name on 365 is do you?

Thanks
Mark

Chris O'Brien said...

@Mark,

Well, that's definitely the name of the property on the SPList class - if it doesn't work in SPO, it could be that the property is blocked there (first example I've heard of this though).

Other thoughts:-

- Could it be that you have the casing (or spelling) wrong?
- Or could it be that you are using the JavaScript CSOM, and need something like set_MajorVersionLimit?

Cheers,

Chris.

Unknown said...

I am trying to import search settings at tenant level but facing issue as shown below

Exception calling "ExecuteQuery" with "0" argument(s): "Access denied. You do not have permission to perform this action or access this resource."
At \PowerShell Scripts\Import.ps1:38 char:1
+ $clientContext.ExecuteQuery()
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : ServerUnauthorizedAccessException

Can we import customized search settings at tenant level through PowerShell?

Also i was able to import manually through UI at tenant level

Toby said...

@Rohit,

I had to collect the password from the console to prevent that error.

$securePassword = read-host -prompt "Password for $username" -AsSecureString

Great post, Chris!
Toby

MOHIT JOSHI said...

Hi Chris,

Can we create/modify Managed Properties and crawled properties using this approach on SharePoint Online?

Chris O'Brien said...

@Mohit,

Yes, you can do this by using PS/CSOM to import search schema XML which creates/updates the Managed Properties, Crawled Properties, mappings etc.

I have a code sample for this here: https://gist.github.com/chrisobriensp/7779915

Cheers,

Chris.

Anonymous said...

Thanks a lot for this great article.

https://gist.github.com/chrisobriensp/7779915 - This link is broken. Can you please fix the link. Thankyou!!

Chris O'Brien said...

@freddiemaize,

Hmm, I just checked and the link works fine for me? Can you re-check please?

Thanks!

COB.

Unknown said...

This is great stuff! Thank you for posting. However, I'm having trouble iterating over my sites site collection. I need to set up our custom master page for all personal subsites but when i run the command it doesn't return anything. I'm Global Admin on Office 365 environment for our company. Any ideas will be greatly appreciated. Thank you in advance!

Tonyp said...

Very interesting post, thanks for sharing!
What software do you recommend for writing complex powershell CSOM? Is there any software which will include IntelliSense for the included DLL?

Fernando Celaya said...

Awesome guides! Really helping me level up my SPO game :)