Thursday 5 December 2013

Using CSOM in PowerShell scripts with Office 365

In my recent “Office 365 developer decisions, tips and tricks” talk I mentioned that we’d been doing a lot of “PowerShell with CSOM” work, and this was enabling us to run scripts against SharePoint Online in the same way that we are used to (for on-premises SharePoint). This is becoming a popular approach, but since I got questions on it I thought it worth writing about.

When we first started to work with Office 365, I remember being quite concerned at the lack of PowerShell cmdlets – basically all the commands we’re used to using do not exist there. Here’s a gratuitous graph to illustrate the point:

image

So yes, nearly 800 PowerShell commands in SP2013 (up from around 530 in SP2010) down to a measly 30 in SharePoint Online. And those 30 mainly cover basic operations with sites, users and permissions – no scripting of, say, Managed Metadata, user profiles, search and so on. It’s true to say that some of these things are now available down at site-collection scope (needed, of course, when you don’t have a true “Central Admin” site but there are still “tenant-level” settings that you want to use script for rather than make manual changes through the UI.

So what’s a poor developer/administrator to do?

The answer is to write PowerShell as you always did, but embed CSOM code in there. More examples later, but here’s a small illustration:

# get the site collection scoped Features collections (e.g. to activate one) – not showing how to obtain $clientContext here..
$siteFeatures = $clientContext.Site.Features
$clientContext.Load($siteFeatures)
$clientContext.ExecuteQuery()

So we’re using the .NET CSOM, but instead of C# we are using PowerShell’s ability to call any .NET object (indeed, nearly every script will use PowerShell’s New-Object command). All the things we came to love about PowerShell are back on the table:

  • Scripts can be easily amended, no need to recompile (or open Visual Studio)
  • We can debug with PowerGui or PowerShell ISE
  • We can leverage other things PowerShell is good at e.g. easily reading from XML files, using other PowerShell modules and other APIs (including .NET) etc.

Of course, we can only perform operations where the method exists in the .NET CSOM – that’s the boundary of what we can do.

Getting started

Step 1 – understand the landscape

The first thing to understand is that there are actually 3 different approaches for scripting against Office 365/SharePoint Online, depending on what you need to do. It might just be me, but I think that when you start it’s easy to get confused between them, or not fully appreciate that they all exist. The 3 approaches I’m thinking of are:

  • SharePoint Online cmdlets
  • MSOL cmdlets
  • PowerShell + CSOM

This post focuses on the last flavor. I also wrote a short companion post about the overall landscape and with some details/examples on the other flavors, at Using SharePoint Online and MSOL cmdlets in PowerShell with Office 365

Step 2 – prepare the machine you will run scripts against SharePoint Online

Option 1 - if you will NOT run scripts from a SP2013 box (e.g. a SP2013 VM):

You need to obtain the SharePoint DLLs which comprise the .NET CSOM, and copy them to a folder on your machine – your scripts will reference these DLLs.

  1. Go to any SharePoint 2013 server, and copy any DLL
  2. which starts with Microsoft.SharePoint.Client*.dll from the C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI folder.
  3. Store them in a folder on your machine e.g. C:\Lib – make a note of this location.

CSOM DLLs

Option 2 - if you WILL run scripts from a SP2013 box (e.g. a SP2013 VM):

In this case, there is no need to copy the DLLs – your scripts will reference them in the original SharePoint install location (C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI).

The top of your script – referencing DLLs and authentication

Each .ps1 file which calls the SharePoint CSOM needs to deal with two things before you can use the API – loading the CSOM types, and authenticating/obtaining a ClientContext object. So, you’ll need this at the top of your script:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

In the scripts which follow, we’ll include this “top of script” stuff by dot-sourcing TopOfScript.ps1 in every script below – you could follow a similar approach (perhaps with a different name!) or simply paste that stuff into every script you create. If you enter a valid set of credentials and URL, running the script above should see you ready to rumble:

PS CSOM got context

Script examples

Activating a Feature in SPO

Something you might want to do at some point is enable or disable a Feature using script. The script below, like the others that follow it, all reference my TopOfScript.ps1 script above:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

PS CSOM activate feature

Enable side-loading (for app deployment)

Along very similar lines (because it also involves activating a Feature), is the idea of enabling “side-loading” on a site. By default, if you’re developing a SharePoint app it can only be F5 deployed from Visual Studio to a site created from the Developer Site template, but by enabling “side-loading” you can do it on (say) a team site too. Since the Feature isn’t visible (in the UI), you’ll need a script like this:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

PS CSOM enable side loading

Iterating webs

Sometimes you might want to loop through all the webs in a site collection, or underneath a particular web:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

PS CSOM iterate webs

(Worth noting that you also see SharePoint-hosted app webs also in the image above, since these are just subwebs (albeit ones which get accessed on the app domain URL rather than the actual host site’s web application URL).

Iterating webs, then lists, and updating a property on each list

Or how about extending the sample above to not only iterate webs, but also the lists in each - the property I'm updating on each list is the EnableVersioning property, but you easily use any other property or method in the same way:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

 PS CSOM iterate lists enable versioning

Import search schema XML

In SharePoint 2013 and Office 365, many aspects of search configuration (such as Managed Properties and Crawled Properties, Query Rules, Result Sources and Result Types) can be exported and importing between environments as an XML file. The sample below shows the import operation handled with PS + CSOM: 

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

PS CSOM import search schema

Summary

As you can hopefully see, there’s lots you can accomplish with the PowerShell and CSOM combination. Anything that can be done with CSOM API can be wrapped into a script, and you can build up a library of useful PowerShell snippets just like the old days. There are some interesting things that you CANNOT do with CSOM (such as automating the process of uploading/deploying a sandboxed WSP to Office 365), but there ARE approaches for solving even these problems, and I’ll most likely cover this (and our experiences) in future posts.

A final idea on the PowerShell + CSOM front is the idea that you can have “hybrid” scripts which can deal with both SharePoint Online and on-premises SharePoint. For example, on my current project everything we build must be deployable to both SPO and on-premises, and our scripts take a “DeploymentTarget” parameter where the values can be “Online” or “OnPremises”. There are some differences (i.e. branching) in the scripts, but for many operations the same commands can be run.

Related post - Using SharePoint Online and MSO cmdlets in PowerShell with Office 365

Using SharePoint Online and MSOL/WAAD cmdlets in PowerShell with Office 365

This post is a companion post to Using CSOM in PowerShell scripts with Office 365. As I mention over in that article, broadly there are 3 different flavors to writing PowerShell for Office 365 – exactly what commands you need to run will dictate which you use, but it’s also conceivable that you might use all 3 in the same script. When you start with Office 365, I think it’s easy to get confused between them, or not fully appreciate that they all exist. What I’m thinking of here is:

Flavor

Notes

Examples

When installed you’ll have:

SharePoint Online (SPO) cmdlets These are SharePoint Online specific, and can be identified by “SPO” in the noun part of the cmdlet. 
  • Get-SPOSite (to list your site collections in Office 365)
  • New-SPOSite (to create a new site collection)
 SPO shell
MS Online (MSOL)/WAAD cmdlets These are commands related to an Office 365 tenancy (but not necessarily specific to Exchange, Lync or SharePoint) and can be identified by “Msol” in the noun part of the cmdlet.
  • Get-MSolUser (to list users in your tenancy)
  • Set-MsolUserPassword (to update a password)
 MSO shell
Using SP CSOM in PS scripts The main focus of my other post, Using CSOM in PowerShell scripts with Office 365
  • Activating a Feature in SharePoint Online
  • Updating webs or lists in SharePoint Online
No install needed – you can run this type of scripts in a regular Windows PowerShell command prompt.

 

A note on MSOL/Windows Azure AD cmdlets

You might be wondering why the MSOL cmdlets show “Windows Azure Active Directory..” in the shortcut title (full name is “Windows Azure Active Directory Module for Windows PowerShell”), despite everything else being labelled “MSOnline” or “MSOL”. The answer is that originally these cmdlets were known as the “Microsoft Online Services Module for Windows PowerShell cmdlets”, but since that time Microsoft have introduced Windows Azure Active Directory as a formal service offering. Every Office 365 tenancy is backed by Windows Azure Active Directory (WAAD) – and since the MSOL cmdlets always were focused around directory stuff (managing users/groups, managing synchronization with your Active Directory and so on), these have now been absorbed into the WAAD offering.

Getting started

If you’re a developer or administrator who will be working with Office 365 regularly, I recommend installing the the ‘shells’ for both the SharePoint Online and Office 365 PowerShell commands – you’ll probably need them at some point.

  1. Install PowerShell 3.0 if you don’t already have it. It’s included in the Windows Management Framework 3.0 - http://www.microsoft.com/en-us/download/details.aspx?id=34595
  2. Install the SPO cmdlets - http://office.microsoft.com/en-gb/sharepoint-help/redir/XT102919083.aspx?CTT=5&origin=HA102919087
  3. Install the MSOL cmdlets - http://technet.microsoft.com/en-us/library/jj151815.aspx#bkmk_installmodule

Once installed, you’re ready to start thinking about the “top of script” stuff (e.g. authenticating to Office 365). You’ll find that it’s very similar for both the SPO and MSOL scripts, but a different cmdlet must be run to start the session:

  • Connect-SPOService
  • Connect-MsolService

Script examples – SPO scripts

Authenticating to SharePoint Online to run SPO cmdlets:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

List all site collections in SharePoint Online:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

Recreate a site collection in SharePoint Online:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

Script examples – MSOL/WAAD scripts

Authenticating to Office 365 to run MSOL/WAAD cmdlets:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

Getting all users in in the directory

A simple MSOL example, just for completeness:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

Further reading

Appendix – full lists of SPO and MSOL/WAAD cmdlets

To help you get a sense of the full range of commands in each family (in case you don’t already have them installed), I’m listing them below:

SPO cmdlets

Add-SPOUser                                       
Connect-SPOService                                
Disconnect-SPOService                             
Get-SPOAppErrors                                  
Get-SPOAppInfo                                    
Get-SPODeletedSite                                
Get-SPOExternalUser                               
Get-SPOSite                                       
Get-SPOSiteGroup                                  
Get-SPOTenant                                     
Get-SPOTenantLogEntry                             
Get-SPOTenantLogLastAvailableTimeInUtc            
Get-SPOUser                                       
Get-SPOWebTemplate                                
New-SPOSite                                       
New-SPOSiteGroup                                  
Remove-SPODeletedSite                             
Remove-SPOExternalUser                            
Remove-SPOSite                                    
Remove-SPOSiteGroup                               
Remove-SPOUser                                    
Repair-SPOSite                                    
Request-SPOUpgradeEvaluationSite                  
Restore-SPODeletedSite                            
Set-SPOSite                                       
Set-SPOSiteGroup                                  
Set-SPOTenant                                     
Set-SPOUser                                       
Test-SPOSite                                      
Upgrade-SPOSite          

MSOL/WAAD cmdlets

Add-MsolForeignGroupToRole   
Add-MsolGroupMember          
Add-MsolRoleMember           
Confirm-MsolDomain           
Confirm-MsolEmailVerifiedDomain   
Connect-MsolService          
Convert-MsolDomainToFederated     
Convert-MsolDomainToStandard      
Convert-MsolFederatedUser    
Get-MsolAccountSku           
Get-MsolCompanyInformation   
Get-MsolContact              
Get-MsolDomain               
Get-MsolDomainFederationSettings  
Get-MsolDomainVerificationDns     
Get-MsolFederationProperty   
Get-MsolGroup                
Get-MsolGroupMember          
Get-MsolPartnerContract      
Get-MsolPartnerInformation   
Get-MsolPasswordPolicy       
Get-MsolRole                 
Get-MsolRoleMember           
Get-MsolServicePrincipal     
Get-MsolServicePrincipalCredential   
Get-MsolSubscription         
Get-MsolUser                 
Get-MsolUserByStrongAuthentication
Get-MsolUserRole             
New-MsolDomain               
New-MsolFederatedDomain      
New-MsolGroup                
New-MsolLicenseOptions       
New-MsolServicePrincipal     
New-MsolServicePrincipalAddresses
New-MsolServicePrincipalCredential
New-MsolUser                 
New-MsolWellKnownGroup       
Redo-MsolProvisionContact    
Redo-MsolProvisionGroup      
Redo-MsolProvisionUser       
Remove-MsolApplicationPassword    
Remove-MsolContact           
Remove-MsolDomain            
Remove-MsolFederatedDomain   
Remove-MsolForeignGroupFromRole   
Remove-MsolGroup             
Remove-MsolGroupMember       
Remove-MsolRoleMember        
Remove-MsolServicePrincipal       
Remove-MsolServicePrincipalCredential  
Remove-MsolUser              
Reset-MsolStrongAuthenticationMethodByUpn
Restore-MsolUser             
Set-MsolADFSContext          
Set-MsolCompanyContactInformation
Set-MsolCompanySettings      
Set-MsolDirSyncEnabled       
Set-MsolDomain               
Set-MsolDomainAuthentication      
Set-MsolDomainFederationSettings  
Set-MsolGroup                
Set-MsolPartnerInformation   
Set-MsolPasswordPolicy       
Set-MsolServicePrincipal     
Set-MsolUser                 
Set-MsolUserLicense          
Set-MsolUserPassword         
Set-MsolUserPrincipalName    
Update-MsolFederatedDomain