Wednesday 25 August 2010

Feature upgrade (part 5) – using PowerShell to upgrade Features

In this article series:

  1. Feature upgrade (part 1) – fundamentals
  2. Feature upgrade (part 2) – a sample to play with
  3. Feature upgrade (part 3) – introducing SPFeatureUpgrade kit
  4. Feature upgrade (part 4) – advanced scenarios
  5. Feature upgrade (part 5) – using PowerShell to upgrade Features (this article)

OK, so I lied in article 4 about this series being finished – I’ve now added PowerShell support to my SPFeatureUpgrade kit, a set of tools designed to help manage the process of upgrading Features in SharePoint 2010. In this article I’ll talk about the PowerShell cmdlet I’m providing, but also some interesting things I learnt about PowerShell along the way – turns out writing a custom cmdlet is a great way to learn some aspects of PowerShell! These might be of interest if, like me, you haven’t done tons of PS to date.

Introduction

There are several reasons why you might prefer to use PowerShell to upgrade Features, rather than the administration pages also in the kit:

  • You have a large farm, and upgrading hundreds of site/web-scoped Features could time out in a web process
  • You’re scripting lots of other things
  • You want to use some advanced logic to upgrade some Feature instances and not others
  • You don’t want to write a C# console app (or similar) to do this, because that would be well, “a bit 2003” :)

As I mentioned in the earlier posts, SharePoint 2010 does not provide any handy cmdlets (or UI) out-of-the-box for upgrading Features, so this is the gap I’m hoping to fill somewhat.

How to upgrade using PowerShell

Install the supplied WSP to get the admin pages (see earlier articles) and the PowerShell cmdlet. Once installed, the command is:

Upgrade-SPFeatures -Scope <SPFeatureScope> [-WebApplication [<SPWebApplicationPipeBind>]] [-Site [<SPSitePipeBind>]] [<CommonParameters>]

Let’s consider some scenarios – the following table shows the command to use to depending on the scope of Features you are upgrading:

Scope

Command

Notes

Farm Upgrade-SPFeatures –Scope Farm All farm-scoped Features requiring upgrade are upgraded.
Web application Upgrade-SPFeatures –Scope WebApplication All web application-scoped Features requiring upgrade are upgraded.
Site

Upgrade-SPFeatures –Scope Site
-WebApplication “http://cob.collab.dev”
---------------------------------- OR -----------------------------------Get-SPWebapplication “http://cob.collab.dev” | Upgrade-SPFeatures –Scope Site

All site-scoped Features requiring upgrade within the specified web application are upgraded.
Web Upgrade-SPFeatures –Scope Web –Site “http://cob.collab.dev/sites/site1”
---------------------------------- OR -----------------------------------Get-SPSite “http://cob.collab.dev/sites/site1” | Upgrade-SPFeatures –Scope Web
All web-scoped Features requiring upgrade within the specified site collection are upgraded.

When the command is executed, if any Feature instances fail to upgrade you will see in the result:

PS_MixedSuccessAndFailure

In terms of the breakdown of commands/parameters, effectively this is the same model my admin pages use – farm and web app Features are upgraded without specifying a filter, but for site or web Features you must specify the parent object. As you can see from the alternate usages for these scopes, the –WebApplication and –Site and parameters can be specified inline or piped-in. Being able to pipe in is useful as it allows you to filter the parent object in order to, say, only upgrade the Features in certain web applications like so:

Get-SPWebApplication | Where-Object {$_.Url -like "http://cob*"} | Upgrade-SPFeatures -Scope Site

Thus this gives you more control over the parent object and therefore what gets upgraded. But what happens if what you’re doing just doesn’t match how my commands are broken down?

Using other logic to selectively upgrade Features

Examples of this might be “upgrade  all web-scoped Features where the web is in this list [‘foo’, ‘bar’]”, or perhaps “upgrade all site-scoped Features where the site has other Feature ‘foo’ activated” and so on. For now, my cmdlet won’t really help you too much in this case so you will need to write some PowerShell which goes straight to the API. For upgrading web-scoped Features, this might look something like:

Function shouldUpgrade([Microsoft.SharePoint.SPFeature]$aFeature)
{
    # add whatever testing logic here..
    if ($aFeature.Definition.Name -eq 'SomeName')
    {
        return $true
    }
    return $false
}
 
$site = Get-SPSite "http://cob.collab.dev/sites/featuretesting" 
$featuresRequiringUpgrade = $site.QueryFeatures([Microsoft.SharePoint.SPFeatureScope]::Web, $true)
 
foreach ($feature in $featuresRequiringUpgrade)
{
    if (shouldUpgrade($feature) -eq $true)
    {
        try
        {
            $feature.Upgrade($false)
            Write-Host ("Upgrade of Feature '" + $feature.DefinitionId + "' with parent '" + $site.Url + "' succeeded")
        }
        catch 
        {
            Write-Host ("Upgrade of Feature '" + $feature.DefinitionId + "' with parent '" + $site.Url + "' failed")
        }
    }
    else
    {
        Write-Host ("Not upgrading this Feature")
    }
}

Obviously this example would need amending for other scopes, proper error reporting etc. Clearly, unless you’re very comfortable cutting PowerShell, having a cmdlet which contains most of the logic does make things easier. So at some point I’ll extend my cmdlet to somehow allow you to filter on which Feature instances to upgrade rather than upgrading all of them - perhaps by specifying a list in an XML file or similar. I could easily get PowerShell to ask you to confirm for each item in the loop, but obviously that doesn’t scale when large numbers of Feature instances (e.g. web-scoped Features) are involved and is no good for unattended use, so that doesn’t feel like the way to go.

Using –WhatIf to see what *would* be upgraded

My cmdlet fully supports the PowerShell –WhatIf switch – this can be very useful as it allows you to see what Feature instances would be upgraded without actually doing anything. In the image below I can see all of the webs in my site collection where my Feature would be upgraded:

 Upgrade-SPFeatures_WhatIf

Future enhancements

There are a couple of shortcomings which spring to mind in the current implementation. Actually an experienced PowerShell eye would probably find lots (feedback welcome, help me learn!), but I do know of these:

  • No ability to filter which Feature instances are upgraded (as discussed earlier)
  • Doesn’t return an appropriate object to the pipeline for downstream processing
    • Actually I only gained a real understanding of this after I’d uploaded to Codeplex. I’m happy to leave it for now, but will improve this in the next version.

The code

The cmdlet and the admin pages I blogged about previously can be downloaded from http://SPFeatureUpgrade.codeplex.com. Here’s what my current code for the cmdlet looks like:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Management.Automation;
using System.Text;
using Microsoft.SharePoint;
using Microsoft.SharePoint.Administration;
using Microsoft.SharePoint.PowerShell;
 
using COB.SharePoint.Utilities.FeatureUpgradeKit.Core;
using COB.SharePoint.Utilities.FeatureUpgradeKit.Core.Exceptions;
using COB.SharePoint.Utilities.FeatureUpgradeKit.Entities;
 
namespace COB.SharePoint.Utilities.FeatureUpgradeKit.PowerShell
{
    /// <summary>
    /// Provides a PowerShell cmdlet for upgrading Features in SharePoint.
    /// </summary>
    /// <created by="Chris O'Brien" date="20 August 2010" />
    [Cmdlet("Upgrade", "SPFeatures", SupportsShouldProcess=true)]
    [SPCmdlet(RequireLocalFarmExist=true, RequireUserFarmAdmin=true)]
    public class SPCmdletUpgradeFeatures : SPCmdlet
    {
        [Parameter(
           Mandatory = true,
           HelpMessage = "Scope of the Features to upgrade.")]
        public SPFeatureScope Scope
        {
            get;
            set;
        }
 
        [Parameter(
           ValueFromPipeline=true,     
           HelpMessage = "Value of the SPWebApplication whose Site-scoped Features should be upgraded.")]
        public SPWebApplicationPipeBind WebApplication
        {
            get;
            set;
        }
        
        [Parameter(
           ValueFromPipeline = true,
           HelpMessage = "Value of the SPSite whose Web-scoped Features should be upgraded.")]
        public SPSitePipeBind Site
        {
            get;
            set;
        }
 
        protected override void InternalValidate()
        {
            if (((Scope == SPFeatureScope.Farm) || (Scope == SPFeatureScope.WebApplication)) && (Site != null || WebApplication != null))
            {
                ThrowTerminatingError(new InvalidFeatureUpgradeDetailsException("When the -Scope parameter is Farm or WebApplication, no parent object (e.g. Farm, Web application, Site or Web) should " +
                    "be passed. Alternatively, specify Site or Web for the -Scope parameter and pass the appropriate parent object."), ErrorCategory.InvalidArgument, null);
            }
            if (Scope == SPFeatureScope.Site && WebApplication == null)
            {
                ThrowTerminatingError(new InvalidFeatureUpgradeDetailsException("To upgrade Site-scoped Features, the parent web application must be piped in or specified using the -WebApplication parameter."), ErrorCategory.InvalidArgument, null);
            }
            if (Scope == SPFeatureScope.Web && Site == null)
            {
                ThrowTerminatingError(new InvalidFeatureUpgradeDetailsException("To upgrade Web-scoped Features, the parent site collection must be piped in or specified using the -Site parameter."), ErrorCategory.InvalidArgument, null);
            }
            base.InternalValidate();
        }
 
        protected override void InternalProcessRecord()
        {   
            SPFeatureQueryResultCollection featuresForUpgrade = null;
            List<SPFeature> featuresForUpgradeList = null;
            SPWebApplication webApp = null;
            SPSite site = null;
 
            switch(Scope)
            {
                case SPFeatureScope.Farm:
                    featuresForUpgrade = SPWebService.AdministrationService.QueryFeatures(Scope, true);
                    featuresForUpgradeList = featuresForUpgrade.ToList<SPFeature>();
                    if (featuresForUpgradeList == null || featuresForUpgradeList.Count() == 0)
                    {
                        WriteResult(string.Format("No Features found requiring upgrade at scope '{0}'.", Scope));
                    }
                    break;
                case SPFeatureScope.WebApplication:
                    featuresForUpgrade = SPWebService.QueryFeaturesInAllWebServices(Scope, true);
                    featuresForUpgradeList = featuresForUpgrade.ToList<SPFeature>();
                    if (featuresForUpgradeList == null || featuresForUpgradeList.Count() == 0)
                    {
                        WriteResult(string.Format("No Features found requiring upgrade at scope '{0}'.", Scope));
                    }
                    break;
                case SPFeatureScope.Site:
                    webApp = WebApplication.Read();
                    
                    featuresForUpgrade = webApp.QueryFeatures(Scope, true);
                    featuresForUpgradeList = featuresForUpgrade.ToList<SPFeature>();
 
                    if (featuresForUpgradeList == null || featuresForUpgradeList.Count() == 0)
                    {
                        WriteResult(string.Format("No Features found requiring upgrade at scope '{0}' with parent web application '{1}'.", 
                            Scope, webApp.GetResponseUri(SPUrlZone.Default)));
                    }
                    break;
                case SPFeatureScope.Web:
                    site = Site.Read();
                    featuresForUpgrade = site.QueryFeatures(Scope, true);
                    featuresForUpgradeList = featuresForUpgrade.ToList<SPFeature>();
 
                    if (featuresForUpgradeList == null || featuresForUpgradeList.Count() == 0)
                    {
                        WriteResult(string.Format("No Features found requiring upgrade at scope '{0}' with parent site collection '{1}'.", 
                            Scope, site.Url));
                    }
                    break;
            }
 
            if ((featuresForUpgradeList != null) && (featuresForUpgradeList.Count() > 0))
            {
                List<FeatureDetails> featureDetailsList = getFeatureDetails(featuresForUpgradeList);
 
                if (ShouldProcess(string.Format("Scope = {0}", Scope)))
                {
                    Dictionary<FeatureDetails, IEnumerable<Exception>> upgradeResults = FeatureManager.UpgradeFeatures(featureDetailsList);
                    presentResultMessages(upgradeResults);
                }
                else
                {
                    string msg = string.Format("Would upgrade the following Features:{0}{0}", Environment.NewLine);
                    foreach(FeatureDetails details in featureDetailsList)
                    {
                        msg += string.Format("Feature Name='{0}', Parent='{1}', ID='{2}'.{3}",
                            details.FeatureName, details.ParentString, details.FeatureID, Environment.NewLine);
                    }
                  
                    WriteResult(msg);
                }
            }
            
            base.InternalProcessRecord();
        }
 
        private void presentResultMessages(Dictionary<FeatureDetails, IEnumerable<Exception>> upgradeResults)
        {
            int errorCount = 0;
            StringBuilder sbSuccessResults = new StringBuilder();
            StringBuilder sbFailedResults = new StringBuilder();
 
            Dictionary<FeatureDetails, IEnumerable<string>> resultOutput = new Dictionary<FeatureDetails, IEnumerable<string>>();
 
            foreach (KeyValuePair<FeatureDetails, IEnumerable<Exception>> kvp in upgradeResults)
            {
                List<Exception> exceptions = null;
                List<string> exceptionMessages = new List<string>();
 
                if (kvp.Value != null)
                {
                    exceptions = kvp.Value.ToList();
                }
 
                if (exceptions != null && exceptions.Count > 0)
                {
                    sbFailedResults.AppendFormat("{2}ERROR: Feature '{0}' with parent '{1}' failed to upgrade with the following errors:{2}{2}",
                        kvp.Key.FeatureName, kvp.Key.ParentString, Environment.NewLine);
                    foreach (Exception exception in exceptions)
                    {
                        sbFailedResults.AppendFormat("  - {0}{1}", exception.ToString(), Environment.NewLine);
                    }
                }
                else
                {
                    sbSuccessResults.AppendFormat("{2}SUCCESS: Feature '{0}' with parent '{1}' upgraded successfully.{2}",
                        kvp.Key.FeatureName, kvp.Key.ParentString, Environment.NewLine);
                }
            }
 
            string failures = sbFailedResults.ToString();
            string successes = sbSuccessResults.ToString();
 
            if (!string.IsNullOrEmpty(failures))
            {
                WriteResult(failures);
            }
            if (!string.IsNullOrEmpty(successes))
            {
                WriteResult(successes);
            }
        }
 
        private List<FeatureDetails> getFeatureDetails(List<SPFeature> featuresForUpgradeList)
        {
            List<FeatureDetails> featureDetails = new List<FeatureDetails>();
            foreach (var featureForUpgrade in featuresForUpgradeList)
            {
                featureDetails.Add(featureForUpgrade.ConvertToFeatureDetails());
            }
 
            return featureDetails;
        }
    }
}

Appendix - some interesting things I learnt about PowerShell

  • The pipeline concept – much like piping elsewhere (UNIX, jQuery spring to mind).
    • This basically allows multiple commands to be chained together into very powerful one-liners. Made possible by the fact that each cmdlet returns an object to the pipeline for the parent script or downstream cmdlets to process.
  • Passing parameters to PS functions – the syntax doesn’t separate parameters with a comma, which no doubt always fools newbies!
    • So an example from a script I wrote yesterday: 
      SetPropertyOnWebApp $webApplication "portalsuperuseraccount" $objectCachePortalSuperUser
  • The SharePoint PowerShell framework comes with custom ‘PipeBind’ objects for many SharePoint objects – using these for your parameters allows the SharePoint object to be identified in different ways (e.g. SPSite by URL or GUID)
  • Implementing Should-Process/-WhatIf in a cmdlet – thanks to Shay Levy (PowerShell MVP) for helping me with this :)
    • If your cmdlet has a property of ‘SupportsShouldProcess=true’ on the Cmdlet attribute, callers can use the –WhatIf switch. You can test for this by calling the ShouldProcess method on the base Cmdlet class – it will be false if –WhatIf was used.
    • A related approach is used when your cmdlet should ask for confirmation (-Confirm) before making a change/writing data
  • Parameter sets in a cmdlet
    • So long as the caller specifies at least one unique optional parameter, you can identify which ‘set’ of parameters was used and do conditional logic on this. This is akin to method overloading in C#.

A great source of learning for SharePoint/PowerShell development is of course, Gary Lapointe – the source code for all his cmdlets is published on his blog.

9 comments:

brad said...

Hi Chris,

Nice articles, one gotcha or additional scenario you may want to test /blog is around deleted sites containing custom features you have written, that would qualify for upgrading and then calling QueryFeatures. Why? well read on...

Supposing you have a custom feature activated on all your sites, you then delete the site, with featues still activated e.g. a users personal site collection in their mysite. We have found that when you use queryfeatures to find all sites with feature id xyz, that needs upgrading it still finds the deleted sites, then when you try to iterate round them you get an .Net exception as the site with id abc does not exist, even though it found the feature within this deleted site. You have to run the SP2010 Gradual Site delete job first, to remove the sites marked for deletion before using the query features. This Timer job is currently set to run daily, so it may not work the day you try it but after a good nights sleep and you come in the next day it might work :) Not sure if i explained it that well, but something to consider.

Brad

Chris O'Brien said...

@Brad,

Very interesting, thanks. Am I right in thinking that the exception is returned by QueryFeatures() in the IEnumerable collection for that Feature instance? Or does the whole operation blow up?

If you use my page to upgrade, the message will show in red text with a red border if it's the former. I'll also take a look myself.

Awesome info, thanks!

Chris.

brad said...

Hi Chris,
The exception was raised as soon as you tried to do a rs.movenext on the ienumerable collection. I was not actually using your tools / powershell etc, was just doing some stuff at work and testing out the upgrade ability and hit this issue so it was really a watch out you may hit the same..
Brad

Chris O'Brien said...

Cool, thanks..

DFelix said...

Hi Chris,

I am trying to get the list of features enabled i.e when we navigate to site settings > site features or in CA when seeing web application service connections. This i am trying to pull through SP Object models. I am seeing a lot of features but not the one in the site settings page or VA. Kindly provide your thought on it.

Daniel

Chris O'Brien said...

@Daniel,

Actually I use PowerShell for that too, since it allows you to see Features which are set to hidden in the UI. For example for a web, I'd use:

Get-SPFeature -Web $web | Out-GridView

[where $web is a variable representing the web you're interested in]

HTH,

Chris.

Rajesh Agadi said...

Hi Chris,

All of your 4 series of related articles are great and appriciate the effort getting it out to the community.

Like who come here to these pages, I was exploring the FeatureUpgrade option to sneak through my custom upgrade activities along with the the default OOTB SP 2010 Upgrade process. Meaning all of my custom FeatureUpgrade actions would have run at the same time as the content upgrade was running. I wanted to avoide the second round of process the content upgrade post the OOTB upgrade process.

Reading your blogs, it sounds like, the FeatureUpgrade call will not be auto observed along with the Content Upgrade, but it will be outside the scope of Content Upgrade job and that it will require an additional API call FeatureUpgrade() to be made.

Considering above scenarior, the limitations of FeatureUpgrade which is only bound to a feature and the lack of deeper and broader support with SP 2010 PowerShell, I might as well right a c# console app to have the full control to post process my upgraded sites.

What are your thoughts?

Regards
Rajesh

Chris O'Brien said...

@Rajesh,

Great question. Actually I haven't done much work around full upgrade (e.g 2007 -> 2010), so haven't really considered feature upgrade as part of an overall upgrade process. I can tell you that PSConfig *does* trigger the feature upgrade process - perhaps the reason for this is that it *will* slot into a full upgrade process. You'd need to test the behaviour around this though - unfortunately I haven't so cannot give any guidance.

In other terms, I prefer feature upgrade to console applications for many reasons. I like the 'documentation' aspect of all developers on the team being able to see the upgrade steps/logic (rather than the code being buried far away from related Features). Also, the QueryFeatures methods provide an easy way to roll out changes, since they efficiently provide collections of sites/webs/whatever which require upgrade (as opposed to iterating over every item to *check* whether it requires upgrade). Finally, feature upgrade shortcuts such as AddContentTypeField can simplify certain tasks.

The one consideration is that PowerShell should generally be used as the trigger for upgrades which could take some time to process.

HTH,

Chris.

Matt said...

Hi Chris

I appreciate this content must take a lot of time and effort to publish so thank you hardly sounds enough to express my gratitude for helping me to understand this (and so many other) SharePoint concepts - but thanks anyway.

Related to Brad's point about deleted sites I had that exact same problem but running the Gradual Site Delete timer job didn't resolve it and I finally realised why. My client is running an SP1 environment so their deleted site collections are sitting in the recycle bin and have to be explicitly removed. I found Chak's article really helpful to get the Powershell commands. After permanently removing the site collections from the recycle bin I found that as well as the Gradual Site Delete timer job I also had to run the Dead Site Delete timer job.

I had a really tough night because of this issue but today the sun is shining thanks to you and your blog!

Cheers