Thursday 22 April 2021

SharePoint Syntex - teaching AI to extract contents of structured documents with Form Processing

In previous articles on SharePoint Syntex I've talked mainly about the document processing approach - in this post I'll discuss it's counterpart, form processing. For those following along, my overall set of articles on this theme so far are:

Syntex - document processing

Syntex - general
That last article in particular is designed to help you understand the difference between the two models and when to use each one. As you read about Syntex you might form the view that "form processing is for things like invoices and order forms and document understanding is for everything else" - certainly some of the guidance infers this. However, that position is far too simplistic - there are differences in licensing, capabilities, supported file types and more - and you'll want to get this decision right to avoid having to rework AI models. My "tips for choosing" article might be helpful since it has a table of differences and details of licensing aspects to look out for. 

But today, we focus on form processing!

Syntex form processing - integrated AI Builder

As the briefest of recaps, Syntex form processing is typically more suited to highly-structured and consistent document formats compared to document processing, that much is true. Since the AI Builder technology within Microsoft's Power Platform is used, there are a few implications to consider:
  • To use, AI Builder credits are needed in addition to Syntex licenses (see AI Builder calculator). However, if your org has 300+ Syntex licenses you receive a generous allowance of 1m credits - this more than gets you started
  • Supported file types include JPG, PNG or PDF - but not Office files
  • Entire tables can be extracted from the document (in contrast to document processing)
  • The model is applied via a Power Automate Flow to the SharePoint document library where your documents reside (i.e. where you create the model from) - but there is no easy way to use this in other locations
In short, it's AI Builder conveniently built into SharePoint document libraries - so you don't have to do the integration or somehow pass each document to the model, it's taken care of for you.

Our invoice format

Before we get started on the process, it's worth seeing the format of documents used in this process. Like many classic examples of this type, they are invoices:

Implementing Syntex form processing

The approach followed here can be summarised as:
  1. Define the information to extract (i.e. teach Syntex what the fields are e.g. "Invoice Reference", "Invoice Date" etc.)
  2. Add documents for analysis
  3. Tag documents (i.e. teach Syntex where to find the relevant content in the document)
  4. Train the model
  5. Test
  6. Use in your document library
In your SharePoint document library, find "Automate" > "AI Builder" > "Create a model to process forms":

You'll see this message alerting you that AI Builder credits are needed:

Give your model a name - I'm using "COB invoice" for now. I want a new SharePoint content type to be created with this name so these documents are easily identified and classified amongst any others:

Syntex then begins to create your AI model:

Once the model has been created we define which information within the document we want to extract:

As the image shows, I start specifying some things I want to extract such as:
  • The invoice date
  • The invoice reference
  • The VAT number
Syntex now allows me to supply a collection of documents to train the model:

I create a new collection of documents for my invoice scenario:

I have some invoices ready to go, so I select those to upload:

Once uploaded I'm ready to analyze!

Once the analysis is complete we move into the tagging phase

The tagging phase

As you move your mouse, Syntex allows you to highlight portions of the document by drawing boxes around identified pieces of text. By doing this, you map them to the fields you defined at the beginning - these appear in a picker for selection, with a checkbox indicating whether you've already mapped this item. So I move through the document teaching Syntex what is the invoice reference, what is the date, the supplier name and so on.

As you can see, Syntex allows me to pick something as granular as an individual word or even character, or expand to pick a phrase or string of characters. Items with a green border are already tagged:

Tables can also be tagged in this way:
Once I'm done tagging I'm presented with a summary of the model, with a list of the fields I've defined:

We're now ready to move into the training phase

The training phase

We start by hitting the Train button:

Once the model has been training you can either run a quick test against a new document (not one used for training) or go ahead and publish it to your SharePoint document library:

Let's go ahead and publish the model. Once I have a published version, any subsequent changes will create a draft - this allows me to test things out (and get them wrong) whilst not disrupting the extraction that's already in place.

Once a model is published, we can go ahead and use it:

This makes the model available for use in a Power Automate Flow, and the person using will need to consent to the connections being used:

The resulting Flow looks like this:

If you're interested in the mechanics, the piece that does the extraction is this - the "Predict" action for AI Builder which links to the model we just created:

The results

So let's go back to the invoice format we are using:

When this file is uploaded to SharePoint, initially it's just any old document:

..but then after a couple of minutes the document is correctly identified and classified as a "COB Invoice" and the values I trained the model for are extracted:

Excellent. Now I can drag in many old invoices and have them properly classified and summarised:

..and after a couple of minutes:


Syntex is hugely powerful in automatically unlocking critical data from documents - it doesn't need to be buried inside any more. At the beginning of this series, we discussed how the best research suggests knowledge workers spend 20-30% of their time just searching for information or expertise, and many of us would recognise that having to open many documents to check their contents can contribute to this. As above, I can build SharePoint document library views so that information is readily-accessible or the view is sorted, filtered or grouped according to extracted information.

These benefits go far beyond search and views though. Having my documents correctly identified means that I can apply security and compliance policies to them, for example a conditional access policy which means employees can't print or download sensitive contracts from an unmanaged device, or a retention policy that means a Master Services Agreement is retained for 6 years. Syntex can drive these approaches so that policies are applied by the AI recognising the document, and this can work across documents of wildly varying formats so long as there's some consistency that a document understanding rule can be applied to.

Being able to automatically extract information also means I can build process automation around my documents, for example if something comes in for a certain region or above a certain value, I can route approval processes or notifications accordingly. There are many possibilities here alone. 

Ultimately it comes down to classification and extraction, and there are so many possible use cases around CVs, proposals, statements of work, RFPs, employee contracts, invoices, sales/purchase orders,  service agreements, HR policies and just about any other document type you can think of. This is democratised AI in action, and it's great to have it so accessible in SharePoint.   

Thursday 1 April 2021

Ignite 2021 announcements summary - Teams, Power Platform, Azure, Security and Compliance

Microsoft announced a lot of changes and enhancements to their cloud products last month at Ignite, their major conference. Keeping up with the firehose of constant change is difficult at the best of times, but the flurry of announcements at these events - not to mention minor updates masquerading as major announcements - only makes it more difficult. You'd be forgiven for missing some key developments frankly. Whilst the "Ignite Book of News" that Microsoft create is useful (link at the end), I find myself needing a more concise summary - something I can reference when talking to my team or with clients. All of which leads me to my "Ignite on a slide" summaries that I'm sharing here.

In this post you'll find slides as images, and a downloadable deck which combines all of them - I cover the following technologies:

  • Teams
  • Power Platform
  • Azure
  • Microsoft Security & Compliance

Feel free to re-use or share.

Microsoft Teams

(Click to enlarge)

Microsoft Power Platform

(Click to enlarge)


(Click to enlarge)

Security & Compliance

(Click to enlarge)

Of course, my summaries are somewhat subjective and you might feel there's something I've missed - but hopefully they're useful somehow.

Download the combined deck


Hopefully these summaries are useful in some form and you're free to use them as you like. When you need more detail on any of these announcements (and you will), I highly recommend using Microsoft's published "Book of News":

Link:  Ignite Book of News