Autonomous agents - Technique 1: Getting AI-suitable descriptions right - data, tools, agents themselves

As I opened with in the first article in this series, AI agents are the next generation of solution we'll build for the next few years - this is how work gets automated, at least some of it. Business systems which take away some of the human processing so we can focus on more impactful work, powered by highly-capable LLMs with the ability to consume data, reason and plan, and use tools like web browsers. In this series we focus on agent-building techniques, and in particular what it takes to build autonomous agents successfully i.e. that ability for an agent to dynamically reason and plan. Guidance here orients around Microsoft Copilot Studio agents, though it's interesting to reflect that many of the techniques will apply across nearly all AI agent platforms. 

In this article, we focus on descriptions - in the context of AI agents, this means for data, tools, sub-processes and other things the agent might use to get to it's goal. It's interesting because descriptions have been pretty innocuous in apps/solutions/automations we've built over the last few decades, because much like comments in code, only humans read them. However, things are very different when AI is reading them and making decisions about how to process based on the words provided.

But first, here's a recap of the full series:

Articles in this series

1. Techniques for autonomous agents in Copilot Studio - intro 
2. Scenario video - Microsoft architect with proposal generation
3. Technique 1 - Getting AI-suitable descriptions right - data, tools, agents themselves (this article)
4. Technique 2 - Define explicit steps in agent instructions when "reasoning the process" isn't appropriate
5. Technique 3 - Provide tools like Agent Flows for steps the agent can’t easily handle
6. Technique 4 - Leveraging Power Platform and Microsoft 365 capabilities in your agents
7. Technique 5 - Understand cost, capability, and governance implications of agents you create

Descriptions - why they're suddenly vital

Despite all the advances in the underlying AI models, AI agents are not magic and they need help. This is especially the case when:

• Agents have some autonomy in terms of getting to an outcome (rather than being defined step-by-step, like a coded application)
• Agents can use data and knowledge sources
• Agents can call on tools and sub-processes
• Agents can call other agents (if they understand what they do)

AI needs good descriptions more than humans do

The critical point is that AI agents use descriptions to understand what data to use and when, what tools to call and when, and what sub-processes to run and when. If your descriptions are poor and don't supply enough context, the behaviour of your agent will be unpredictable and won't be as expected.

 

As an example which may hit many agent builders, many agents fail or behave unpredictably because they don’t understand what their knowledge sources or tools are for. This often happens when default or auto-generated descriptions are used — they’re vague, generic, and unhelpful. For example, if you provide a file as a knowledge source for your agent, Copilot Studio will helpfully use AI to generate a description of the a file - but this will essentially be "this knowledge source uses this file" - so when I provide a detailed Excel file of Microsoft 365 license SKUs and capabilities (from the excellent M365maps.com), the descriptions generated today do nothing to help the agent understand what this knowledge represents and how it could be used:

The Fix:

Write clear, detailed descriptions for every knowledge source, tool, and even the agent itself (to support agent-to-agent interactions in the future). These descriptions are used by the orchestration engine to decide what to use and when. 

Let's focus on a knowledge source (i.e. some data used by an agent) in Copilot Studio. In the agent I'm using in this series, one knowledge source is an Excel export of 'technology landscape' data for key Advania clients (redacted for the video in the last post). 

Bad example (this is the default description generated by Copilot Studio):

“This knowledge source searches information contained in Microsoft license coverage by client.xlsx.”

Good example:

Instead, a far better description helps the agent understand what's in the data/knowledge - for example: 

“This knowledge source details the technologies, plans, and current Microsoft licensing in place for key Advania clients. It covers aspects like how many Microsoft 365 E3 and E5 licenses are held, and which technologies are used for Security & Compliance, messaging, endpoint protection, and document management.”

The agent can now understand exactly what's in this data and how it can be used. This context is essential for establishing how it should go about the process and what to use when.

Agent behaviour with poor knowledge descriptions

Before data and knowledge sources are given good descriptions (rather than the AI-generated ones from Copilot Studio), you'll find all sorts of strange behaviours. Things may look good at first because you see the agent referencing the sources:

However, dig deeper and you'll find some sources simply aren't used in the agent's processing:

In fact, what we see is the agent simply referenced one knowledge source and then stopped there - it didn't actually consider the use case it was given or do any kind of reasoning, and while it gives a response that looks rich at first glance, in fact the agent is simply trotting out some generic info related to Microsoft 365 products. We see some of this in the activity map in Copilot Studio which shows what happened when the agent ran:

Agent behaviour with good knowledge descriptions

Fix the descriptions, and suddenly the agent knows how to consider each knowledge source - if we ask the same question to the agent, we see that BOTH knowledge sources are now used. Firstly the agent queries the "Microsoft 365 license SKU matrix" source to consider endpoint protection options:

Next, the agent starts a specific query on Microsoft 365 F1 licensing, which makes sense given this client is referenced in the client data knowledge source as having this restriction for some employees:

Better - but still not right

On the surface, it seems that the agent is coming closer to it's purpose - being an effective tech architect with deep Microsoft knowledge. However, if the agent's final responses are closely analysed at this point we see that:

• Unfortunately, the agent isn't truly considering the passed client requirement (as per the demo video shown in the previous article, I'm asking it to consider a particular client's need to replace endpoint protection along with some granular requirements). Instead, I'm still getting some 'lightweight consideration' and generic Microsoft 365 product info at best
• The agent is also not delivering on another element - I'm not getting a draft proposal document generated, although I'm asking for that in the agent instructions

So in short, the agent is now using data and it's knowledge sources in a more sensible way, but the overall process being followed isn't what we need. We need a step-by-step consideration of the client's need and the granular requirements specific, any Microsoft 365 products or capabilities which might meet these needs, a lookup and consideration of the technologies the client is currently licensed for, and then a genuinely thought through recommendation based on all the above. I won't show the agent output at this stage to avoid confusing things, but suffice to say we have an agent at "first day intern" level at the moment and for this to be valuable we need a lot more cognitive processing.

On to the next challenge - getting agent instructions right

Now our agent understands the data and knowledge it has access to more clearly, we need to turn our attention to how the agent is guided in the instructions we give. If we make some changes here, the agent will go through a step-by-step process and reason over the key questions being asked of it much more effectively. Which leads us to the next article in this series:

Next article

Technique 2 - Define explicit steps in agent instructions when "reasoning the process" isn't appropriate - coming soon