Evaluating Agentic AI in Microsoft Foundry: Beyond Final-Answer Quality

As AI technology develops from simple assistants to more advanced agents that can utilise tools, the methods we use to evaluate them must also adapt.

In traditional language learning models (LLMs), teams usually focus on whether the end response is relevant, coherent, or based on factual information. However, with agents, that’s just part of a bigger picture.

An agent might generate a seemingly valid answer but could still make mistakes such as selecting the wrong tool, sending incorrect parameters, ignoring the output from tools, not adhering to user guidelines, making unnecessary calls, or failing to complete the task entirely.

This complexity is why Microsoft Foundry breaks down agent evaluations into various layers, offering both system-level and process-level assessments. According to Microsoft, agent evaluators provide a systematic way to measure quality, safety, and performance throughout agent workflows, not just focusing on the final answer.

Learn more about how to evaluate agents here.

Foundry allows for evaluations against various targets:

• Agent — test inputs against a selected agent and assess its responses.
• Model — directly evaluate a model’s performance.
• Dataset — assess outputs that are already stored in a database.

This differentiation is crucial because the workflow for evaluation varies depending on the target. If the target is an Agent, Foundry generates a new response for each test input to evaluate the outcome. Conversely, when working with a Dataset, it assesses the already existing responses.

Find out more about evaluating generative AI apps here.

When it comes to assessing agent systems, I suggest looking at two main angles:

Individual turns for in-depth debugging related to tool usage and response behaviour.

Full conversations to ensure task completion over multiple interactions, assess conversational coherence, and gauge user satisfaction. Microsoft recommends beginning with full conversations using simulated data for controlled testing, and gradually transitioning to real conversations during production.

The primary focus is on the end-to-end outcome.

Key questions to consider include:

Did the agent fulfill the user’s request?

For instance:

“Find black running shoes under £120 in size 9 in Dallas.”

An agent might successfully search for products but fail to check their availability. The response could seem reasonable, yet the task remains incomplete.

Did the agent comply with its instructions, policies, and the constraints set by the user?

If a customer states their budget is £120 and the agent suggests a £129 product without noting the budget breach, it falls short of expected adherence.

Did the agent genuinely grasp what the user wanted?

A technically valid tool usage could still be semantically incorrect. For example, searching for generic sneakers when the user specifically requested running shoes is a clear oversight.

This approach goes beyond mere technical correctness and considers whether the overall interaction is likely to satisfy the user.

The metrics used help answer a straightforward but vital question:

Did the system effectively resolve the user’s issue?

Microsoft categorises these measurements under system evaluation for agent workflows.

This is where agent evaluation becomes particularly intriguing.

Foundry incorporates specific process evaluators designed for agents that utilise tools.

This evaluates whether the agent selected the correct tool for the task.

For example, imagine an agent with the following tools:

• search_products
• check_inventory
• get_promotions
• web search

If a user poses the question:

“Is DailyRun X available in size 9 in Dallas?”

The appropriate action is to employ the inventory tool. If the agent resorts to a web search instead, while the response might still sound plausible, the process is flawed.

This also assesses if the selected tool received correct input parameters.

For instance:

check_inventory( product_id=”DEMO-SHOE-002″, size=”9″, location=”Dallas-TX” )

The agent might select the correct tool but input the wrong size or location. Hence, Tool Selection can succeed while Tool Input Accuracy fails.

The evaluation also covers whether the tool was invoked successfully.

For example, it should assess if:

• the API responded correctly
• there was no time-out
• there were no errors during execution

However, a successful tool call doesn’t guarantee that it was the right one. An API might return a successful status for the incorrect SKU or location.

So,

Tool Call Success measures execution reliability, not semantic correctness.

This checks whether the agent accurately utilised the information returned by the tool.

For instance, if the inventory tool responds with:

{ “available”: false }

but the agent states:

“Yes, size 9 is available.”

While the tool functioned properly and the input was correct, the mishap lies with the agent’s failure to interpret the output accurately.

These evaluations provide a clearer indication of whether the agent followed the correct operational workflow, rather than simply producing a convincing answer.

Even when the process itself is executed correctly, the final output can still lack quality.

Foundry supports a range of quality evaluators, including:

• Relevance
• Groundedness
• Completeness
• Coherence
• Fluency

Groundedness examines whether claims made by the agent are backed up by existing context or evidence. This is especially crucial for enterprise applications, as responses should rely on authoritative sources instead of unverified model memory.

Did the agent cover all essential aspects of the request?

If a user asks:

“Is this available and are there any member discounts?”

then an answer addressing inventory without mentioning the discount inquiry is inadequate.

These evaluators focus on how well communication is conducted.

Microsoft defines coherence around the logical arrangement of ideas while fluency deals with readability, grammar, vocabulary, and clarity.

More details on general-purpose evaluators can be found here.

Different types of agents, such as retail, finance, claims, or operations, each come with specific business needs that generic evaluators can’t fully capture.

Microsoft currently advises using a rubric evaluator as a key measurement when teams want to specify criteria unique to their application, like policy compliance, tool-use accuracy, or communication standards. You can then layer built-in evaluators for broader coverage.

For instance, a retail-agent rubric might outline:

• never claim availability without verifying inventory
• never make up discounts
• never quietly violate a stated budget
• prioritise enterprise catalog tools over public searches
• avoid duplicate or unnecessary tool calls
• clearly identify compromises when no exact match is available

This process starts to resemble business acceptance testing.

Discover more about rubric evaluators here.

Foundry’s evaluation datasets are reusable collections of tests.

They prove beneficial for comparing:

• Prompt V1 vs V2
• one model against another
• changes in tool definitions
• adjustments in orchestration
• release candidates vs production versions

When using Foundry to evaluate an agent against a dataset, it generates a new response for every input and assigns a score. If evaluating a live agent, Foundry overlooks existing responses in the dataset.

This shifts the focus of prompt engineering from:

“This prompt appears better.”

to:

“This version performs better under the same regression suite.”

This method represents a more robust engineering practice.

A comprehensive evaluation strategy embraces multiple data sources:

Golden datasets: Ideal for critical known scenarios and predictable regression testing.

Synthetic data: Useful for broadening coverage and generating edge cases before sufficient production traffic is available.

Full-conversation simulation: Great for testing multi-turn user flows and complete task behaviour.

Production traces: Helpful for analysing real user experiences.

Foundry allows for evaluations directly from Application Insights traces and can assess deployed interactions without needing to replay the original request.

Learn how to evaluate deployed interactions here.

Issues in production can differ significantly from failures during development.

Real users introduce challenges such as:

• unexpected phrasing
• missing information
• contradictory constraints
• unusual sequences of tool usage
• edge cases that synthetic testing might overlook

Foundry can evaluate traces already collected in Application Insights. The trace-evaluation process allows for selecting traces via trace ID or agent filter and includes intelligent sampling to choose a representative sample rather than evaluating every interaction.

This creates a powerful feedback loop:

Run → Trace → Evaluate → Identify failures → Add to regression suite → Fix → Re-evaluate

Microsoft Foundry’s evaluation toolset is particularly valuable as it enables teams to assess not only what an agent communicated but also how it acted operationally, which tools were utilised, if the task was performed successfully, and how that behaviour shifts over time.