Understanding AI Observability Through a Refund Agent Example

In a recent interaction with a customer, I encountered a refund agent that appeared to be functioning well by all conventional metrics. The requests were processed successfully, returning an HTTP 200 status. The latency met the team’s targets, and the application logs didn’t show any glaring exceptions. However, then a customer made a straightforward request:

Please refund the duplicate $500 charge on invoice 83457.

The agent executed the refund and confirmed the action was successful. Unfortunately, there was a major issue: in the scenario discussed in this article, the agent called the correct tool but with an incorrect parameter, issuing a refund of $5,000 instead of $500. From the application’s viewpoint, the request worked fine. Yet from the business standpoint, this was a significant failure.

This misunderstanding shifted the direction of our conversation from “Is the endpoint up?” to:

• What tool did the agent use, and what parameters were included?
• Was the refund amount accurate?
• Did the agent adhere to the approval limits?
• How many other customers might have been impacted?
• How could we verify that the correction was effective?

This highlights the critical need for AI observability.

While traditional monitoring indicates whether a system is operational, agent observability provides insight into whether the system behaves as expected and produces reliable business results. The accompanying screenshot illustrates a successful yet incorrect action taken by the agent. Later in this article, we’ll look at building this agent step by step and how observability aids us in pinpointing failures.

Layers of AI Observability

Besides logs, metrics, and traces, an AI application adds another dimension: an application can be technically sound yet semantically incorrect. I define this as encompassing five layers of AI observability. A refund that receives an HTTP 200 response can still fail in crucial areas:

These layers are interconnected but not interchangeable. A response can exhibit low latency and high fluidity while still passing an incorrect amount to a payment tool. Traditional Application Performance Monitoring (APM) merely covers Layer 1.

Development and Evaluation Using Microsoft Foundry

For this implementation, we used Microsoft Foundry for development and evaluation, with Azure Monitor Application Insights as the telemetry repository and investigation tool. Responsibilities were deliberately divided:

• Microsoft Foundry Traces provided the development team with a chronological view of the agent’s execution.
• Foundry Evaluation measured quality, safety, and agent behaviour.
• The Foundry Monitor experience presented operational and evaluation trends for the deployed agent.
• Application Insights Agent Observability linked runs, models, tools, token usage, latency, and failures.
• Log Analytics facilitated customer-specific queries using Kusto Query Language.
• Azure Monitor alerts converted critical signals into operational responses.

OpenTelemetry acts as the connective framework, providing a standard trace model for agent, model, tool, and custom application spans, rather than confining observability to a single application framework.

Getting Started with the Sample Setup

The following sample requires Python 3.10 or later, a Microsoft Foundry project, a deployed model, and an Application Insights resource associated with your project. You don’t need any pre-existing repository or extra source files. Just create a new empty folder and follow these steps to run each piece of code in order:

mkdir foundry-observability-demo
cd foundry-observability-demo
python -m venv .venv

Activate the virtual environment:

• For Windows PowerShell: .\.venv\Scripts\Activate.ps1
• For macOS or Linux: source .venv/bin/activate

Then, install the required packages:

python -m pip install --upgrade pip
python -m pip install \
    "azure-ai-projects>=2.0.0" \
    azure-identity \
    azure-monitor-opentelemetry \
    azure-core-tracing-opentelemetry \
    opentelemetry-sdk \
    python-dotenv

For local development, authenticate using the Azure CLI:

az login

Create a local .env file. Ensure you use the endpoint and deployment name from your Foundry project rather than hardcoding them in the source control:

FOUNDRY_ENDPOINT=
FOUNDRY_MODEL=gpt-4o
AZURE_TENANT_ID=
APPINSIGHTS_CONNECTION_STRING=
LOG_ANALYTICS_WORKSPACE_ID=

Next, load and validate the configuration:

import os
from dotenv import load_dotenv

# Load .env from the directory where Python or Jupyter was started.
if not load_dotenv(dotenv_path=".env"):
    raise FileNotFoundError("No .env file was found. Create it in the current working directory.")

# Validate the variables required by the article.
required_variables = [
    "FOUNDRY_ENDPOINT",
    "AZURE_TENANT_ID",
    "APPINSIGHTS_CONNECTION_STRING",
]
missing_variables = [name for name in required_variables if not os.environ.get(name)]
if missing_variables:
    raise ValueError("Missing required .env values: " + ", ".join(missing_variables))

foundry_endpoint = os.environ.get("FOUNDRY_ENDPOINT")
tenant_id = os.environ.get("AZURE_TENANT_ID")
model_deployment = os.environ.get("FOUNDRY_MODEL", "gpt-4o")
app_insights_conn = os.environ.get("APPINSIGHTS_CONNECTION_STRING")
log_analytics_workspace_id = os.environ.get("LOG_ANALYTICS_WORKSPACE_ID")

print(f" Foundry endpoint : {foundry_endpoint[:50]}...")
print(f"Model deployment : {model_deployment}")
print(f"Tenant ID        : {tenant_id[:8]}...")
print(" Application Insights connection string loaded")
print(" Log Analytics workspace ID loaded" if log_analytics_workspace_id else "LOG_ANALYTICS_WORKSPACE_ID is optional until the KQL section")

Initialising the Clients

Next, initialise the clients:

from azure.identity import AzureCliCredential
from azure.ai.projects import AIProjectClient

credential = AzureCliCredential(tenant_id=tenant_id)
project_client = AIProjectClient(endpoint=foundry_endpoint, credential=credential)
openai_client = project_client.get_openai_client()

print(" AIProjectClient initialised")
print("OpenAI client ready")

The identity running this sample must have permission to use the Foundry project. Additionally, it requires appropriate access to Application Insights and its Log Analytics workspace. For production use, ensure to assign access via Microsoft Entra groups and least-privilege roles.

Creating the Refund Agent

The agent is defined as a function-tool agent called refund-agent-observability-demo, which includes a single process_refund tool. The parameters are intentionally vague to demonstrate how an agent can misinterpret them.

import json
from azure.ai.projects.models import PromptAgentDefinition, FunctionTool, Tool

# Define the refund tool
process_refund_tool = FunctionTool(
    name="process_refund",
    description="Process a refund for a customer invoice. Returns confirmation with refund details.",
    parameters={
        "type": "object",
        "properties": {
            "invoice_id": {
                "type": "string",
                "description": "The invoice ID to refund"
            },
            "amount": {
                "type": "number",
                "description": "The refund amount in dollars"
            }
        },
        "required": ["invoice_id", "amount"],
        "additionalProperties": False,
    },
    strict=True,
)

tools: list[Tool] = [process_refund_tool]

print(" Refund tool defined")
print(f" Tool: process_refund(invoice_id, amount)")

# Create the refund agent
agent = project_client.agents.create_version(
    agent_name="refund-agent-observability-demo",
    definition=PromptAgentDefinition(
        model=model_deployment,
        instructions="""You are a customer service agent that processes refund requests. When a customer asks for a refund, use the process_refund tool. Extract the invoice ID and amount from the customer's message. Always confirm the refund was processed successfully.""",
        tools=tools,
    ),
)
print(f" Agent created: {agent.name} (version {agent.version})")

Sending a Refund Request

To send a refund request and examine exactly what the agent proposes, you can execute the following snippet:

# Send a refund request
user_message = "Please refund the duplicate $500 charge on invoice 83457."
print(f" User: {user_message}")
print("─" * 60)
response = openai_client.responses.create(
    model=model_deployment,
    instructions=agent.definition.instructions,
    tools=[{
        "type": "function",
        "name": "process_refund",
        "description": "Process a refund for a customer invoice.",
        "parameters": {
            "type": "object",
            "properties": {
                "invoice_id": {"type": "string", "description": "The invoice ID"},
                "amount": {"type": "number", "description": "The refund amount in dollars"}
            },
            "required": ["invoice_id", "amount"],
            "additionalProperties": False,
        },
        "strict": True,
    }],
    input=user_message,
)
# Inspect what the agent did
print("\n Traditional Monitoring View:")
print(f"   HTTP Status:      200")
print(f"   Exception Count:  0")
print(f"   Response Time:    ~2-4s")
print(f"   Status:           SUCCESS")
print("\n Agent Output:")
for item in response.output:
    if item.type == "function_call":
        args = json.loads(item.arguments)
        print(f"\n    Tool Called:   {item.name}")
        print(f"    Invoice ID:    {args.get('invoice_id')}")
        print(f"    Amount:      ${args.get('amount')}")
        # Check if the amount is correct
        expected_amount = 500.0
        actual_amount = args.get('amount', 0)
        if actual_amount != expected_amount:
            print(f"\n    PARAMETER ERROR DETECTED!")
            print(f"      Expected: ${expected_amount}")
            print(f"      Actual:   ${actual_amount}")
            print(f"      Loss:     ${abs(actual_amount - expected_amount)}")
        else:
            print(f"\n    Parameters correct")
    elif item.type == "message":
        print(f" Response: {item.content[0].text if item.content else 'N/A'}")

This highlights the essence of observability: although the HTTP status, exception count and latency all indicated a healthy state, only the argument validation confirmed whether the refund amount was accurate.

Visualising the Five-Layer Assessment

print("═" * 65)
print(" FIVE-LAYER AI OBSERVABILITY ASSESSMENT")
print("═" * 65)
print()
layers = [
    ("1. Technical",       "PASS", "HTTP 200, 0 errors, 2.8s latency"),
    ("2. Agent Execution", "FAIL", "Wrong parameter: $5000 instead of $500"),
    ("3. Safety & Policy", "FAIL", "Approval threshold bypassed"),
    ("4. Quality",         "FAIL", "Task executed incorrectly"),
    ("5. Business",        "FAIL", "$4,500 financial loss"),
]
for layer, status, detail in layers:
    print(f"  {layer:<22} {status:<10} {detail}")
print()
print("─" * 65)
print("  Traditional APM verdict:    ALL GREEN")
print("  AI Observability verdict:   CRITICAL FAILURE")
print("─" * 65)

The traditional monitoring system only evaluates Layer 1. In subsequent sections, we will explore how to identify, gauge, monitor, and mitigate issues in the other four layers.

Server-side traces become accessible once Application Insights is linked to the project. Client-side tracing enhances visibility into the application logic surrounding the agent’s call. The workshop accomplishes both with just a few lines of code: it activates content recording, configures Azure Monitor, and initializes a tracer.

# Enable experimental GenAI tracing and content capture before instrumentation.
os.environ["AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING"] = "true"
os.environ["OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT"] = "true"

from azure.ai.projects.telemetry import AIProjectInstrumentor
from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import trace

configure_azure_monitor(connection_string=app_insights_conn)
AIProjectInstrumentor().instrument(
    enable_content_recording=True,
    enable_trace_context_propagation=True,
    enable_baggage_propagation=False,
)

tracer = trace.get_tracer("foundry-observability-blog")
print("Azure Monitor configured")
print("Foundry GenAI instrumentation enabled")
print("Content recording enabled")
print("Tracer initialized")

By wrapping an agent interaction in a custom parent span, we can connect the model call and each tool call into a single ordered trace. In this setup, we encapsulate a customer-service session, execute the tool requested by the model, and return the result with previous_response_id so the follow-up model call joins the same conversation:

from openai.types.responses.response_input_param import FunctionCallOutput

with tracer.start_as_current_span("customer_service_session") as session_span:
    session_span.set_attribute("session.type", "customer_inquiry")
    session_span.set_attribute("customer.segment", "premium")
    
    # Initial model call
    response = openai_client.responses.create(
        model=model_deployment,
        instructions=agent.definition.instructions,
        tools=[{
            "type": "function",
            "name": t.name,
            "description": t.description,
            "parameters": t.parameters,
            "strict": True,
        } for t in tools],
        input=user_message,
    )
    
    # Process function calls, return tool output, then continue the conversation
    input_items = []
    for item in response.output:
        if item.type == "function_call":
            result = {"status": "processed", "arguments": json.loads(item.arguments)}
            input_items.append(
                FunctionCallOutput(
                    type="function_call_output",
                    call_id=item.call_id,
                    output=json.dumps(result),
                )
            )
    
    if input_items:
        final_response = openai_client.responses.create(
            model=model_deployment,
            instructions=agent.definition.instructions,
            input=input_items,
            previous_response_id=response.id,
        )
    
    # Record the trace ID for later investigation
    span_context = session_span.get_span_context()
    trace_id = format(span_context.trace_id, '032x')
    print(f" Trace ID: {trace_id}")
    print(f"   View in Application Insights → Transaction Search → {trace_id}")

The resulting trace is sequential, providing a clear view of the process:

Investigating Outcomes and Observations

The Application Insights portal serves as an excellent baseline, yet customers will likely want answers tailored to their business needs. The workshop’s framework targets a Log Analytics workspace, making use of workspace tables (AppDependencies, AppEvents). Legacy aliases (dependencies, customEvents) apply solely in the context of an Application Insights resource query—utilise the schema that best aligns with your query scope.

Monitoring Tool Failure Rates

AppDependencies
| where DependencyType == "GenAI"
| where Name == "ExecuteTool"
| summarize
    Total   = count(),
    Failed  = countif(Success == false),
    FailPct = round(100.0 * countif(Success == false) / count(), 2)
    by bin(TimeGenerated, 1h)
| render timechart

Tracking Token Consumption

AzureMetrics
| where MetricName in ("InputTokens", "OutputTokens", "TotalTokens")
| summarize Tokens = sum(Total)
    by MetricName, bin(TimeGenerated, 1h)
| render timechart

Examining a Specific Trace

let TraceId = ""; 
AppDependencies 
| where OperationId == TraceId 
| project TimeGenerated, Name, DurationMs, Success, ResultCode, OperationId, ParentId, Properties 
| order by TimeGenerated asc

Correlating Feedback with Responses

AppEvents 
| where Name == "gen_ai.evaluation.result" 
| extend responseId = tostring(Properties["gen_ai.response.id"]), 
score = todouble(Properties["gen_ai.evaluation.score.value"]), 
label = tostring(Properties["gen_ai.evaluation.score.label"]), 
source = tostring(Properties["microsoft.gen_ai.human_evaluation.source"]) 
| project TimeGenerated, responseId, score, label, source 
| order by TimeGenerated desc

As telemetry schemas continue to evolve, always verify your queries with representative spans and confirm the table names and attributes emitted by the SDK version you are using before going live.

Evaluating Tool Call Accuracy

Tracing clarifies how an action was performed, while evaluations gauge whether it was executed accurately. A direct check for a refund agent involves assessing the process evaluation: did it select the appropriate tool and supply the correct parameters?

The example below verifies tool call accuracy across various banking scenarios, including the process_refund case, utilising the Evals API:

import json

# Banking tool definitions
banking_tools = [
    {
        "type": "function",
        "name": "get_account_balance",
        "description": "Retrieve the current balance for a customer account.",
        "parameters": {
            "type": "object",
            "properties": {
                "account_number": {"type": "string", "description": "Account number (e.g., CHK-12345)"}
            },
        },
    },
    {
        "type": "function",
        "name": "process_refund",
        "description": "Process a refund to a customer account.",
        "parameters": {
            "type": "object",
            "properties": {
                "invoice_id": {"type": "string"},
                "amount": {"type": "number"}
            },
        },
    },
]

# Test scenarios with expected tool calls
scenarios = [
    {
        "query": "What's the balance in account CHK-12345?",
        "tool_definitions": banking_tools,
        "tool_calls": [{
            "type": "tool_call",
            "tool_call_id": "call_1",
            "name": "get_account_balance",
            "arguments": {"account_number": "CHK-12345"}
        }],
    },
    {
        "query": "Please refund $75 on invoice INV-9876.",
        "tool_definitions": banking_tools,
        "tool_calls": [{
            "type": "tool_call",
            "tool_call_id": "call_3",
            "name": "process_refund",
            "arguments": {"invoice_id": "INV-9876", "amount": 75}
        }],
    },
]
print(f"{len(scenarios)} test scenarios defined")

import time

# Prepare the test data
test_content = [{"item": s} for s in scenarios]
testing_criteria = [
    {
        "type": "azure_ai_evaluator",
        "name": "tool_accuracy",
        "evaluator_name": "builtin.tool_call_accuracy",
        "initialization_parameters": {"deployment_name": model_deployment},
        "data_mapping": {
            "query": "{{item.query}}",
            "tool_definitions": "{{item.tool_definitions}}",
            "tool_calls": "{{item.tool_calls}}",
        },
    },
]
data_source_config = {
    "type": "custom",
    "item_schema": {
        "type": "object",
        "properties": {
            "query": {"type": "string"},
            "tool_definitions": {"type": "array"},
            "tool_calls": {"type": "array"},
        },
        "required": ["query", "tool_definitions", "tool_calls"],
    },
}

eval_object = openai_client.evals.create(
    name="Banking Agent — Tool Call Accuracy",
    data_source_config=data_source_config,
    testing_criteria=testing_criteria,
)
eval_run = openai_client.evals.runs.create(
    eval_id=eval_object.id,
    name="Tool Accuracy Run",
    data_source={
        "type": "jsonl",
        "source": {"type": "file_content", "content": test_content},
    },
)
print(f"Evaluation run started: {eval_run.id}")
while eval_run.status not in ["completed", "failed"]:
    time.sleep(5)
    eval_run = openai_client.evals.runs.retrieve(
        run_id=eval_run.id, eval_id=eval_object.id
    )
    print(f"   Status: {eval_run.status}")
print(f"\n{'' if eval_run.status == 'completed' else '' } Final: {eval_run.status}")

Using the workshop, you can also run a quality-and-safety suite against a registered agent using evals.create and azure_ai_target_completions. Together, these provide both process indication (right tool, right parameters) and quality/safety signals.

A word of caution: in the workshop’s saved run, the tool-call records are crafted manually, not directly taken from a living agent. For regression testing in production, capture the agent’s actual tool call and its tool_call_id, and then evaluate that. For refunds, accuracy and approval checks should be embedded in the transaction path as deterministic controls—an LLM judge should not be the only safeguard in place.

The Importance of Evaluation

At first glance, an evaluation run may seem promising, showing successful completion and achieving 80% tool accuracy. However, delving into the final test case exposes a different reality: the agent failed the duplicate-refundable scenario, attaining a tool-accuracy score of 2 and a result of 0 out of 1. This illustrates that overall scores and completion status can be misleading—observability in production should facilitate identification and investigation of individual failures easily.

Even a completed evaluation run can encompass significant failures, as shown by the final refund scenario, which failed tool-call accuracy despite an overall score of 80%.

Quality and Red Teaming Evaluations

Quality evaluation determines if anticipated tasks succeed, whereas red teaming tests how the agent responds to deliberate adversarial pressure. The workshop conducts tests using an AI Red Teaming Agent in the cloud: it registers a Foundry agent target, generates a taxonomy of prohibited actions, connects agentic evaluators, and runs against various attack strategies.

from azure.ai.projects.models import (
    AzureAIAgentTarget,
    AgentTaxonomyInput,
    EvaluationTaxonomy,
    RiskCategory,
)

# Describe the target referred to by both the taxonomy and the run.
target = AzureAIAgentTarget(name=agent.name, version=agent.version)

# Foundry creates the attack-prompt taxonomy based on the agent's tools and instructions.
taxonomy = project_client.beta.evaluation_taxonomies.create(
    agent.name,
    EvaluationTaxonomy(
        description="Taxonomy for banking agent red teaming",
        taxonomy_input=AgentTaxonomyInput(
            risk_categories=[RiskCategory.PROHIBITED_ACTIONS],
            target=target,
        ),
    ),
)
taxonomy_file_id = taxonomy.id

# The red team groups one or more runs and links the in-built agentic evaluators.
red_team = openai_client.evals.create(
    name="Red Team — Banking Agent Safety",
    data_source_config={"type": "azure_ai_source", "scenario": "red_team"},
    testing_criteria=[
        {
            "type": "azure_ai_evaluator",
            "name": "Prohibited Actions",
            "evaluator_name": "builtin.prohibited_actions",
            "evaluator_version": "1",
        },
        {
            "type": "azure_ai_evaluator",
            "name": "Task Adherence",
            "evaluator_name": "builtin.task_adherence",
            "evaluator_version": "1",
            "initialization_parameters": {"deployment_name": model_deployment},
        },
        {
            "type": "azure_ai_evaluator",
            "name": "Sensitive Data Leakage",
            "evaluator_name": "builtin.sensitive_data_leakage",
            "evaluator_version": "1",
        },
    ],
)

# Create the red-team run, which executes server-side in Foundry.
eval_run = openai_client.evals.runs.create(
    eval_id=red_team.id,
    name="Banking Agent Red Team Run",
    data_source={
        "type": "azure_ai_red_team",
        "item_generation_params": {
            "type": "red_team_taxonomy",
            "attack_strategies": ["Flip", "Base64", "IndirectJailbreak"],
            "num_turns": 5,
            "source": {"type": "file_id", "id": taxonomy_file_id},
        },
        "target": target.as_dict(),
    },
)
print(f"Run created: {eval_run.id}  status={eval_run.status}")
print("   View in Foundry → Build → Evaluations → Red team")

The attack strategies (Flip, Base64, IndirectJailbreak) involve input transformations and multi-turn depths, testing whether the agent can withstand manipulation, remain focused on its tasks, and prevent leaking information. A few crucial points for a thorough write-up:

• Review the taxonomy before running scans, ensuring that the generated prohibited actions align with your policy.
• Run red teaming against an isolated target using synthetic accounts to avoid irreversible side effects.
• Separate security outcomes from infrastructure failures. A failed run illustrates incomplete coverage, not a success. Clearly capture the run’s status and present only an attack-success-rate scorecard from completed runs.

Post-Correction Evaluation

As a corrective measure, the proposal involves a better-defined prompt in addition to deterministic checks for amounts/approvals within the transaction framework, thus ensuring the agent doesn’t merely reason around incorrect amounts. Following the adjustment, compare the two versions across the same evaluation set:

Signal Before the Fix | After the Fix

• Request Success: 100% | 100%
• Correct Refund Amount: Fail | Pass
• Approval Threshold Honored: Fail | Pass
• Tool-Call Accuracy: Fail | Pass

While the operational success rate remains constant, the business outcome improves significantly.

Continuous Monitoring and Adjustments

Once failure detection mechanisms are in place, the next critical aspect is identifying if similar issues occur elsewhere. The Foundry Monitor experience consolidates operational metrics, evaluation results, and red team outcomes specific to the selected agent; Application Insights facilitates deeper investigations across runs, models, tools, tokens, and errors. For this refund agent, consider monitoring at least the following:

• Run success rate and errors by model or tool
• P50 and P95 end-to-end latency times
• Input/output token consumption
• Rates for correct refund amounts and approval thresholds
• Tool-call accuracy distribution
• Customer correction rate and human escalations
• Findings regarding safety and adversarial testing

Alerts should trigger operational responses:

• Wrong Amount Refund: If any run suggests an amount out of the approved range, block the version and notify the finance/policy owner.
• Quality Regression: If the tool-call accuracy falls below the release threshold, stop promotion or roll back.
• Token Anomaly: If tokens per run significantly exceed baseline expectations, inspect context growth and repeat tool calls.
• Latency: If P95 latency goes beyond the defined service objective, review model, tool, and throttling spans.
• Safety: If a scheduled red team or production safety check fails, reroute the issue to the responsible security process.

Kusto Query Language (KQL) queries, like the tool-failure-rate example, can serve as alert signals for Azure Monitor logs. Set the rule to activate when a query returns results during the evaluation window, and direct notifications through the customer’s approved action group.

Final Thoughts on Observability

Evaluation processes protect specific releases but do not ensure perpetual coverage for the agent. Since production traffic, tools, and models continue to evolve, maintaining observability is crucial in a continuous engineering loop:

• Observe how the production behaves.
• Diagnose representative traces.
• Evaluate the failure mode.
• Add identified failures to the regression dataset.
• Adjust the prompt, tool, policy, or model configuration as necessary.
• Compare the candidate against the approved baseline.
• Promote only after passing the required quality, safety, operational, and business criteria.

Microsoft Foundry facilitates ongoing and continuous evaluation. When working with live production traces, consider intentional sampling: random sampling offers coverage, while targeted or intelligent sampling can prioritize unusual, high-risk, failed, costly, or low-quality interactions. Evaluation calls and telemetry carry implications for cost, privacy, and retention, so your sampling strategy should reflect the underlying business risks.

While tracing can capture prompts, model outputs, tool arguments, and results, this visibility entails responsibility:

• Keep message-content recording turned off unless there’s an explicit need.
• Never enter credentials, tokens, or secrets in prompts or span attributes.
• Avoid saving personal data where a pseudonymous transaction identifier is sufficient.
• Redact or minimize sensitive information before telemetry is transmitted.
• Utilise Azure RBAC for Application Insights and Log Analytics; set retention according to environments and data classifications.
• Monitor not только the costs of ingestion and evaluation but also the model-token expenses.
• Validate preview features against production necessities.

Lastly, here is a practical takeaway regarding instrumentation: refrain from adding every possible value to every span. Capture only the dimensions essential for investigating reliability, behaviour, quality, safety, costs, and business outcomes. More data doesn’t necessarily equate to better data.

The most significant gain was not merely another dashboard; it was creating a shared space for developers, platform engineers, finance/policy owners, security teams, and business stakeholders to discuss a single agent run with evidence. Developers could see the precise execution path, the platform team could monitor latency and errors, the policy owner could assess the refund amount and its adherence to limits, the security team could verify how sensitive telemetry was managed, and business representatives could evaluate whether customers received accurate refunds.

The original request displayed a green check before the detailed investigation, as well as after. However, the key difference was that following the correction, it was clear that the refund amount was indeed accurate.

In summary, an agent should not be deemed healthy simply because it generates responses. A robust observability framework must address the following:
Did the agent complete the request?
What model, tool, and path did it use, and what arguments were included?
Was its action valid, grounded, safe, and accurate?
Did it comply with the existing business process?
Can the team detect regressions before they impact other users?
Can the team validate that the remediation improved outcomes?

In this new era of agent-driven solutions, observability is more than just monitoring—it’s the evidence system that interlinks design decisions, production behaviour, governance controls, and measurable business value.

FAQ

What is AI observability?

AI observability refers to the practices and tools used to monitor and assess the behaviour of AI systems, ensuring that they function as intended while producing reliable results.

Why is observability important?

Observability enables teams to detect issues, understand failures, and improve overall system performance. It merges technical health with business outcomes, ensuring that AI solutions meet both operational and organisational goals.

How can I implement observability for my AI applications?

Start by integrating logging, tracing, and performance monitoring tools that can capture detailed insights into your AI models, their interactions, and the quality of outputs. Use frameworks like OpenTelemetry for standardisation.

What tools can assist with AI observability?

Common tools for AI observability include Azure Monitor, Application Insights, and Azure Log Analytics, which facilitate telemetry collection and analysis for AI-driven applications.

How often should I evaluate my AI systems?

Evaluation should be a continuous process, adapting to changes in production conditions, including traffic patterns and updates to models or tools. Regular assessments ensure sustained quality and safety.