Understanding AI Observability in Microsoft Foundry

In Part 1 of this series, we discussed why AI observability extends far beyond simply monitoring dashboards. Once you have an AI agent in production, the crucial question isn’t whether it can respond. Instead, it revolves around its ability to respond accurately, safely, efficiently, and consistently, especially as your models evolve, prompts change, retrieval indexes drift, and real-world traffic introduces unforeseen edge cases that may not have been caught in tests. This is why Microsoft Foundry’s approach to observability is rooted in three interconnected areas: evaluation, monitoring, and tracing—rather than relying solely on traditional uptime metrics.

This article serves as the practical counterpart to that discussion, remaining within the Foundry (new) experience while focusing on the configuration and operational patterns that truly matter as you transition your agents into production. Here are some key areas we will cover:

• Connecting Application Insights
• Enabling continuous evaluation
• Interpreting the monitoring dashboard like a pro
• Utilising traces to diagnose quality issues
• Setting up alerts in Azure Monitor
• Integrating quality gates in CI/CD
• Implementing guardrails and ensuring compliance
• Working with Defender for Cloud and Microsoft Purview while closely monitoring costs, quotas, and token usage across your projects.

Portal Note: Everything in this article assumes you’re using the Foundry (new) portal. If you don’t see the Build, Operate, and Admin options in your navigation, please switch to the new experience before proceeding.

Getting Started with Application Insights

All effective observability workflows in Foundry hinge on one essential element: Azure Monitor Application Insights. This acts as the storage and correlation layer for telemetry, driving monitoring, tracing, and quality views for production. Skipping this step will leave your experience incomplete.

To confirm that your project is correctly set up, navigate to Operate > Admin > Connected resources. Here, you can check if an Application Insights resource is linked to your project, and add one if it’s missing. Although some specific agent experiences may provide alternative connection points, Connected resources is the most dependable view for verification before moving forward.

Setting this up is simple. Open your project, go to Operate > Admin, select your project, and look under Connected resources to verify the presence of the Application Insights resource. If it’s not there, add it and ensure it’s in the same region as your other project resources. This connection provides the framework for Foundry to emit and correlate the telemetry discussed throughout this article.

Remember: those who will analyse traces or logs later must have access to the relevant monitoring resources. Connecting the resource is just half the battle; ensure that your operations or engineering team can also access Foundry’s outputs.

Enabling Continuous Evaluation

As mentioned in Part 1, evaluation should evolve from an occasional pre-release ritual to a constant production discipline. The portal now supports this: once your project is connected, enable continuous evaluation so production traffic is assessed over time, rather than only during offline tests.

The place to start is the agent’s monitoring surface. Head to Build > Agents, select the agent you want to monitor, and then click on Monitor. This section consolidates operational metrics and evaluation signals for the specific agent, avoiding the need to juggle multiple disconnected tools. If your tenant has the built-in assistant turned on, Ask AI (Preview) can summarise the dashboard insights and highlight any unusual behaviour during your selected time frame.

When adjusting the monitoring settings, you’re not just flipping a switch; you’re defining what “success” looks like for your workload. For general conversational agents, starting dimensions like coherence and relevance are crucial. For agents heavy on retrieval-augmented generation (RAG), parameters like groundedness and retrieval quality quickly become imperative, as they help distinguish issues in model behaviour from those in retrieval. Safety-sensitive operations should also address relevant safety metrics right from the outset, rather than as an afterthought.

Bear in mind: while continuous evaluation is valuable, it does not come without a cost. It’s wise to start with a modest sample size and assess how stable the results are in your environment—then adjust the sample rate based on risk, volume, and budget.

A key point that often gets overlooked: if you’re activating continuous evaluation using production traffic, ensure your project’s managed identity has the necessary permissions to carry out those evaluation processes.

Using the Agent Monitoring Dashboard

The Agent Monitoring Dashboard is often mistaken for just a status page, but its utility is far greater. When used effectively, it becomes a vital tool for correlating quality, latency, failures, token consumption, and potentially safety findings, all in one view supported by Azure Monitor data.

This is important because issues in production AI rarely manifest as singular problems. For example, a latency increase might not be concerning if quality remains steady and the cause is evident. Conversely, a stable latency curve can mask serious regression if groundedness or retrieval quality decrease. Token consumption trends can alert you to cost issues much sooner than financial alerts would. The dashboard’s ability to present these signals alongside one another allows you to determine whether the issue lies solely in quality, operations, or a combination of both.

You can reach this dashboard from Build > Agents > Monitor for a focused agent analysis, or from Operate > Overview for a broader overview across your subscription. Microsoft’s current documentation for Ask AI (Preview) suggests using both of these interfaces for tailored analysis.

One important operator habit to develop is the understanding that metrics indicate something is shifting. If you spot a low groundedness score or notice a spike in errors, initiate a trace investigation rather than simply documenting the dashboard’s findings for future discussion. Think of the dashboard as your initial insight tool, not the full diagnosis.

Diving Deeper with Tracing

Monitoring alerts you to changes, while tracing reveals what actually transpired during a run. This distinction transforms observability into a robust operational system instead of just a series of charts. Microsoft’s tracing model relies on OpenTelemetry, aiming to capture each step in the execution path—model calls, tool interactions, retrieval steps, orchestration decisions, along with the timings and token details tied to each phase.

For Foundry-managed operations, much of this visibility becomes accessible once the telemetry framework is linked. If you’re working with custom agents and mixed stacks, the guidance is straightforward: ensure your agent is registered and observable from the control panel, then dispatch standards-based telemetry to the same monitoring backend. Operationally, think of Foundry as the development interface, while Azure Monitor acts as the storage and querying layer underneath.

A critical technical update relates to the authentication and wiring model. The latest guidance advocates using the Application Insights connection string rather than older configurations based on standalone instrumentation keys. For example, a clean Python implementation today would employ the AzureAIOpenTelemetryTracer from the LangChain integration path, including the connection string.

from langchain.agents import create_agent
from langchain_azure_ai.callbacks.tracers import AzureAIOpenTelemetryTracer
 
application_insights_connection_string = "..."
 
tracer = AzureAIOpenTelemetryTracer(
    connection_string=application_insights_connection_string,
    enable_content_recording=True,
)
 
def get_weather(city: str) -> str:
    """Get weather for a given city."""
    return f"It's always sunny in {city}!"
 
agent = create_agent(
    model="openai:gpt-5.1",
    tools=[get_weather],
    system_prompt="You are a helpful assistant",
).with_config({"callbacks": [tracer]})

For teams using different frameworks, while the code may vary, the basic principle remains the same: emit OpenTelemetry spans and send them to the Application Insights resource connected to your Foundry project. This way, traces, quality signals, and operational telemetry coexist in a single backend instead of getting scattered across various tools and teams.

However, a critical reminder: traces can document prompts, responses, tool arguments, and other sensitive runtime information. In enterprise environments, managing trace retention, role-based access control (RBAC), and ensuring content hygiene are just as important as the completeness of the traces themselves. Ensure that you address the potential for recording sensitive content before scaling your tracing efforts.

Integrating Alerts and Compliance Measures

Foundry helps you identify issues, while Azure Monitor is where you should take action. This separation establishes a sustainable operating model, as it preserves AI-specific insights in the portal while allowing Site Reliability Engineering (SRE) and platform teams to use familiar alerting, routing, and automation tools across all workloads.

Developing an effective alerting strategy doesn’t start with a long list of metrics. Begin with a concise selection of conditions that warrant operator attention. For most agents, this typically includes:

• Long-term increases in latency
• Sudden spikes in run failures
• Unusual token consumption
• Significant drops in quality or safety signals that pose a risk to users

Foundry’s monitoring surfaces can guide you in determining which thresholds are essential, but the rigorous enforcement usually resides with Azure Monitor alerts and action groups linked to the Application Insights resource. This pattern aligns with Microsoft’s observability guidance: the portal aids your understanding of the signals, while Azure Monitor manages and automates responses.

Embedding Observability into CI/CD

The real measure of whether observability is a core part of your engineering culture is its integration into CI/CD processes. If evaluation only occurs post-deployment, you risk facing regression issues too late. Instead, establish a pattern of executing a repeatable evaluation set during the build or release sequence, publish the results, and fail promotions if key business thresholds are not met. This effectively transforms “quality” from a casual discussion into a substantial gate.

Because task and action names can vary over time, the most robust approach is to avoid hardcoding potentially short-lived marketplace extension names. Instead, maintain a stable pipeline design and explicitly call your evaluation script. This way, you offer something for others to implement, whether they’re using GitHub Actions or Azure DevOps.

name: foundry-evaluation-gate
 
on:
  pull_request:
  push:
    branches: [ main ]
 
permissions:
  id-token: write
  contents: read
 
jobs:
  evaluate:
    runs-on: ubuntu-latest
 
    steps:
      - uses: actions/checkout@v4
 
      - uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
 
      - run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
 
      - run: |
          python ci/run_foundry_evals.py \
            --project-endpoint "${{ secrets.FOUNDRY_PROJECT_ENDPOINT }}" \
            --dataset "tests/evalset.json" \
            --output "artifacts/eval-results.json"
 
      - run: |
          python ci/assert_eval_thresholds.py artifacts/eval-results.json

The principle is essential, even if the wrapper varies: use the same evaluation definitions across all stages—development, CI, and production monitoring. A “good” score should have uniform meaning across these different contexts. This consistency was one of the core themes highlighted in Part 1, transforming evaluation into an operational system rather than an isolated testing harness.

Merging “Responsible AI” into Daily Operations

Once your agent is operational, the principles of “responsible AI” must transition from theoretical frameworks into active processes. This is where the Compliance workspace comes into play. In Part 1, we framed compliance through the lens of policy enforcement, guardrails, and remediation across operations. In practice, this means integrating Operate > Compliance into your everyday routines, rather than treating it as a task reserved solely for audit periods.

This workspace provides more than just visibility; it enables organisations to compare their anticipated standards—guardrail controls, security postures, and data governance—against the reality of individual assets’ behaviour. The policy view indicates non-compliance, while the asset perspectives reveal deployment drift. The remediation workflows help rectify issues before they escalate into serious incidents.

Utilise the Operate > Compliance feature to review policy adherence, pinpoint violations, compare necessary controls against actual deployment settings, and address any discrepancies. The specific terminology may change over time, but the operational process will remain consistent.

Expansion into Microsoft’s Security and Governance Ecosystem

While Foundry’s built-in controls are critical, enterprise deployments often require the broader Microsoft security and governance tools to strengthen their efficacy. Here, Defender for Cloud and Microsoft Purview play essential roles by extending the focus from application behaviour to include security posture, threat detection, audits, and data governance.

From an operational standpoint, Defender for Cloud equips security teams to see posture recommendations and AI-related threats, such as suspicious prompt activity or insecure configurations within their existing work environments. Meanwhile, Purview enhances the governance framework, allowing organisations to treat prompts and responses as governed data, thus adhering to the same auditing, classification, and security protocols that apply elsewhere in the enterprise.

To review these integrations effectively, check the security and compliance interface under Operate, especially the views centred around posture and governance. The specifics of availability and onboarding processes may vary by tenant, role, and licensing, but the key takeaway for any organisation is this: If you depend on Defender or Purview in other areas, your AI workloads should not be an exception.

Bringing Everything Together

One frequent oversight in AI operations is treating quality, safety, and cost as isolated systems. In reality, they’re intricately connected. Token consumption influences spending, spending affects sampling decisions, and those choices dictate the insights you gain regarding quality in production. The Operate surfaces in Foundry are particularly valuable since they allow you to view these factors collectively, rather than across disjointed tools.

The three most critical views remain those discussed in Part 1: Operate > Overview offers a high-level view of overall operations, including token and cost trends; Operate > Assets provides a searchable inventory highlighting significant resource consumers and problematic assets; and Operate > Quota helps you monitor deployment capacity, ensuring you avoid unnecessary throttling or scaling issues.

It’s essential to be transparent about the cost model related to observability itself. While monitoring doesn’t incur an additional Foundry platform charge, tracing does depend on the underlying Azure Monitor/Application Insights storage and retention requirements. Moreover, continuous evaluation incurs execution expenses since more model-related tasks are being performed. In simpler terms, you’re not just monitoring your agents; you’re also managing a telemetry system—so treat it that way.

The most significant shift between Part 1 and Part 2 is that once the necessary features are activated, observability must become a consistent practice. Strong teams don’t merely activate dashboards; they decide which signals are significant, which thresholds warrant intervention, where human oversight is necessary, and how these decisions are embedded into daily operations and release engineering.

Best Practices for AI Observability

• Avoid monitoring everything at once. Start with key signals directly linked to user harm or operational challenges.
• Utilise consistent evaluation logic across all environments. Quality should not vary between CI and production settings.
• Keep traces functional, not excessive. Rich traces provide value until they start recording content that you shouldn’t retain.
• Prioritise cost as a fundamental signal. Consider token usage and trace volume as operational limits rather than background noise.
• Revisit your configuration periodically. The platform evolves rapidly, and what was once in preview may now be operationally relevant in your environment.

In conclusion, Part 1 highlighted the need for a unique kind of observability in production AI due to the diverse failure modes of AI systems. Part 2 transitions this understanding into practical operations. By connecting Application Insights, enabling continuous evaluation, utilising the monitoring dashboard to detect changes, following traces to discover root causes, and incorporating compliance, security, and cost management into the same operational framework, you can move beyond considering quality and safety as mere bureaucratic tasks. Instead, they should become part of your ongoing operational practices.

Foundry’s observability model enables builders, operators, and governance teams to collaboratively answer the essential question: Is this agent performing correctly in production, and how do we ascertain that? If Part 1 served as the architectural blueprint, Part 2 acts as the operational playbook. Together, they illustrate the necessary transition for any serious AI team—from simply ensuring “the agent works” to ensuring “the agent remains reliable, explainable, and governable in production.”

FAQs

What is AI observability in Microsoft Foundry?

AI observability in Microsoft Foundry refers to the comprehensive framework that allows you to monitor, evaluate, and trace the performance of your AI agents in production. It emphasizes ongoing assessment rather than just traditional monitoring metrics.

How can I connect Application Insights for my project?

You can connect Application Insights by navigating to Operate > Admin > Connected resources within your project. Here, you can check if an Application Insights resource is linked and, if not, add it.

What steps can I take to ensure continuous evaluation?

To enable continuous evaluation, first ensure your Application Insights is correctly connected. Then, go to your agent’s monitoring surface and adjust the settings to define what “good” means for your specific workload.

Why is compliance important for AI operations?

Compliance ensures that your AI models adhere to necessary policies and regulations. By maintaining oversight through the Operate > Compliance workspace, you can identify violations and rectify any drift before it leads to significant issues.

How do I effectively monitor and act on AI performance?

Start with a manageable list of key metrics that require operator attention. Use Azure Monitor for configuring alerts based on these metrics, and take action when thresholds indicate potential problems.