Monitoring AI Coding Agents with Grafana: A Step-by-Step Guide

A few days ago, I opened my Azure SRE Agent and entered four prompts without much thought:

• find grafana dashboard for: GitHub Copilot
• run the panels queries. list top 3 very long sessions in the past 3 days
• what was session 1 doing?
• what was session 2 doing?

I didn’t use KQL, mention table names, specify what I meant by a “session,” or clarify what “long” meant. I also didn’t indicate which Application Insights resource housed my GitHub Copilot telemetry. Ten minutes later, I received this unexpected output:

#SessionStart (UTC)DurationSpansLLM callsTool calls
19b195ee3…Sep 10 19:55142.7 min663291365
293d53744…Sep 08 10:1747.9 min1034953
36da6be34…Sep 10 19:2428.6 min264108153

From here, I gained two insights that I wouldn’t have uncovered solely from the dashboard:

1. Session 1 (142.7 min) involved a GitHub Copilot CLI session focused on building a Grafana dashboard. It included 291 model calls (with 233 on gpt-6-astra, P50 17 s), 365 tool calls (162 were KQL queries through the Azure Managed Grafana MCP endpoint), three dashboard writes, and two failures. In a nutshell: “long, but not stuck.” The primary cost driver was a notable input-to-output token ratio of ~147:1, as query results and dashboard JSON accumulated and were repeatedly sent.
2. Session 2 (47.9 min) was stuck in a wait loop consisting of 23 cost_analysis → bash sleep (60, 120, or 180 s) → retry cycles. The sleep commands accounted for approximately 44 of the 47.9 minutes. Verdict: “latency-long, not work-long” — a discrepancy worth flagging.

Both sessions appeared identical on the dashboard as long bars in Agent Run Duration, yet they told very different stories underneath.

Understanding the Role of a Dashboard

Before diving deeper, let’s clarify what a dashboard really is. Each panel represents a query designed to provide meaningful insights, appropriately linked to the correct resource, and titled with the question it addresses. Each panel also contains annotations highlighting potential misinterpretations. The variables anchor it to a specific subscription, resource group, or an Application Insights component, ensuring its accuracy over time as team members constantly interact with it.

Ultimately, a dashboard embodies a team’s understanding of a system, encapsulated in a way that can be executed. This is why an on-call team member can quickly access insights without needing to consult anyone else. This functionality extends to agents too. The insight isn’t found on the screen; it’s embedded in the queries, scopes, titles, and descriptions beneath them, so agents can read and analyse them directly.

Think of a dashboard as a roadmap for the agent. It illustrates where the data resides, highlights meaningful questions, demonstrates how to ask each question, and identifies potential pitfalls. Surprisingly, the agent didn’t require an additional context file or a custom playbook. Most of what a human needs to understand the system is already integrated within the dashboard, including the essential knowledge an agent needs. The groundwork has been laid.

Connecting SRE Agent to Your Grafana Instance

The SRE Agent reached my Grafana instance via the Azure Managed Grafana MCP endpoint. This is how you can do the same:

1. Navigate to https:///api/azure-mcp which is the standard endpoint for instances in the Azure public cloud.
2. Attach it as an MCP connector. SRE Agent now has a built-in connector tailored for this purpose.
3. Test the connection and select the tools needed.

Understanding the Tool Trace

The trace generated from my initial prompts was concise:

• amgmcp_dashboard_search — “GitHub Copilot” led to one result: uid GitHubCopilot, tags github-copilot, opentelemetry, application-insights.
• amgmcp_dashboard_inspect — provided a list of all panels, current values of the template variables, the time range, and each panel’s KQL with resolved variables.
• amgmcp_datasource_list — displayed the UID for the Azure Monitor data source.
• amgmcp_query_resource_log — a query composed by the agent, executed against the Application Insights resource linked to the dashboard.

Your subsequent prompts for “what was session N doing?” would trigger additional query_resource_log calls.

Going Deeper with Panel Queries

Now, let’s delve deeper into what happens when you inspect the dashboard:

Using summary mode (no arguments) reveals the layout of the dashboard: 21 panels complete with IDs, titles, and types; six template variables holding their current values; the default time range; and a nextSteps map indicating how to dig deeper.

{
  "title": "GitHub Copilot",
  "panelCount": 21,
  "panels": [
    { "panelId": 5,   "title": "Time to First Token by Model (P50 / P90)", "type": "barchart" },
    { "panelId": 203, "title": "Tool Latency by Tool (P50 / P90)",         "type": "barchart" },
    { "panelId": 204, "title": "Agent Run Duration by Source and Agent",   "type": "table" },
    { "panelId": 202, "title": "Telemetry Freshness",                      "type": "table" },
  ],
  "variables": [
    { "name": "sub",    "current": "" },
    { "name": "rg",     "current": "my-resource-group" },
    { "name": "res",    "current": "my-app-insights" },
    { "name": "source", "current": "copilot-chat,github-copilot" },
  ],
  "timeRange": { "from": "now-7d", "to": "now" },
  "nextSteps": {
    "panel_queries": "Set includeQueries=true to retrieve underlying queries for each panel along with `resources`, the Azure resource IDs each target actually runs against …"
  }
}

Accessing panel-queries mode provides the query, data source, and resource scope for each panel, with variables resolved upon request. Here’s an example from Panel 204, which displays long session durations:

dependencies 
| where cloud_RoleName in (${source:singlequote}) 
| where tostring(customDimensions["gen_ai.operation.name"]) == "invoke_agent" 
| extend agent = tostring(customDimensions["gen_ai.agent.name"]) 
| summarize Runs = count(), 
    ['P50 Duration'] = round(percentile(duration, 50), 0), 
    ['P90 Duration'] = round(percentile(duration, 90), 0), 
    ['Max Duration'] = max(duration), 
    ['Runs > 10 min'] = countif(duration > 600000) 
    by Source = cloud_RoleName, Agent = iff(isempty(agent), "(unnamed)", agent) 
| order by Runs desc

Despite the agent needing to finalize the last details of the query, the tool description provides clear guidelines for implementation. It ensures that the agent isn’t left guessing.

Common Pitfalls and Considerations

It’s crucial to understand what assumptions may mislead one’s interpretation of the data. Here are three examples from the dashboard descriptions:

• The “Time to First Token” metric might vary between VS Code and Copilot CLI, leading to misleading cross-comparisons.
• Copilot CLI tool calls that cluster at certain time markers (60 s, 120 s, or 180 s) represent timeouts and should not be interpreted as successful calls.
• Agent-run roll-up spans (invoke_agent) which repeat their child requests may suffer from data duplication errors.

These elements wouldn’t be obvious from just looking at the schema but can dramatically influence interpretation—and they’re insights that haven’t been communicated directly before.

The Questions We Ask

The 21 panel titles align with the essential questions an experienced engineer would ask:

• Time to First Token by Model
• Tool Latency by Tool
• Tool Call Failures by Tool
• LLM Call Outcomes Over Time
• Telemetry Freshness

An agent reading these titles benefits from a pre-defined triage plan, eliminating guesswork.

What’s Missing and What You Can Do

Interestingly, the dashboard lacks a per-session view, meaning there’s no “longest sessions” panel available. However, the agent ingeniously constructed one from the dashboard’s established vocabulary:

dependencies 
| where timestamp > ago(3d) 
| where cloud_RoleName in ('copilot-chat', 'github-copilot') 
| extend op = tostring(customDimensions["gen_ai.operation.name"]), 
    sessId = coalesce(tostring(customDimensions["copilot_chat.chat_session_id"]),
                      tostring(customDimensions["gen_ai.conversation.id"])) 
| where isnotempty(sessId) 
| summarize Start = min(timestamp), End = max(timestamp), Spans = count(), 
    LLMCalls = countif(op == "chat"), ToolCalls = countif(op == "execute_tool") 
    by sessId 
| extend DurationMin = round(datetime_diff('second', End, Start) / 60.0, 1) 
| top 3 by DurationMin

Notice how every identifier in that query originated from a panel. The agent effectively navigated the boundaries but retained the dashboard’s coordinate system.

Your Next Steps

If you want to leverage the full potential of your dashboard, consider your next actions carefully. The recent operations table connects every run to Trace Visualization, which is instrumental for deciphering the slowdown in one specific tool call—session 1’s longest run, for example.

Unlike the visual layout, the aggregate data provided the agent with clarity, showcasing executed tool spans grouped by function while allowing the breakdown of chat spans by model response.

To capture even more context, enabling content capture allows spans to exhibit gen_ai.tool.call.arguments, helping clarify what calls were intentional and which were merely timed out.

Final Thoughts

A useful takeaway from this entire process is recognising that dashboards serve both human and agent users alike. The title designs and descriptions stem from the needs of those on-call, yet they provide crucial insight for agents as well. If the dashboard meets the needs of a human, it’s likely sufficient for an agent too. Similarly, if it is lacking for the agent, it probably isn’t comprehensive for the next person on call, either.

By viewing a dashboard as a living document—an input formatted for both human scrutiny and machine execution—you elevate its significance greatly. Each adjustment benefits both parties, as fewer artifacts mean less maintenance work.

I anticipated that connecting the SRE Agent with Azure Managed Grafana would be tedious, involving finding an endpoint and generating a token. It turned out to be straightforward, thanks to the native Azure Managed Grafana connector within the SRE Agent.

Getting Started

To begin, search for “grafana” under Connectors within the SRE Agent platform. Select the card, link it to your instance, test the connection, and pick the tools you wish to incorporate. Additionally, for anything else—like VS Code with GitHub Copilot, Claude Code, or your custom client—point it directly to https:///api/azure-mcp.

Now, feel free to ask it something casual. Happy monitoring!

FAQs

What is the Azure SRE Agent?

The Azure SRE Agent is a tool that helps track and analyze the performance of applications, specifically within Azure environments. It allows users to query data and gather insights effortlessly.

How do I connect my Grafana to Azure SRE Agent?

To connect your Grafana to the Azure SRE Agent, search for ‘Grafana’ under Connectors in the SRE Agent platform. Follow the prompts to link and test the connection.

What kind of insights can I gain from using Grafana?

By using Grafana, you can visualize data, monitor application performance, track user activities, and gain historical context on the performance metrics, which can help in troubleshooting and optimizing services.

Why is a dashboard important?

A dashboard serves as a central point for displaying critical information visually, allowing teams to quickly understand system health, performance metrics, and potential issues, facilitating faster reaction times during incidents.