Introducing Azure SRE Agent in Azure Managed Grafana: The Dashboard as Command Center
Discover the Azure SRE Agent Integration in Azure Managed Grafana
Weโre thrilled to announce that the Azure SRE Agent integration is now available in preview with Azure Managed Grafana! By pairing an Azure SRE Agent with your Grafana workspace, an agent icon will appear in the dashboard toolbar. Click on it, and the agent opens up in a panel next to your dashboard. It reads the dashboard and queries the same data sources, enabling you to ask questions without needing to draft the queries yourself. If youโve granted it the necessary permissions, it can also take action based on its findings, but always with your approval first.
How It Works in Just 30 Seconds
In this demo, I created a small web app intentionally introducing a bug: an Orders API on Azure App Service, where the in-memory order cache never gets evicted. I set up a Grafana dashboard and began generating a steady stream of traffic. After about 15 minutes, the dashboard turned red, indicating issues. I opened the agent’s pane and typed a straightforward question:
Order lookups have started failing. Whatโs going on?
I didnโt specify the app name, resource group, table, or metric, yet approximately 30 seconds later, I received a detailed response:
The agent concluded, โOrder lookups are failing because the order cache is growing indefinitely. Each lookup scans a larger cache, and eventually, this causes timeouts and results in a 503 error.โ It then provided evidence and the sources:
- Source: Prometheus (app metrics) – Observed that the orders_cache_entries increased from 147 to 1,587, with steady process uptime, indicating that no restart occurred.
- Source: Application Insights – Noted the P95 lookup time increased significantly, leading to 503 errors starting at a specific time.
- Source: Console log in Log Analytics – Documented 14 OrderCacheLookupTimeout errors with a sample message indicating issues scanning cached orders.
The agent emphasised what it did not do: โI didnโt read the code; my conclusions are drawn from telemetry data.โ It suggested restarting the app (which would clear the cache) and proposed a fix: bounding the cache and switching to a keyed lookup. It concluded with, โI havenโt made any changes yet; I can restart the orders-api-2611 web app if you approve.โ
Taking Action with Approval
I responded with my approval:
Yes, restart it, then confirm from the dashboardโs data that itโs recovered.
The agent paused before executing, displaying the exact command it intended to run:
After I clicked โApprove,โ I was pleased to see that the restart command executed successfully, yet the agent didnโt mark the incident resolved. Instead, it returned to checking data and reported, โI canโt confirm recovery. Although the restart command succeeded, the dashboard data indicates the app hasnโt recovered.โ This was insightful, as the uptime gauge was rising, and the cache was still full.
Continuing the Investigation
My third prompt was clear:
Donโt restart again. Give it a moment and check the data once more.
After waiting, the data showed a significant improvement: uptime had dropped dramatically, the cache entries decreased, and there were no 503 errors recorded. The agent provided a valuable warning, โThis restart only buys time.โ It suggested that if the cache kept refilling, failures could return in 10 to 15 minutes.
Documenting the Incident
For my fourth prompt, I asked the agent to create an incident note:
Provide a short incident note for the ticket.
The agent then crafted a detailed timeline, identified the cause of the issue, outlined mitigation steps, and highlighted any gaps in its analysisโall while the success rate stayed at 100%.
This entire processโfrom my initial question to the incident noteโtook roughly 11 minutes. I composed just four sentences without creating any queries and remained within the dashboard the whole time!
What Makes a Dashboard Valuable?
A well-structured dashboard is essentially a shared resource for your team, encapsulating the understanding of a system. Each panel corresponds to a significant query that someone has crafted to answer critical questions, making the dashboard a vital tool for both humans and AI agents alike. This integration bridges both perspectives onto a single interactive map.
How the Agent Operates Within the Dashboard
Every interaction starts with a host context block that Grafana provides:
The user is engaging with the SRE Agent embedded chat in Grafana.
Current Grafana URL: https:///d/orders-api/orders-api?from=now-30m&to=nowโฆ
This URL includes relevant information like the dashboard, time range, and variable values, ensuring the agent understands the context. From there, the agent interacts through the workspace’s MCP endpoint, similar to other clients. For each action taken, the pane will display real-time calls being made to retrieve and act on data.
Setting Up the Azure SRE Agent Integration
If you want to leverage this powerful integration, you need an Azure Managed Grafana workspace on the Standard tier and an Azure SRE Agent. Follow these three simple steps:
- Connect the agent to Grafana: Go to the SRE Agent portal and add the Azure Managed Grafana connector for your workspace. Opt for Viewer for read-only access or Editor if you want the agent to create and modify dashboards.
- Enable the integration: In the Azure portal, access your Grafana workspace, navigate to Integrations > Azure SRE Agent (Preview), select the agent, and click Save. You’ll see a message confirming the connection.
- Grant access: Ensure each user who wants to use the pane has the SRE Agent Standard User role on the agent and the appropriate Grafana role on the workspace. It’s best to assign roles to a Microsoft Entra group if possible.
Final Thoughts and Next Steps
This integration is in preview, so expect variations and always validate the work done by the agent, as AI-generated content isn’t always perfect. Remember, while investigating works smoothly through the Grafana connector, actions require specific permissions on the target. For more tips on leveraging the integration, refer to the Integrate Azure SRE Agent (preview) with Azure Managed Grafana document.
FAQ
- What is the Azure SRE Agent?
- The Azure SRE Agent is a tool that helps manage workloads in Azure by providing insights into application performance and enabling actions from within the Grafana dashboard.
- What do I need to use the Azure SRE Agent?
- You need an Azure Managed Grafana workspace on the Standard tier and an Azure SRE Agent to begin using the integration.
- Can the SRE Agent act without my approval?
- No, the agent will always ask for your approval before performing any actions based on its findings.
Share this content:
Discover more from Qureshi
Subscribe to get the latest posts sent to your email.