Building a GitHub Copilot Agent Usage Dashboard

When engaging with organisations that are working on creating their own GitHub Copilot custom agents, it’s crucial to understand how these agents are being adopted within their communities. Questions often spring to mind, such as: “How effectively are we using this tool?”, “Which agents are popular, and which ones aren’t gaining traction?”

While native metrics offer a broad overview of adoption, they fall short in providing detailed information. For instance, they don’t really answer questions about which workflows are most popular or how usage patterns shift over time.

In this article, I’ll guide you through creating an enterprise-level GitHub Copilot usage dashboard. This dashboard will capture in-depth telemetry from VS Code, process the data via Azure Monitor, and visualise the findings in Grafana—all using an infrastructure-as-code approach that makes it straightforward to replicate. Plus, this dashboard can be shared with anyone who might need access to it.

To start, you can set up VS Code to emit metrics using OpenTelemetry, which is a standard configuration in VS Code. Essentially, you’ll need to direct it to an OpenTelemetry Collector, which serves as an endpoint for consuming the telemetry data.

In this implementation, I used a container image hosted on Azure, opting for Azure Container Apps (ACA) due to its user-friendly managed environment. You could also run it in Azure Kubernetes Service (AKS), although that would require a bit more setup.

A prebuilt image of the OpenTelemetry collector is available, which I’ve modified to configure it for sending telemetry data to Azure Application Insights.

For hosting and defining the dashboard, I chose another Azure managed service, Azure Managed Grafana.

The example dashboard I’ve created showcases a variety of visualisations based on data collected in Application Insights. With Azure Managed Grafana, you can easily create these dashboards visually, or use a JSON file for implementation and tweak it as needed.

It’s important to note that the telemetry from VS Code does indicate the users’ locations—such as city, region, and country—but it does not include any personally identifiable information (PII). This feature is intentional for privacy reasons.

Managed Grafana has its own permission settings, which allow you to control who has access to the dashboard.

You can find a detailed guide on setting this up in the GitHub repo for the Copilot Usage Dashboard. The repository includes instructions for creating it through either traditional “click-ops” or Terraform. Here’s a summary of what you’ll need in Azure:

• An Azure Container App (ACA) to host the collector—this should have public ingress.
• An Azure Container Registry (ACR) to store the customised Docker image.
• A Key Vault to keep the Application Insights connection string that ACA refers to.
• Application Insights, created with a flag to ensure compatibility with Grafana.
• A Log Analytics Workspace compatible with Application Insights.
• Azure Managed Grafana, to host your Grafana dashboard.

Bear in mind that VS Code should be set up to emit OpenTelemetry.

{
  "github.copilot.nextEditSuggestions.enabled": true,
  "github.copilot.chat.otel.enabled": true,
  "github.copilot.chat.otel.exporterType": "otlp-http",
  "github.copilot.chat.otel.otlpEndpoint": "https://"
}

Replace FQDN with the URL of your Azure Container App’s public ingress.

Inside the GitHub repository, you’ll find a Dockerfile that injects the correct configuration file into the OpenTelemetry collector image. This config tells the collector to send data to Application Insights. Here’s what it looks like:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
  attributes:
    actions:
      - key: environment
        value: "prod"
        action: upsert

exporters:
  azuremonitor:
    connection_string: "${APPLICATIONINSIGHTS_CONNECTION_STRING}"
  debug:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, attributes]
      exporters: [azuremonitor, debug]

    metrics:
      receivers: [otlp]
      processors: [batch, attributes]
      exporters: [azuremonitor]

In this configuration, you’ll see a placeholder for the Application Insights connection string. In the ACA setup, this becomes an environment variable that refers to a secret in the Key Vault.

If everything is set up correctly, VS Code should send telemetry data to the container image running in ACA, which will then relay it to Application Insights, making it available for the Grafana dashboard.

The GitHub repository also covers troubleshooting steps, but here’s a quick list to guide you:

1. If Grafana isn’t displaying any data, confirm it has access to Application Insights.
2. Check for incoming telemetry in Application Insights by reviewing the logs—specifically, check the Dependencies table. If data appears there, it’s likely a Grafana permissions issue. If not, investigate ACA.
3. Look into ACA logs to ensure everything is functioning correctly. Check for received logs.
4. Test the ACA by using a curl request to send a sample log (a demo is in the repository) to see if it’s capable of accepting logs.
5. Ensure the connection to Application Insights is accurate and that it’s being retrieved from the Key Vault. You could also directly input the connection string into the environment variable as a quick test.
6. If you’re still facing problems, there might be an issue with the VS Code configuration; double-check its accuracy.

These detailed steps should help you resolve issues swiftly.

This implementation aims to create a dashboard that tracks GitHub Copilot agent usage while adhering to a standard set of security protocols. However, there may be room for improvement. Here’s a list of possible enhancements:

1. Create a more refined dashboard. This should be fairly straightforward, as there are examples available for various visualisations that provide a better focus on agent and model usage.
2. The OpenTelemetry collector hosted on ACA has public access. You might want to tighten security at the network level with address restrictions or a non-public ingress, ensuring it remains accessible to the intended VS Code users.
3. The ACA collector endpoint currently lacks authentication. You could introduce an authenticating proxy within the Dockerfile or at the ACA ingress level. Further investigation will be necessary to ensure that VS Code configuration remains compatible with this approach.
4. Automation for modifying the VS Code configuration for multiple users hasn’t been explored here. However, organisations might find ways to implement these changes through their application deployment strategies.

This approach enables organisations to monitor the usage of GitHub Copilot agents and their models, offering insights that the GitHub Enterprise dashboards do not provide. With this information, organisations can assess whether their investments in custom agents are paying off. Moreover, the dashboards can be easily shared beyond the GitHub Enterprise environment, reaching a wider audience.