Browser automation with Pydantic-AI + Playwright

When creating agents, a common goal is to enable them to navigate the web: opening web pages, moving between them, and reading their content. By integrating Pydantic AI with the Playwright features from Pydantic AI Harness, we can develop agents that can browse the web in a secure and automated manner.

Pydantic AI is an open-source framework that allows for the development of LLM-based applications and agents, providing type safety and support for OpenTelemetry. This makes it an excellent option for developing production-grade applications.

You can use Pydantic AI together with Microsoft Foundry models, leveraging either API keys or Entra token-based authentication. Whenever feasible, we suggest opting for the keyless method, which is what we’ll cover in this guide.

We will utilize the azure-identity package for authentication with Entra, either through local or managed identity, to obtain a token provider callback function for our credentials:

from azure.identity.aio import AzureDeveloperCliCredential, get_bearer_token_provider

# Use ManagedIdentityCredential for production environments on Azure
credential = AzureDeveloperCliCredential(tenant_id=os.environ["AZURE_TENANT_ID"])
token_provider = get_bearer_token_provider(credential, "https://cognitiveservices.azure.com/.default")
    

Next, we configure the model connection using the OpenAI package:

from openai import AsyncOpenAI

client = AsyncOpenAI(
  base_url=os.environ["AZURE_OPENAI_ENDPOINT"] + "/openai/v1",
  api_key=token_provider,
)
model = OpenAIChatModel(
  model_name=os.environ["AZURE_OPENAI_CHAT_DEPLOYMENT"],
  provider=OpenAIProvider(openai_client=client),
)
    

Let’s clarify some of the parameters used above:

• base_url: This points to the OpenAI-compatible endpoint for our Foundry model. This endpoint is compatible with Azure OpenAI models (such as gpt-5.4, which is being deployed in this project) and Foundry models that comply with the OpenAI v1 API like Kimi-K2.7-Code. The base URL format is “https://AZURE_OPENAI_SERVICE_NAME.openai.azure.com/openai/v1”.
• api_key: Here, we feed in the token provider callback function that generates OAuth2 tokens via our Entra credential. If API keys were being used, we would supply the key string instead.
• model_name: This should represent the deployment name, not the model name itself. Although these names sometimes match, it largely depends on how the setup is configured in the Portal or through infrastructure-as-code files. Remember, when using Foundry models, you always need a specific deployment for your desired model before it can be accessed.

Playwright is a powerful browser automation library. It was initially designed for writing end-to-end tests to ensure websites function correctly and still excels in that area today. Its automation features also allow agents to access various websites. When developing your own website, Playwright can help you provide your agent with the ability to browse, conduct quality assurance, and make design adjustments. You can also use it to visit other sites, provided their terms of service allow for automated access.

To mesh Pydantic AI with Playwright, we incorporate the PlaywrightBrowser capability from pydantic-ai-harness, which gives Pydantic AI agents additional functionalities.

from pydantic_ai_harness.playwright import PlaywrightBrowser

browser = PlaywrightBrowser(
    allowed_domains=[website_hostname],
    block_private_addresses=True,
    headless=False,
    max_content_tokens=30000,
    action_timeout_ms=5_000,
    navigation_timeout_ms=30_000,
    screenshot_on_navigate=False,
    auto_install_chromium=False,
)
    

Let’s go over these parameters:

• allowed_domains: This parameter restricts navigation and data requests to the specified hostnames. It safeguards against unexpected navigation and data transfers, ensuring the agent remains focused on the target scenario.
• block_private_addresses: Set to True by default, this option prevents navigation to localhost and private or reserved IP addresses. Change it to False only if the agent needs access to a trusted application running locally.
• headless: Playwright operates in headless mode by default, meaning the browser window isn’t visible. During development, I usually set this to False to observe Playwright interacting with the browser.
• max_content_tokens: This parameter sets a limit on the amount of webpage text returned to the agent. The default is 4000 tokens, but I’ve increased it to 30,000 to accommodate longer web pages. Remember, increasing content volume will impact performance and latency for subsequent LLM calls.
• action_timeout_ms and navigation_timeout_ms: Each action, like clicking or typing, along with page navigation, has its own deadline as they fail for different reasons. A missing selector should fail quickly, so the action timeout defaults to 5 seconds, while I’ve allowed 30 seconds for page loads. Tools can also set a custom timeout_ms when they anticipate a slow step.
• screenshot_on_navigate: This controls whether Playwright takes a screenshot after every navigation for the agent’s session. The default is False, as screenshots can fill the context window. You might enable it for complex workflows or auditing by humans.
• auto_install_chromium: When set to True, the library will automatically download the Chromium browser binary. It’s off by default, so you need to install Chromium manually. Typically, you would do this in your environments to cache it across runs, particularly in CI/CD scenarios.

With the Foundry model connection and Playwright browser set up, we can create a Pydantic AI agent that combines these elements along with the FileSystem capability, allowing the agent to write its Markdown reports easily into a specified folder.

agent = Agent(
  model=model,
  capabilities=[browser, FileSystem(root_dir=OUTPUT_ROOT)],
  system_prompt="You are a detailed manual QA agent testing a website owned by the user...",
)
    

We can now execute the agent, instructing it to perform a manual quality assurance check on the designated website:

 result = await agent.run(
    f"Conduct a manual QA evaluation on {url}. Start by loading this URL, creating a testing strategy, and examining the most significant usability risks and functional issues you can safely replicate. Document your findings in outputs/qa-report.md.",
)
    

As the Pydantic AI agent processes the query, it sends it to the Foundry model alongside the defined Playwright tools. The model determines which Playwright tools to deploy until the task is completed:

After the agent completes its task, we can review the generated report to verify its findings. However, we often want more insight: Which pages did it visit? What commands did it execute? How many tokens were consumed during the process?

Fortunately, we can enhance any Pydantic AI agent with OpenTelemetry, allowing us to export traces to any OpenTelemetry-compatible service, such as Pydantic Logfire or Azure Application Insights.

Let’s look at how to configure the code to send traces to Logfire:

trace_file = (OUTPUT_ROOT / "traces.jsonl").open("a", encoding="utf-8")
configured_logfire = logfire.configure(
    send_to_logfire="if-token-present",
    token=os.getenv("LOGFIRE_TOKEN"),
    service_name="pydanticai-playwright-qa",
    console=logfire.ConsoleOptions(),
    additional_span_processors=[SimpleSpanProcessor(ConsoleSpanExporter(out=trace_file))],
)
    

This configuration allows traces to be sent to Logfire based on the environment token provided. It also logs traces to the console, which is useful while developing the agent, and saves additional traces to a local file. By using this file, the agent can audit the Playwright browser interactions and suggest improvements for the prompts and parameters.

Next, we need to set up instrumentation specific to our used packages:

configured_logfire.instrument_openai(client)
configured_logfire.instrument_pydantic_ai(agent, include_content=True)
    

This code invokes instrument_openai for calls made via the OpenAI package and instrument_pydantic_ai for calls made through the Pydantic AI package. Both packages generate traces using the Generative AI semantic conventions to ensure that calls to LLMs, tools, and agents are consistently traced across various platforms.

After executing the agent, we can explore the traces generated. Here’s what a single run looks like:

To export traces to Azure Application Insights, we can simply add an additional span processor from the azure-monitor-opentelemetry-exporter package, specifying our App Insights connection:

connection_string = os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"]

logfire.configure(
    # other arguments
    additional_span_processors=[
        SimpleSpanProcessor(ConsoleSpanExporter(out=trace_file)),
        SimpleSpanProcessor(
            AzureMonitorTraceExporter.from_connection_string(connection_string)
        ),
    ],
)
    

As both platforms support OpenTelemetry, the traces will be consistent across both.

But what if the target website needs user authentication? When Playwright initializes a browser instance, it operates in isolation from your personal browser, meaning it can’t access existing cookies. This is usually a security advantage since we don’t want our agents to have direct access to our logged-in accounts. However, if your agent requires access to a logged-in website, you can manually pass a session state to the PlaywrightBrowser instance:

browser = PlaywrightBrowser(
    storage_state=json.loads(Path("playwright/.auth/site.json").read_text())
)
    

To create that state JSON file, execute the Playwright codegen command to open the desired website. After logging in and closing the browser, the session state will be saved in the specified location. It’s crucial to keep this storage file secure—do not include it in version control!

uv run playwright codegen https://your-owned-site.example/ --save-storage=playwright/.auth/site.json
    

Next, provide the saved state to your agent via a command-line option:

uv run python pydanticai_playwright.py https://your-owned-site.example/ \
    --session-state playwright/.auth/site.json 
    

You can download the complete code for the Pydantic AI agent from this project:

github.com/pamelafox/pydanticai-playwright-agent

This repository also offers infrastructure-as-code (Bicep) to provision an Azure OpenAI model and configure the complete environment for you.

Feel free to fork the code, personalise it, and create your very own browsing agent!