Microsoft Foundry Now Has an AI Gateway Control Plane — What Changes for App Service

Back in May, I shared a runnable sample that showcased how Azure API Management (APIM) integrates with an AI agent hosted on Azure App Service. APIM took care of managing token limitations, semantic caching, token metrics, and providing access to the AI model. The key takeaway was straightforward: maintain an interchangeable agent framework while handling production controls at the gateway level.

A recent post on Apps on Azure, titled From AI Adoption to AI Governance, reached a similar architectural insight from the perspective of platform governance. It utilises APIM to create a model identity and track token consumption, resulting in a chargeback signal for Microsoft Foundry workloads.

While this is certainly useful confirmation, the more exciting news lies within the product updates: Microsoft Foundry can now directly create or link an APIM-based AI Gateway through its Admin console. The data pathway remains APIM, but the configuration and governance are now accessible to a wider audience.

This post will delve into what these changes mean for an App Service-hosted agent, what Foundry can now manage, the aspects that still fall under APIM’s purview, and modifications I’d suggest for the sample we previously built.

The architecture maintains its familiar format:

Client -> App Service agent -> APIM AI Gateway -> Foundry model

APIM continues to serve as the traffic gateway. It handles authentication, enforces policies, forwards requests, and monitors telemetry. Meanwhile, App Service remains the application runtime for both the agent and its APIs.

The new addition is the Foundry control plane. Through Operate > Admin console > AI Gateway, platform teams can now:

• Create a fresh APIM instance or associate an existing one that’s already supported.
• Activate the gateway for individual Foundry projects.
• Set independent token limits for projects sharing the same gateway.
• Monitor gateway traffic and check logs.

Once the gateway is activated, different Foundry experiences can register custom agents and manage supported MCP tools. While these actions don’t occur directly on the AI Gateway tab, the gateway facilitates the traffic pathway they govern.

This shift is significant. In our previous sample, the infrastructure owner was tasked with creating APIM, importing the model API, deploying policy XML, and connecting the agent to the gateway. Foundry doesn’t eliminate APIM or replace its intricate capabilities; instead, it provides Foundry administrators with a first-party approach to the same framework.

Each AI Gateway is linked to a Foundry resource and then activated for projects contained within it. New projects can automatically inherit the gateway while existing ones need to be added manually.

This creates an effective governance boundary, allowing multiple teams to share one APIM instance without intertwining their token pools. Each project can have its own token limit. If a project goes beyond its configured cap, the gateway returns a 429 Too Many Requests response while other projects continue to operate within their allocations.

In our earlier example, we implemented an APIM token-limit policy based on an APIM subscription. This remains a viable approach, especially when the consumer grouping includes an application, business unit, or external client.

The Foundry integration introduces another useful counter key: the Foundry project. For organisations that already categorise models and agents by project, limits can now align with that structure rather than having to be replicated in every client application.

The AI Gateway experience is expanding beyond model endpoints in two notable ways. Foundry can now register custom agents that operate outside Foundry, including those using A2A for communication. Additionally, AI Gateway can govern supported MCP tools. It’s worth noting that the current tools governance preview is specific to MCP and does not encompass all Foundry or OpenAPI tool types.

This is important for App Service since the runtime can remain unchanged. An agent can still function on App Service while being part of Foundry’s inventory and direct supported MCP traffic through the gateway.

As these features are still in the preview stage, I recommend viewing them as a means for control-plane integration rather than a reason to overhaul an operational production agent.

Foundry simplifies the standard process, but it doesn’t render the full APIM features irrelevant.

I would still recommend using APIM directly for:

• Custom policy expressions and headers specific to your organisation.
• Semantic caching supported by Azure Managed Redis.
• Backend pools, priority routing, and circuit breakers.
• Private networking, multi-region gateways, and sophisticated topology choices.
• Custom metrics and dimensions required for internal chargebacks.

The clear distinction is: Foundry manages AI assets and common project boundaries; APIM governs traffic agreements and advanced gateway functionalities.

The current framework-agnostic AI Gateway sample hosts a Microsoft Agent Framework agent alongside an MCP server on a single Linux App Service. The agent interacts with its model through APIM, which handles token limits, semantic caching, token metrics, and managed identity authentication.

Most of this architecture should ideally remain untouched.

The agent should still reach out to a stable gateway endpoint. It shouldn’t need to differentiate whether APIM was set up through Bicep, associated in Foundry, or managed by a central platform team. This separation was precisely why the gateway was introduced in the first place.

For a fresh proof of concept, Foundry can create a Basic v2 APIM instance. For production, Microsoft suggests assessing Standard v2 or Premium v2, depending on throughput and networking needs.

If you have an existing enterprise gateway, link an APIM instance only after making sure it meets the requirements: it must reside within the same Microsoft Entra tenant and Azure subscription as the Foundry resource, you should have the API Management Service Contributor role (or Owner), and the instance must utilise a supported v2 tier.

Important migration note: The original sample defaults to the APIM Developer tier. This instance won’t show up in Foundry’s Use existing APIM list since current direct integration requires a v2 tier. Don’t assume that a classic-tier gateway can be easily attached.

Creating or linking the gateway at the Foundry resource level doesn’t automatically enable every existing project. You’ll need to add each existing project to the gateway and confirm its status is Enabled.

Refrain from layering limits without a clear ownership model. A Foundry project limit, an APIM subscription limit, and a model deployment TPM limit serve different purposes:

Use all three when the boundaries are intentional. Otherwise, operators might find themselves troubleshooting a 429 error without clarity on which layer caused it.

After activating a project, conduct a model request to confirm that the APIM request metric updates. Microsoft also suggests checking the GatewayLogs table for a successful response and ensuring the API name aligns with the AI Gateway.

Next, test the failure path. Set a deliberately low project limit, exceed it, and check that the gateway responds with a 429. A governance feature isn’t complete until the team has observed both the permitted and denied actions.

The key takeaway is not that everyone should reconstruct an AI Gateway in a new portal. Rather, it’s that the composable architecture now offers a more accessible control plane.

App Service continues to support the agent. APIM manages the traffic. Foundry now provides model and platform owners a direct way to link projects, allocate resources, and unify governance for agents and tools.

If you’ve already developed the earlier sample, the application structure doesn’t need to shift. Assess the Foundry integration, determine whether its project model aligns with your organisation’s framework, and review the APIM tier before considering any migration. The gateway remains as a valuable contribution; Foundry now simplifies its operation.