All Azure Technologies @ one Place

Six Limitations of AI per Application

Before we delve into architectural changes, let’s review some symptoms. If you can identify more than two of the following issues, this article is tailored for you.

1. Accountability for spending is unclear. An invoice might say “OpenAI account 2 used 40 million tokens,” but it doesn’t determine which department, feature, or user was responsible.
2. One disruptive application can limit another. All quotas are allocated per account and region, meaning a batch job and a customer-facing assistant sharing the same account fall under the same restrictions.
3. Reserved capacity often sits unused. It’s common to find provisioned throughput (PTU) operating at low levels of utilisation while simultaneously, another application accrues charges for pay-as-you-go usage of the same model.
4. All modifications necessitate code adjustments. Changes such as adopting a new region, retiring a model version, or altering a deployment name require editing settings in every individual app followed by redeployment.
5. No single point to enforce policies. Content safety, prompt injection checks, approved model lists, and logging are either duplicated in every app or absent altogether.
6. API keys are scattered across various locations. App settings, pipeline variables, and configuration files frequently result in model keys being stored in plain text.

These issues won’t be resolved by simply adding more Azure resources; they require a shift in where shared decision-making occurs.

The AI Landing Zone: Four Zones, Four Responsibilities

Microsoft’s AI landing zone expands upon the conventional Azure landing zone (encompassing subscriptions, identity, hub networking, and policies) by introducing three AI-specific zones. The purpose of each is to provide a clear answer to the question, “Who owns this?”

Two key points should be noted beyond the categorisation of zones.

The importance of private endpoints. Services in the AI Services zone can only be accessed through private endpoints, and essentially only from the gateway. An application desiring a model has a single pathway to follow.

Subscription vending. New subscriptions are automatically configured to connect to the hub, complete with route tables, Azure Policy, Defender for Cloud, and budgets. No one manually constructs a landing zone more than once.

The AI Gateway: Single Call, Multiple Decisions

The gateway acts as Azure API Management strategically placed in front of the model endpoints, utilising AI-specific policies introduced by Microsoft specifically for this purpose: managing token limits, collecting token metrics, enabling semantic caching, ensuring content safety, and maintaining backend pools with circuit breakers.

One common misunderstanding arises here: an API Management instance that oversees your application APIs does not qualify as an AI gateway. It transforms into one only when overseeing the models. Many environments may have APIM set up without possessing a dedicated AI gateway.

Let’s explore the journey of a single chat request once the gateway is operational.

Every step of this journey corresponds to a policy, which typically consists of concise instructions. Let’s look at two examples.

Setting a token budget for each team. Every consuming team receives an APIM product and a subscription; the policy generates a cap on token usage per minute and per month for that subscription.

                                    

                                    XML
                                    
                                    


This ensures that the Tuesday incident mentioned earlier cannot recur. If the document pipeline exhausts its budget, it receives a 429 error, while the website assistant remains unaffected.
Utilising reserved capacity primarily, with pay-as-you-go options for spikes. This involves establishing a backend pool with predetermined priorities and a circuit breaker that temporarily excludes an overwhelmed backend for ten seconds.
                                    

                                    BICEP
                                    
                                    
resource pool 'Microsoft.ApiManagement/service/backends@2024-06-01-preview' = {
  parent: apim
  name: 'openai-pool'
  properties: {
    type: 'Pool'
    pool: {
      services: [
        { id: ptuBackend.id,  priority: 1 }   // reserved capacity, used first
        { id: paygBackend.id, priority: 2 }   // spillover only when PTU is busy
      ]
    }
  }
}
This method addresses the issue of idle PTU. The aim is to reserve less while fully utilising the available capacity, thereby opting for pay-as-you-go only during peak demands.
Benefits Summarised in a Table
Advantage
Practical Implications
Centralised Authentication
Applications leverage Entra ID, eliminating model keys from any app. Revoking access involves a single change within the gateway.
Token Limits for Each Team
Establishes a budget for each department, ensuring one team cannot starve another.
Cost Attribution
Each call is labelled with the caller's identity, enabling chargeback to be represented in a Power BI report.
Load Balancing and Overflow Management
Maximises utilisation of reserved capacity, employing pay-as-you-go options for peak periods, with no 429 errors conveyed to users.
Independence Across Regions and Models
Applications interact with a single, stable URL, while model locations are managed via gateway configurations.
Semantic Caching
Frequently asked questions, such as the same question posed 500 times daily, are answered by the model only once.
Content Safety and Prompt Shields
One overarching policy safeguards every application.
Comprehensive Audit Trail
Providing clarity on "Who sent what to which model" is easily achievable.
Model Governance
Only sanctioned deployments are publicly accessible, and once a model is retired, it ceases to exist for all users simultaneously.
Significance of This Structure in Emerging Regions
In regions where the AI service catalogue has yet to match the larger markets, the gateway evolves from being a supplementary feature to a necessity.
Consider a scenario in which Azure OpenAI becomes available in a region after your applications are already deployed. Lacking a gateway, each app would need a new endpoint, new key, a testing phase, and subsequent release as soon as the service launches. However, with a gateway in place, you simply add a backend and adjust the priority, with the applications remaining oblivious to the model's location.

This reasoning also applies to data residency matters. The type of deployment (Global, Data Zone, Regional Standard, Provisioned) dictates where inferences are processed, irrespective of the account’s region. A gateway enables visibility, routing, and verification of which deployment type each team's traffic accesses.

Key Takeaways

• AI per application is a common starting point for most environments. This approach may work initially but becomes cumbersome after the addition of a second application.
• The AI landing zone effectively separates applications, gateways, and models into distinct zones, each managed by a specific owner. The AI Services zone operates solely through private access via the gateway.
• The AI gateway must be viewed as the APIM that stands in front of the models, not merely the applications. This framework transforms budgets, chargebacks, and safety policies from being hardcoded in each application to established gateway policies.
• If you are in the midst of a migration, there is no requirement to implement these changes simultaneously. Focus on transitioning first, ensuring the process does not obstruct the future addition of the gateway: maintain consistent deployment names, employ managed identities, and handle model traffic management through configuration instead of coding.

The next article will delve into the implications when multiple teams share this platform, exploring Microsoft’s operational model known as Citadel.