The Economics of Agent Optimization: Cost-Effective Strategies for AI

Welcome to the second post in our four-part series, The Economics of Agent Optimization. Here, we’ll explore practical strategies and capabilities to help you manage agent costs effectively while leveraging AI as a managed investment system on Microsoft Foundry. In the first article, we discussed the three critical decisions that support this system: optimising each request in real-time, improving workflows over time, and continuously monitoring expenditures. This time, we’ll focus on the first aspect, which has a direct impact on every penny spent on AI.

Understanding Agents and Costs

To grasp AI’s economics, let’s define an agent. An agent functions as a loop around a model. It plans, makes calls to tools, evaluates results, and reasons again, which means an individual completed task can involve multiple model requests. Consequently, businesses must prioritise the cost tied to a successful outcome rather than simply counting token expenses.

Each step in that loop counts as a model request, and each request embeds choices regarding the model, the tools leveraged, the components reused, and the inputs provided. When these decisions are spot on, savings can accumulate with every loop iteration. So, it’s clear why we start with agent optimisation here.

The Pitfalls of Production AI

Most AI applications are developed following a standard blueprint. Initially, you choose the best model available, feeding it all the necessary input in the prompt to prove the concept works. This instinct is correct during prototyping but can lead to issues in production, where defaults from the prototype become the operational structure.

Here’s where things start to go awry. AI workloads are incredibly diverse. A single application can involve intent classification, extraction, formatting, summarisation, and involved multi-step reasoning—all of which carry different complexities. Sending all requests to one advanced model means overpaying for many requests that don’t actually need that level of complexity.

Moreover, unlike in prototypes where you pay for a single call, agents incur costs for the entire loop. Thus, inefficiencies compound, making it crucial to consider how you can avoid unnecessary requests. The goal shouldn’t be merely to reduce tokens; instead, you should focus on minimising the cost of successfully completing tasks without sacrificing quality, safety, or speed. This brings us to four vital decisions that impact the economics of every request.

Four Key Strategies for Runtime Optimization

With Microsoft Foundry, you have four powerful levers to help you navigate these trade-offs consciously instead of settling for those established during prototyping. You can utilise each lever independently, evaluate performance, and reverse any changes if they don’t yield positive results.

1. Assign the Right Model for Each Request

The principle is straightforward: tailor the outcome to match task complexity. Routine tasks shouldn’t bear the costs typically associated with top-tier models, while complex requests shouldn’t settle for lower quality just to save on tokens.

The Model Router in Foundry analyses incoming requests and redirects them to the most suitable underlying model in real time, all through a unified endpoint. You can choose routing modes based on what matters most—whether that be cost, quality, or a combination of the two. By aligning model subsets with Azure Policy, you can also ensure compliance within your routing decisions.

2. Eliminate Duplicate Token Costs

Agents can be highly efficient with caching. Since the same system instructions, tool definitions, and policy texts are repeated at each turn, an agent that loops ten times incurs the token cost for those prefixes ten times. Prompt caching allows for reusing previously processed data instead of redoing the work. This can lead to significant savings on standard deployments.

To benefit from caching, it’s essential to structure prompts wisely. Begin with stable content and follow with variable content. For example, ensure system instructions and tool definitions are placed at the top, while user input and any other changing elements should be positioned below them. This way, the cache can match accurately with incoming requests.

3. Enhance Prompts and Agents

The choice of model significantly influences pricing, but optimising the prompt plays a vital role too. Improving prompts is the least complex fix—it doesn’t require any infrastructure changes. For starters, clearly state the task at the beginning of the prompt, specify the desired output style and length, and use illustrative examples rather than lengthy explanations.

Foundry automates the intricate tuning process previously required for optimization. Its Prompt Optimizer helps to rewrite system instructions using best practices and gives clear reasons for each adjustment. This ensures you can refine, test, and implement changes quickly.

4. Monitor Performance with Observability and Evaluation

You can’t adjust what you can’t observe, nor can you shout about savings that haven’t been tracked. Foundry’s observability feature provides essential insights at the request level. This includes metrics like input and output token counts, cache hit rates, latency, and model performance scores.

Two metrics are particularly crucial: cost per request and cost per completed outcome. The first reveals if a cheaper option meets quality standards, while the latter reflects what was actually spent to achieve the desired outcome. It’s critical to review both metrics to ensure that savings in one area don’t come at the expense of overall effectiveness.

A Continuous Improvement Cycle

These levers are not just one-off measures; together they create a continuous loop that becomes more efficient each time it’s executed. Microsoft AI encapsulates this concept with the idea of a “hill-climbing machine,” which continuously improves cycle by cycle using better data and more effective evaluations.

Decisions regarding models and offers determine where requests are processed, while fine-tuning ensures tasks can be executed at a reduced cost over time. Caching lowers the expense of every cycle, enabling you to repeat the process often enough to see significant benefits. The optimisation of prompts and agents generates further refinements, and observability helps track progress and ensures that changes are productive.

Ready to Get Started?

Have you caught up on our prior posts in The Economics of Agent Optimization series? If you’re eager to enhance your AI strategies, don’t miss out on the valuable insights from the first instalment.

FAQs

What is agent optimisation?

Agent optimisation refers to improving the performance and cost-effectiveness of AI agents by making informed choices about models, caching, prompts, and monitoring.

How can I reduce costs with AI agents?

By choosing the right model for each task, avoiding duplicate token costs through caching, optimising prompts, and keeping track of performance metrics.

Why is observability important in AI?

Observability allows you to track the performance of your AI agents, enabling you to make informed adjustments and measure the effectiveness of your optimisations.

Can I fine-tune models in Microsoft Foundry?

Yes, Microsoft Foundry offers fine-tuning capabilities that allow you to adjust smaller models for specific tasks, improving their efficiency and reducing operational costs.