Are You Running Azure GPT Realtime in Production?

If you’re using Azure GPT Realtime in a live environment, you might have encountered an important question: how do you maintain conversation context across different turns? While it may seem like a minor technical detail, it can significantly impact your costs, response times, and overall user experience.

The method you choose to manage context can influence whether your costs spike to 90 cents per call or remain at a manageable 30 cents. It also affects how quickly your contact centre responds—a critical factor in keeping your customers satisfied. I explored seven different strategies for managing conversation context in a healthcare support scenario involving a 10-turn patient onboarding dialogue, using the same model, region, and system prompt.

Understanding Token Management

With each turn in a real-time voice call, audio tokens are sent to and received from the model. What you choose to include in turn N will depend on what you’ve opted to retain from preceding turns. This decision directly affects your token count, influencing your billing.

Here’s a quick overview of Azure GPT Realtime pricing for the API:

• Audio Input: $32/M
• Cached Audio Input: $0.40/M (an incredible ~99% discount!)
• Audio Output: $64/M

The savings from caching are substantial, making it essential to keep a stable prefix. Below are seven different strategies for what to include in the prompt prefix for turn N:

A) Full History

This strategy keeps a single persistent WebSocket where the server remembers every conversation item. By turn 10, the system retains all previous turns (1–9), resulting in 18,123 tokens. Although you benefit from perfect memory and simplicity in code, the downside is high latency. For example, turn 6 took 6.3 seconds due to the growing context. If the WebSocket drops, you lose the context entirely.

B) Stateless

In this approach, each turn uses a fresh WebSocket, where the model only sees the system prompt and the current question. Total tokens for this method are 6,557—the lowest among all strategies—however, there’s no conversational memory, making it suitable only for tasks where each turn is independent.

C) Sliding Window

This method uses a fresh connection each turn while including the last 2 Q&A pairs. Total tokens: 8,655. Although it provides short-term coherence like “yes, that one”, details from turn 1 are lost by turn 5, and cache rates drop to 20.4% as the moving window disrupts the prefix.

D) Compression Every 5 Turns

Here, a compact summary (about 70 words) is created every five turns using the gpt-realtime-mini and included in the system prompt, dropping earlier turns. This strategy totals 8,660 tokens but has a cache hit rate of only 7.1%. Our test scenario only allowed for one compression event, limiting its effectiveness on longer calls.

E) Sliding + Compression

This hybrid approach combines a compressed summary of previous turns with the last 2 turns retained verbatim. With 7,975 tokens, it has the best balance in memory preservation, covering both previous context and immediate concerns. The cache rate is 10.4%. This is an excellent choice for contact centres that handle reconnections.

F) Server-Side Truncation

This strategy allows the API to manage context automatically. Like method A, it retains a persistent WebSocket but uses a configuration to drop the oldest items when context exceeds 8,000 tokens. Total tokens here are 18,247. This method could serve as a secondary strategy since it rarely fires under the default thresholds.

G) In-Session Delete

This approach employs a single persistent WebSocket throughout the call, with the client checking the total token usage. If it exceeds a preset limit (e.g., 600), the client generates a summary of the oldest turns and deletes them server-side, inserting the summary as a system message. This strategy results in 14,721 tokens and a cache rate of 23.9%. It’s an effective pattern for voice interactions.

Key Takeaways

1. Focus on optimising total tokens, rather than trying to achieve a high cache hit rate.
2. High cache rates on large contexts may lead to higher costs than lower rates on smaller contexts.
3. Choose the most cost-effective strategy that aligns with your specific needs. For instance, use approach G for live voices and B for lookups.

Best Practices for Azure GPT Realtime in Enterprise Applications

Here are a few guiding principles for deploying Azure GPT Realtime in business settings:

• Start with your operational requirements rather than just benchmark results; what matters most is fitting the strategy to your typical session length.
• Prioritise total tokens for cost efficiency, as caching provides only partial savings.
• Treat your system prompt carefully to avoid disruptions in context, keeping stable content at the start.
• Establish your latency requirements early; if you need sub-2-second response times, avoid strategies that require reconnecting.
• Use server-side truncation primarily for safety, rather than as a main strategy.
• Gather data on token counts, cache hit rates, and latency for real-world insight before making decisions based on benchmarks.

FAQs

What is Azure GPT Realtime?

Azure GPT Realtime is a state-of-the-art platform for generating conversational AI in real-time, allowing businesses to automate dialogues and manage customer interactions efficiently.

How can I optimise costs when using Azure GPT Realtime?

To cut costs, focus on minimising total token usage and select the strategy that aligns best with your specific use case while ensuring low latency.

Why is managing conversation context important?

The way conversation context is handled affects not just the cost of using the service but also the customer’s experience during interactions.

Can I measure the effectiveness of my chosen strategy?

Yes, by monitoring token counts, cache rates, and latency during real customer interactions, you can evaluate how well your strategy works in practice.

For more detailed insights, feel free to explore further resources.