For the first time, real-time transcription goes multilingual

Earlier this year, we launched Post-Stream Refinement, which finally bridged the gap in real-time speech processing. With this feature, you could receive immediate streaming results along with a precise final transcript, all without experiencing any latency issues. However, there was one requirement: you had to specify a single language beforehand. But we know that real-life conversations don’t work that way. People often switch languages mid-sentence, names of products and brands cross over languages, and users of a global application might speak different languages in different sessions.

Today, we’re excited to announce a game-changer. Our Multilingual Post-Stream Refinement is now in public preview for Azure AI Speech within Microsoft Foundry. For the very first time, a single real-time stream can automatically transcribe several languages within a single session. There’s no need to declare a language in advance, and the spoken language is detected automatically, with the final transcript refined for top-notch accuracy. Rest assured, everything you loved about Post-Stream Refinement remains the same; the key difference is that the refinement process is now multilingual. Read the Documentation

If you’ve used Post-Stream Refinement before, let’s dive into what’s different with this multilingual preview — as well as what remains unchanged:

What’s new: the same real-time contract, now multilingual. One multilingual model runs the refinement process, adding automatic language detection, code-switching within a single spoken phrase, an open-range setup without a candidate list, and built-in diarization — while the initial streaming latency remains unchanged.

In our internal tests, multilingual Post-Stream Refinement achieved a reduction in word error rate (WER) by around 10% on average across leading locales. We noticed even greater improvements in challenging scenarios — particularly with long sentences, unique names, and multilingual or code-switched speech. It’s important to note that while the initial partial-result latency remains the same, only the final transcript sees enhancements.

These improvements are relative to the standard real-time model and can differ based on language, acoustic settings, and content type. The revised final outcome may slightly increase latency at the end segment, but partial results won’t be affected.

Quality gains (relative improvement against standard real-time transcription): approximately 10% decline in average word error rate in Tier-1 locales; significant reductions for long sentences; considerable improvements for unique names and entities; no change to partial-result (first-token) latency. Relative gains can vary based on locale and audio quality during public preview.

The public preview currently supports 15 Tier-1 locales. Because the service detects languages automatically, you can have any mix of these languages within a single stream:

Supported languages: 25 auto-detected languages across 29 market locales — any combination of which can appear in a single real-time stream without pre-defining a locale.

This feature is available in selected Azure regions:

Availability: multilingual Post-Stream Refinement is operational in six Azure regions during public preview — East US, West US, North Europe, Central India, Southeast Asia, and Japan East.

Early users from various sectors — including travel, electronics, automotive, aviation, and media — have reported noticeable enhancements in transcription quality. Those testing this multilingual feature have seen the most significant improvements with complex content, such as unique names and long-form discussions. Several are actively evaluating the feature using their own audio in preparation for its broader release.

To enable multilingual Post-Stream Refinement, you’ll need to make a quick configuration adjustment to your existing SpeechConfig. Here’s what you’ll need:

• Speech SDK 1.50 or later. Older versions won’t support the multilingual option.
• A Speech resource in one of the supported regions mentioned earlier.
• Auto-detect language configuration (set to open range), allowing the service to identify the spoken language without requiring a candidate list.

To set up the post-processing option, use PostRefinement and pass an open-range AutoDetectSourceLanguageConfig when creating the recognizer. Here’s a full Python example that you can easily copy and paste, including an optional line for detecting the end of an utterance:

import azure.cognitiveservices.speech as speechsdk

speech_config = speechsdk.SpeechConfig(
    subscription="YourSpeechKey",
    region="YourSpeechRegion")

# 1) Refine the final transcript (Post-Stream Refinement)
speech_config.set_property(
    speechsdk.PropertyId.SpeechServiceResponse_PostProcessingOption,
    "PostRefinement")

# 2) Multilingual auto-detect - no candidate language list needed
auto_detect_config = speechsdk.languageconfig.AutoDetectSourceLanguageConfig()

audio_config = speechsdk.AudioConfig(use_default_microphone=True)

recognizer = speechsdk.SpeechRecognizer(
    speech_config=speech_config,
    auto_detect_source_language_config=auto_detect_config,
    audio_config=audio_config)

Helpful Tip: Post-Stream Refinement is crucial for applications that store or process final transcripts, such as meeting notes, call analysis, compliance records, or AI summarisation. If your usage involves only partial results for immediate display and you discard them afterwards, your real-time experience will remain fast while the final transcript you retain will significantly improve.

Enhance the accuracy and language awareness of your Azure AI Speech applications with just one configuration tweak. This feature is currently live in public preview within Microsoft Foundry.

Read the Documentation

We’d really love to hear your thoughts. Try out Post-Stream Refinement in your applications and see how it elevates your transcription quality.