Free tools Windows power users keep installed
One-click scans. No signup required.
Microsoft Foundry voice agents can handle live speech in two ways: a single real-time model can take audio in and return spoken audio, or a cascaded pipeline can split recognition, language-model reasoning and speech synthesis into separate stages. The choice affects conversational flow, implementation control and who operates the voice stack. How does speech-to-speech work in Microsoft Foundry voice agents? It starts with a streamed audio connection and a model choice that determines the architecture.
Speech-to-speech versus a cascaded pipeline
Microsoft documents two voice architectures. In native speech-to-speech, one real-time model receives audio and produces spoken output. A cascaded system separates the work: speech recognition converts audio to text, a text model reasons over it, and speech synthesis turns the response into audio. The service derives the architecture from the model selected; there is no separate architecture switch. Microsoft states: “The service derives the architecture, real-time or cascaded, from the model you select.” (Microsoft Learn, “Configure a voice agent”.)
As an Amazon Associate I earn from qualifying purchases.
| Consideration | Native speech-to-speech | Cascaded pipeline |
|---|---|---|
| Conversation flow | Positioned for natural, latency-sensitive interaction, including interruptions and backchanneling. | Separate processing stages add boundaries between recognition, reasoning and synthesis. |
| Component choice | More integrated: the selected real-time model handles audio input and spoken output. | More flexibility to choose text models, voices, locales and transcription behavior. |
| Implementation ownership | With a managed voice-based prompt agent, Foundry and Voice Live orchestrate the voice interaction. | Can be assembled from distinct components; a hosted-agent setup also lets the developer own its framework and endpoint behavior. |
| Availability | Model and region availability depend on the Foundry deployment. | Options likewise depend on selected models and regions; check the target resource and current service documentation. |
Neither architecture guarantees a particular end-to-end response time. A model’s stated latency is not the full duration a user experiences: processing stages, tool calls and network conditions can add delay. Microsoft recommends prioritizing time to first audio and keeping prompts and tool inventories focused. (Configure a voice agent; Voice Live best practices.)
Two ways to run a Foundry voice agent
Architecture is only part of the decision. Foundry supports managed voice-based prompt agents as well as code-based hosted agents; they place different responsibilities on the application team.
Managed voice-based prompt agent
You configure the agent’s model, instructions and voice behavior in Foundry, while Foundry and Voice Live manage the voice orchestration. The live route uses a WebSocket connection. The client authenticates the WebSocket upgrade with a Microsoft Entra bearer token. (Configure a voice agent.)
Code-based hosted agent
A hosted agent runs a developer-supplied container and framework. Microsoft documents invocations_ws, a persistent, full-duplex WebSocket protocol that relays text and binary frames end to end, allowing the client and agent to stream audio in both directions. The hosted-agent version must declare that protocol. Microsoft lists Voice Live, Pipecat and LiveKit Agents as validated sample framework options; the application team remains responsible for its framework and endpoint behavior. (Build a voice agent with hosted agents.)
Rank #2
Connecting Voice Live to a hosted agent
Voice Live can integrate with hosted agents through Responses or Invocations. In an Invocations integration, the agent accepts transcription input, emits text through the documented output_audio_transcription server-sent events, and identifies its compatibility in the agent manifest. (Integrate Voice Live with hosted agents.)
What the live audio connection does
WebSocket streaming keeps a live, bidirectional connection open while the conversation is in progress, rather than treating each turn as an isolated upload and download. For a managed voice-based prompt agent, the client opens the documented real-time route and presents a Microsoft Entra bearer token during the upgrade. For a hosted agent using invocations_ws, the connection relays text and binary frames between client and agent. These are related streaming approaches, but they are not interchangeable implementation details: use the protocol and authentication documented for the agent type you deploy. (Configure a voice agent; Build a voice agent with hosted agents.)
What you configure—and what affects conversation quality
A voice-agent definition brings together the model and instructions with audio handling, output behavior and tools. Configuration choices shape both what the agent says and how it handles turns.
- Model and instructions: Choose a supported model and give it concise, speech-friendly instructions. Spoken answers should be easy to follow aloud; focused prompts and a limited, relevant tool inventory can help reduce avoidable delay.
- Greeting and turn detection: An optional greeting and the selected turn-detection behavior influence how a conversation begins and when the system treats a user turn as complete. Microsoft documents server-side VAD and semantic VAD options.
- Input audio: Configuration includes input transcription and audio processing choices. Near-field noise reduction is intended for headsets and handsets; far-field processing is intended for speakerphones and rooms. Echo cancellation can help when the agent’s output may be picked up again by the microphone. (Configure audio for a voice agent.)
- Output: Select the voice and output modalities available for the chosen configuration. A managed real-time voice path is integrated, while a cascaded design can give developers separate choices for synthesis voice and locale.
- Tools: Function tools are executed by the client. MCP and toolbox tools are service-connected options, so tool choice affects the integration and ownership model as well as the agent’s capabilities.
When the agent must wait for a tool, Microsoft recommends considering an interim response so the user is not left with silence. A headset with a microphone can be useful for local development checks, but it is not a requirement of the cloud architecture; Microsoft’s hosted-agent walkthrough lists working microphone and speakers among its prerequisites. (Build a voice agent with hosted agents.)
Rank #4
Locale, model availability and pricing need deployment-time checks
Microsoft’s Voice Live overview states that speech-to-text supports over 140 locales and that more than 600 standard text-to-speech voices are available across 150+ locales. These are figures on Microsoft’s overview page, whose publication year is not stated; they are not independent performance measurements or a guarantee that every model supports every locale. Microsoft also notes that supported models and regions vary. Check the current supported-model and region details for the Foundry deployment you intend to use. (Voice Live overview; Voice Live regions.)
Microsoft’s Voice Live pricing page says its listed pricing took effect July 1, 2025, and groups model choices into Pro, Standard and Lite tiers. The page also identifies possible separate training and hosting charges for custom speech, custom voice and custom avatar. Because prices, model lists, regional availability and preview labels can change, consult the current pricing page and supported-region information for your target resource rather than assuming a tier or feature is available everywhere. (Voice Live pricing; Voice Live regions.)
Quick Recap
How to choose the architecture
- Start with the interaction. If interruptions, backchanneling and a natural, low-latency conversational feel are central, consider a native real-time speech-to-speech model.
- Decide how much component control you need. If you need to select or tune recognition, text reasoning, voice, locale or transcription behavior separately, a cascaded design may fit better.
- Choose the operating model. Use a managed voice-based prompt agent when you want Foundry and Voice Live to handle voice orchestration. Choose a hosted agent when your team needs to run its own container and framework and can own endpoint behavior.
- Verify deployment specifics. Check the model, region, feature status and current pricing against the actual Foundry resource. Preview capabilities may have limited support and are subject to their current preview terms.
- Test the whole turn, not just the model. Measure time to first audio and full response behavior with the intended network, tools and audio setup. Model latency alone does not represent the complete user experience.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
