To build a reliable voice agent, start with one spoken task and choose an architecture that fits its interaction and control needs. Then implement the session and tool flow, protect consequential actions, and test completed application outcomes—not just whether the conversation sounds convincing.
Choose an architecture for the interaction you need
There is no universally best voice-agent design. The key trade-offs are whether you need direct access to intermediate text, whether an existing backend should remain in place, how important natural turn-taking and interruptions are, and how much session and transport coordination your application can own.
As an Amazon Associate I earn from qualifying purchases.
| Architecture | How it works | Best fit | Main trade-off |
|---|---|---|---|
| Full-duplex conversation with delegated backend work | A live voice interface listens and speaks while separate backend reasoning or tools handle work. | Keeping an established text-agent workflow, choosing a backend independently, or allowing backend tasks to continue while the caller speaks. | Requires coordination between the live voice interaction and delegated backend work. |
| Speech-to-speech Realtime session | A single session handles audio turns, tools, interruptions, and handoffs. | Immediate-feeling conversation where natural turn-taking and barge-in are central. | Session lifecycle, transport, and event handling become part of the application design. |
| Chained speech pipeline | The application coordinates speech-to-text, an agent workflow, and text-to-speech as separate stages. | Inspecting or transforming intermediate text, storing transcripts, applying policy checks, or replacing components individually. | The application coordinates multiple stages, which can make timing and orchestration more involved. |
Make the choice based on workflow and deployment constraints, not a model leaderboard. A staged pipeline is useful when the transcript needs inspection; a Realtime session fits when conversational timing and interruption handling are core; delegated work can preserve an existing backend while the voice experience remains live.
Start with one narrow task and a clear success condition
Pick a task with a result the application can verify, such as answering a policy question or booking an appointment. A broad instruction like “help the caller” is difficult to evaluate and gives the system an unclear boundary. Instead, define the intended outcome, required information, permitted actions, and what to do when the caller’s request is outside scope.
#1 Best Overall
- The Raspberry Pi Raphael Starter Kit for Beginners: The kit offers a rich learning experience for beginners aged 10+. With 337+ components, 161 projects, and 70+ expert-led video lessons, this kit makes learning Raspberry Pi programming and IoT engaging and accessible. Compatible with Raspberry Pi 5/4B/3B+/3B/Zero 2 W /400, RoHS Compliant
- Expert-Guided Video Lessons: The Raspberry Pi Kit includes 70+ video tutorials by the renowned educator, Paul McWhorter. His engaging style simplifies complex concepts, ensuring an effective learning experience in Raspberry Pi programming
- Wide Range of Hardware: The Raspberry Pi 5 Kit includes a diverse array of components like Camera, Speaker, sensors, actuators, LEDs, LCDs, and more, enabling you to experiment and create a variety of projects with the Raspberry Pi
- Supports Multiple Languages: The Raspberry Pi 4 Kit offers versatility with support for 5 programming languages - Python, C, Java, Node.js and Scratch, providing a diverse programming learning experience
- Dedicated Support: Benefit from our ongoing assistance, including a community forum and timely technical help for a seamless learning experience
- Define the task: State what the agent should accomplish in one interaction.
- List required inputs: Identify details the caller must supply, including how to handle missing or ambiguous information.
- Specify allowed tools: Name the actions the agent may request and the conditions for using them.
- Set completion criteria: Define which application state or tool result proves success.
- Make confirmations truthful: The spoken response should reflect what the application actually completed, not what the agent intended to do.
Build the session and transport flow
For a browser-first Realtime implementation, the documented flow is for the application server to create an ephemeral client secret, the frontend to create a Realtime session, and the session to connect using WebRTC in the browser or WebSocket on a server. The session can manage audio turns, tools, interruptions, and handoffs. Phone deployments have separate telephony/SIP guidance; using a related transport does not make API modes interchangeable in their handshakes, credentials, or event formats.
The TypeScript Agents SDK provides helpers including RealtimeAgent and RealtimeSession, alongside transport handling. Its guides cover WebRTC, WebSocket, SIP, custom transports, turn detection, interruptions, function tools, handoffs, guardrails, and tracing.
Rank #2
- Pi5 8GB Pack: RasTech Pi 5 8GB kit includes 1 x Pi5 8GB board ,1 x 64GB Card, 2 x Card Readers,1 x Active Cooler,1 x Case for Pi5, 2 x 4K Micro HD Out Cable,1 x GaN 27W 5A USB-C Power supply,1 x Screwdriver and 1 x instructions.
- Pi5 8GB Board: The Pi5 board is equipped with a 64-bit quad-core Arm Cortex-A76 processor running at 2.4GHz and an 800MHz VideoCore VII GPU with support for OpenGL ES 3.1 and Vulkan 1.2, which delivers a significant increase in graphics performance. Dual HD Out 4Kp60 display outputs and a built-in dual 4-channel MIPI camera/display transceiver provide state-of-the-art camera support. The Pi 5 offers a 2-3 times increase in CPU performance compare to Pi4.
- Important Graphics Features: Equipped with an 800MHz VideoCore VII GPU and providing better graphics performance, suitable for multimedia applications,gaming,and graphics intensive tasks.Provides 1 UART interface,1 card slot that supports high-speed operation, 2 USB. 3 0.5 ports that support synchronous 0Gbps operation,2 USB 2.0 port ports,2 4Kp60 display outputs that support HDR.Built-in dedicated dual 4-channel 1Gbps MIPI DSI/CSI connectors,triple the total bandwidth.
- Cooling Kit for Pi 5: Compatible with Active Cooler for Raspberry Pi5, It can provide Pi 5 board with better cooling effect in using. The Case can accurately access usb-c power jack,Micro HD Out ports, usb ports, Ethernet jack, card slot, power button, 4-lane MIPI DSI/CSI connectors and so on, and it also supports installation of cooling fan.
- 64GB Card Kit and GaN 27W USB-C Power Supply: With extra 64GB card to store more files and card readers for multiple medium, keep better performance for Raspberry Pi 5, 27W USB C Power Supply is Compatible with Pi5 8GB, offers a variety of output voltage options, including 5.1V at 5A, 9.0V at 3.0A, 12.0V at 2.25A, and 15.0V at 1.8A, providing for different device requirements.
Keep privileged work in a trusted environment
Function tools run in the same environment as the session. Do not place privileged logic or secrets in a browser session. Have sensitive operations call a trusted backend that can enforce authentication, authorization, validation, and application rules.
Recommended Free Tools
Account for the current API interface
As of October 2026, the Realtime getting-started guidance advises applications using the beta interface to migrate to the GA interface before continuing new work. The migration covers the beta header, ephemeral client secrets, session and event shapes, and event names. These implementation details can change, so check the current migration guidance for the interface you are deploying.
Rank #3
- 1. Barebones Kit for Raspberry Pi CM4 – Made for users who want to add their own Raspberry Pi CM4 module to a portable handheld Linux computer kit.
- 2. Multiple Configurations Available – Choose from with WiFi-only or WiFi + 4G LTE connectivity options.
- 3. 5 Inch QWERTY Handheld Design – Features a 5 inch IPS display, compact QWERTY keyboard, mini trackball, speakers, and handheld cyberdeck-style layout.
- 4. ClockworkPi v3.14 Rev 5 Mainboard – Uses the ClockworkPi v3.14 revision 5 mainboard platform with CM4 adapter support for Raspberry Pi CM4 users.
- Important Package Note – Raspberry Pi CM4 module, 18650 batteries, TF card, SIM card, and cellular service plan are not included.
Give tools and consequential actions explicit boundaries
Voice agents use familiar agent building blocks: tools, orchestration, handoffs, guardrails, approvals, and observability. Put speaking style and conversational behavior in the voice prompt; enforce business rules and permissions in the backend where appropriate. Do not rely on a spoken instruction alone to authorize an action.
For actions with meaningful consequences, make permission checks and any required review explicit in the workflow. If the application identifies individual end users, the Realtime guidance recommends sending a stable, privacy-preserving safety identifier with requests. That is an API recommendation, not a universal legal requirement.
Rank #4
- Voice assistant complete set: with XIAO ESP32S3, XMOS XU316, 2-microphone array, 5W speaker and acrylic housing.
- 2 MICROPHONE ARRAY: Two digital MEMS microphones, 3m remote field recording, noise reduction for clear voice recognition.
- 5 W mono speaker: integrated amplifier, clear sound, additional 3.5 mm jack output.
- Acrylic casing: laser-cut matte black kit housing, easy to assemble yourself.
- Open and compatible: supports Arduino, Raspberry Pi and Home Assistant for your own language projects.
Evaluate task completion, timing, and reliability separately
A pleasant voice or a successful connection does not establish that the agent did the right thing. Save the audio, event traces, tool results, and resulting application state for representative test scenarios. Distinguish a genuine task failure from a broken evaluation run: the former is evidence about the agent, while the latter means the test did not validly assess it.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteCheck outcomes, not just responses
- Task outcome: Did the agent preserve the caller’s intent, choose the right tool, use correct arguments, respect permissions, and produce the intended application state?
- Conversation timing: Was the useful spoken response timely? Was there unwanted silence or overlap? Did the agent yield when interrupted?
- Speech and language: Test recognition with different accents, background noise, language changes, names, and numbers. Assess output intelligibility and language choice separately.
- Reliability: Record connection failures, dropped audio, timeouts, and sessions that end before the task is complete.
- Human listening: Have people assess pronunciation, naturalness, and pacing alongside automated scores.
Make latency comparisons fair
When comparing latency, hold the caller, recording, backend model, prompt, transport, audio cadence, and grader fixed. Compare median and 95th-percentile latency across comparable calls, and separate backend processing time from the time until the caller hears a useful answer. Change one factor at a time and rerun the same scenarios.
Best Value
- Vilros Complete Starter Kit for Pi 4 Includes Raspberry Pi 4 Model B Board and all the accessories you need to get started.
- 9-PART KIT WILL HAVE YOU READY TO GET UP AND RUNNING: Kit Includes 1. Raspberry Pi 4 Model B Board 2. Case With Easy to connect Built-in fan 3. 64GB Micro SD card Preloaded with RP OS 4. Vilros Pi 4 Compatible Power Supply with Inline on/off switch (power supply color may vary white/black) 5. Micro HDMI to Standard HDMI cable (5ft) 6. Micro SD to USB adapter to reflash card if desired 7. Neoprene Storage Bag to store all parts when not in use 8. Set of 4 Heatsinks 9. Vilros QuickStart Guide instruction booklet for Pi 4
- PASSIVE & ACTIVE COOLING: The included case is well-vented and the kit also includes a set of heatsinks with thermal stickers for easy application and a pre-installed fan to keep the board cool in any use.
- CONVENIENT ACCESSORIES: The power supply features an inline on/off switch neoprene bag that holds and protects all the parts when not in use and the QuickStart guide is updated and written for Raspberry Pi 4.
- IMPORTANT: Kit does NOT include Keyboard, Mouse or Monitor
Increase test realism in stages
- Fixed synthetic speech: Start with repeatable single-turn cases so you can isolate basic behavior and tool outcomes.
- Representative human recordings: Add variation in voices, microphones, pauses, and acoustic conditions.
- Independent simulated callers: Test multi-turn conversations involving clarification, interruptions, changed requirements, and recovery.
For a booking scenario, a valid success check is that the appointment was actually saved and the agent’s confirmation matches that saved state. A fluent confirmation without the corresponding state change is a task failure.
Interpret model benchmarks as component evidence
In its August 28, 2025 announcement, OpenAI said its gpt-realtime model processes and generates audio directly through one model and API. The company reported 82.8% accuracy on its Big Bench Audio evaluation, compared with 65.6% for its prior December 2024 model on that same reported evaluation. Those are OpenAI-published benchmark results, not proof of general task success or an independent comparison across vendors. Your application still needs tests that reflect its users, tools, permissions, and failure conditions.
A September 25, 2026 arXiv preprint argues that no single metric captures real-time voice-agent performance and proposes looking together at timing, recovery after disruptions, and grounded outcomes. It is a recent research framing, not a settled standard. Its practical implication is straightforward: low latency or a strong component score alone cannot show that an agent reliably completed the task it was built for.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
