# Building Low-Latency Voice AI Agents: WebRTC, LiveKit, and Streaming LLMs > **Published**: 3 November 2026 | **Reading Time**: 9 min read | **Author**: Mohammad Shadikur Rahman > **Tags**: Voice AI, WebRTC, LiveKit, LLMs, DevOps, Software Engineering > **Canonical Article**: https://shadikur.com/blog/building-low-latency-voice-ai-agents-webrtc-livekit --- ## Abstract / Excerpt Learn how to architect sub-second conversational voice AI agents using WebRTC, LiveKit, streaming Speech-to-Text, and speculative TTS pipelines. --- ## Article Content Building a text-based chatbot with a Large Language Model is straightforward today: you open an HTTP connection or WebSocket, stream Markdown tokens, and display them on a screen. When users read text, a latency of 1 to 2 seconds feels completely natural. In conversational voice, however, that same latency feels like an awkward pause on an international call. Human conversation operates on tight turn-taking rhythms. Human turn-taking often involves gaps of only a few hundred milliseconds, although timing varies by language, speaker, and context. For voice agents, a long pause after the caller finishes speaking can feel unnatural. Instead of treating any single threshold as universal, measure end-of-turn-to-first-audible-response latency and its p50, p95, and p99 distributions. To build genuinely conversational voice AI agents, you cannot simply glue sequential REST APIs together. You need a dedicated real-time media pipeline built on low-latency transports like WebRTC, streaming voice activity detection (VAD), token-level LLM streaming, and speculative chunked text-to-speech (TTS). --- ## Deconstructing the Sub-Second Latency Budget A sub-600 ms end-of-turn-to-first-audio response is an ambitious optimization target, not a guaranteed baseline. Measure each stage, account for overlap, and include network conditions. Streaming STT often processes speech before the user finishes; its total processing time should not automatically be added to post-turn latency. ``` [ User Speaks ] │ ▼ (50 - 150ms) [ Voice Activity Detection (VAD) ] ── User stopped speaking │ ▼ (100 - 200ms) [ Streaming Speech-to-Text (STT) ] ── Final transcript chunk │ ▼ (150 - 300ms) [ LLM Time-to-First-Token (TTFT) ] ── Initial sentence clause ready │ ▼ (80 - 150ms) [ Streaming Neural TTS ] ──────────── First synthesized PCM audio frame │ ▼ (20 - 50ms) [ WebRTC Media Egress (UDP) ] ────── Audio arrives at user ear ``` | Component | Target Latency | Optimization Strategy | | :--- | :--- | :--- | | **VAD & Endpointing** | 80–160 ms | Silero VAD running locally on frame chunks; dynamic silence windows. | | **Streaming STT** | 100–220 ms | Deepgram Nova-3, AssemblyAI, or local streaming Whisper via WebSocket/gRPC. | | **LLM TTFT** | 150–300 ms | Fast reasoning models (Cerebras, Groq, Claude 3.5 Haiku, GPT-4o-mini). | | **Neural TTS** | 80–180 ms | Chunk-based streaming (Cartesia Sonic, ElevenLabs Flash, Deepgram Aura). | | **Network & Transport** | 20–60 ms | WebRTC SRTP / UDP edge routing via LiveKit or Cloudflare Calls. | | **Illustrative sum** | **430–920 ms** | **Not a measured round trip: stages can overlap, and endpointing, buffering, and jitter affect actual first-audio delay.** | --- ## WebRTC vs. WebSockets: Choosing the Right Transport Many voice AI systems use WebSockets successfully, especially between backend services. For interactive browser or mobile audio, WebRTC offers advantages in media timing and network adaptation. The important trade-offs are: 1. **Head-of-Line (HoL) Blocking:** TCP guarantees lossless, in-order packet delivery. If a single audio packet drops over a mobile network, TCP's ordered delivery can delay later data after loss. For live audio, WebRTC's media stack can prioritize timely playout over perfect delivery; WebSockets remain useful for server-to-server streams. 2. **Buffer Bloat & Acoustic Echo:** TCP retransmission and buffering can add delay under loss or congestion. Actual behavior depends on application buffering, network quality, and implementation. 3. **No Native Hardware DSP:** Browser WebRTC capture APIs can enable **Acoustic Echo Cancellation (AEC)**, Automatic Gain Control (AGC), and noise suppression where supported. These features depend on the capture path and device; AEC helps reduce the agent's playback leaking into the microphone. ### Why LiveKit as the Media Plane? Using raw peer-to-peer WebRTC between a browser and a Python server requires writing complex STUN/TURN handling, ICE negotiation, and custom RTP packetizers. **LiveKit** solves this by acting as a high-performance Selective Forwarding Unit (SFU) written in Go. Instead of maintaining fragile direct WebRTC sockets, your agent connects as a participant in a LiveKit room. It consumes the caller's incoming audio track, processes it, and publishes the synthesized voice track back to the room with low-latency media routing; actual overhead depends on topology, network, and deployment. ``` +──────────────────────+ | Client / Browser | | (WebRTC SDK + AEC) | +──────────────────────+ │ │ WebRTC (SRTP/UDP) ▼ +──────────────────────────────────────────────────────────+ | LiveKit SFU Cluster | | (Distributed Media Routing) | +──────────────────────────────────────────────────────────+ ▲ │ High-speed local RPC / Egress ▼ +──────────────────────────────────────────────────────────+ | Voice AI Agent Worker | | | | 1. Silero VAD (Frame-by-frame speech detection) | | 2. Streaming STT (Chunked audio via gRPC) | | 3. Streaming LLM Engine (Prompt + Tool Context) | | 4. Chunked TTS Synthesizer (Sentence-boundary split) | | 5. WebRTC Track Publisher (Opus 48kHz audio frames) | +──────────────────────────────────────────────────────────+ ``` --- ## Critical Architectural Challenges Building a robust voice agent involves solving three engineering problems that rarely occur in text chat. ### 1. Speculative Text-to-Speech Chunking You cannot wait for an LLM to generate an entire 50-word paragraph before passing it to TTS. You must stream tokens to the TTS engine as they arrive. However, feeding raw tokens word-by-word into a TTS model produces disjointed, robotic cadence because neural voice models need context to calculate natural prosody, pitch, and intonation. A common approach is **clause-aware buffering**, tuned to the TTS model and desired prosody: - Buffer tokens until a meaningful clause or sentence boundary; avoid splitting blindly on abbreviations or every comma. - Use a configurable minimum chunk length and flush policy, balancing first-audio delay against prosody. - Decouple token consumption, TTS synthesis, and audio publishing so work can overlap. Maintain ordered playback and enforce backpressure. ```python # Illustrative async pipeline; adapt synthesize_stream() and publish() # to your provider's SDK. Cancellation and error handling are omitted. import asyncio async def speak_stream(llm_token_stream, tts_client, audio_track): text_queue = asyncio.Queue(maxsize=8) audio_queue = asyncio.Queue(maxsize=8) END = object() async def collect_text(): buffer = "" async for token in llm_token_stream: buffer += token if len(buffer.split()) >= 6 and buffer.rstrip().endswith((".", "?", "!", ";", ":")): await text_queue.put(buffer.strip()) buffer = "" if buffer.strip(): await text_queue.put(buffer.strip()) await text_queue.put(END) async def synthesize(): while True: chunk = await text_queue.get() if chunk is END: break async for frame in tts_client.synthesize_stream(chunk): await audio_queue.put(frame) await audio_queue.put(END) async def publish(): while True: frame = await audio_queue.get() if frame is END: break await audio_track.write_frame(frame) async with asyncio.TaskGroup() as tasks: tasks.create_task(collect_text()) tasks.create_task(synthesize()) tasks.create_task(publish()) ``` ### 2. Handling Barge-In & Interruptibility In real human conversation, people interrupt each other. If the agent is reading a detailed explanation and the user says *"Wait, let me change that,"* the agent must cut off immediately. When user speech is detected while the agent is speaking: 1. **Detect genuine caller speech:** Combine VAD with echo-aware checks and configurable interruption thresholds; a fixed 120 ms rule is not reliable in every environment. 2. **Cancel active generation:** Cancel the current LLM stream and in-flight TTS work where the provider supports cancellation. 3. **Clear queued playback:** Discard application-managed TTS queues and stop or clear the playback buffer using the client SDK's supported interruption controls. Sending a silent RTP packet does not universally flush WebRTC playback. 4. **Reconcile conversation state:** Record only the assistant audio actually played (or a best-effort playback acknowledgment), not the entire generated answer, before processing the interruption. ### 3. Bridging into Telephony & PBX Systems (SIP / PSTN) While WebRTC works seamlessly on web and mobile, enterprise customer support often requires connecting to traditional telephone networks (PSTN) or cloud PBX systems (FreeSWITCH, Asterisk, Kamailio). LiveKit SIP or external SIP-to-WebRTC gateways bridge this gap: - Inbound phone calls arrive via SIP trunking (e.g. Twilio, Telnyx, or carrier direct). - The gateway can decode G.711 μ-law / PCMU telephony audio at 8 kHz, resample it as needed, and encode it for the WebRTC media path (often Opus at a 48 kHz sampling rate). Upsampling does not recreate high-frequency detail absent from the telephone signal. - The Voice AI agent interacts with the caller identically whether they are calling from an iPhone browser or a traditional landline. --- ## Measuring Real-World Voice Latency Instrument timestamps for user speech end, endpointing decision, final STT result, first LLM token, first synthesized audio frame, and first audible client playback. Report p50/p95/p99 end-of-turn-to-first-audio latency, plus interruption cancellation time and the rate of false turn endings. Compare results across Wi-Fi, cellular loss, and SIP/PSTN calls. The table above is an illustrative budget, **not a benchmark from a deployed system**. For reference, consult [LiveKit Agents documentation](https://docs.livekit.io/agents/), [LiveKit SIP documentation](https://docs.livekit.io/sip/), [MDN WebRTC documentation](https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API), and [WebRTC audio capture constraints](https://developer.mozilla.org/en-US/docs/Web/API/MediaTrackConstraints/echoCancellation). Provider-specific latency figures should be validated with current documentation and your own measurements. --- ## Architectural Checklist for Production Before moving a conversational voice pipeline to production, verify these benchmarks: - [ ] **Acoustic Echo Cancellation (AEC):** Activated and tested on both desktop and mobile web clients. - [ ] **Turn Detection Window:** Tune endpointing and silence thresholds against real recordings; 200–350 ms is an aggressive starting range, not a universal setting. Evaluate interruptions, accents, background noise, and language-specific pauses. - [ ] **First-Packet Synthesizer Latency:** TTS provider returns the first playable PCM audio frame within 150ms of receiving the initial text chunk. - [ ] **Graceful Degradation:** If network jitter spikes, the playback buffer dynamically expands slightly rather than stuttering or dropping words completely. - [ ] **Session State & Tool Execution:** Heavy tool calls (e.g. CRM queries or database writes) are executed asynchronously or accompanied by verbal filler responses (*"Let me look up your account details..."*) to mask processing lag. --- ## Conclusion Conversational voice AI is fundamentally an audio engineering and distributed systems problem, not just a prompt engineering task. By combining WebRTC’s low-latency media transport, LiveKit’s scalable room routing, and overlapping STT, LLM, and TTS stages, you can reduce perceived response delay while retaining a measurable latency budget. As speech models and inference silicon continue to accelerate, real-time voice will rapidly become the primary interface for autonomous customer operations and interactive applications. --- ### Frequently Asked Questions **How do you reduce voice AI response latency?** Stream STT during the caller's speech, optimize endpointing, choose a low-TTFT model, synthesize speech in chunks, and measure the time until audio is actually heard. **Do I need WebRTC to use LiveKit?** LiveKit's real-time room media uses WebRTC; telephone callers can enter through its SIP bridge, while other backend integrations may use different protocols. **How should an agent handle interruptions?** Detect incoming speech, cancel active model and synthesis work, clear queued audio through supported playback APIs, and reconcile the conversation with what the caller actually heard. --- ### Related Posts - [Automating Business Ops with MCP: Connecting LLM Agents to Internal CRMs](https://shadikur.com/blog/automating-business-ops-mcp-crm-integration) - [How Playwright MCP and Local LLMs Are Changing QA Automation](https://shadikur.com/blog/playwright-mcp-local-llms-qa-automation) - [The Hugging Face AI Agent Breach: What Happened and Why It Matters](https://shadikur.com/blog/hugging-face-ai-agent-breach-what-happened) --- ## Author Biography Mohammad Shadikur Rahman is a Senior Infrastructure Architect and Full Stack Engineer based in Hamburg, Germany. Specializing in AWS CDK, Cloudflare R2, DevOps, microservices, and VoIP platforms. - **Blog Index**: https://shadikur.com/blog.txt - **LLM Index**: https://shadikur.com/llms.txt