Optimizing Inference for Real-Time Voice AI: VAD, ASR, and TTS Pipelines

# Optimizing Inference for Real-Time Voice AI: VAD, ASR, and TTS Pipelines

*By the OnlineCloud engineering team · Audio Systems · ~10 min read*


Voice is the least forgiving interface we build against. A chat user tolerates a two-second spinner; a caller on a voice assistant will not — they’ll talk over the bot, repeat themselves, hang up, or assume the system is broken. Every stage of a conversational voice pipeline is an inference problem, and each has a latency budget measured in milliseconds, not seconds.

This post is a field guide to optimizing the four-model pipeline behind real-time voice AI — voice activity detection (VAD), automatic speech recognition (ASR), the LLM that generates the reply, and text-to-speech (TTS) — plus the transport that glues them together. It’s written from the trenches: what we benchmark, what breaks at the 99th percentile, and how we make trade-offs on real hardware.

## The p99 Latency Challenge

Conversational latency is a distribution problem, and most teams measure it wrong. Optimizing the median turn time means optimizing for a caller who never exists — the p50 hides the spikes: a GC pause in the ASR worker, a cold TTS model load, a WebSocket reconnect, an audio thread that got starved. Callers feel the p99. One bad turn in a hundred is enough to sink a session, because in voice a late response isn’t slow, it’s *wrong*: the user has already started speaking again.

Psychophysics gives us the frame. Humans perceive a conversational partner as lagging at roughly 500–700 ms of response delay, and treat anything past ~800 ms–1 s as a dropped turn. The working target for a voice assistant is therefore **≤ 800 ms from end of user speech to start of system speech**, with a hard p99 ceiling near 1.2 s. Budget that back through the pipeline and it’s sobering:

| Stage | Budget |
|—|—|
| VAD endpointing + margin | 50–100 ms |
| ASR (transcribe user turn) | 150–300 ms |
| LLM first token (TTFT) | 150–300 ms |
| TTS first audio frame | 100–250 ms |
| Transport, jitter, codec | 50–100 ms |
| **Total** | **500 ms – 1.05 s** |

Everything below is about defending that budget at the tail, stage by stage.

## Voice Activity Detection: Cutting the Turn Cleanly

The VAD is the cheapest model in the pipeline and the one that most determines perceived latency, because it decides *when* the clock starts. If the VAD holds the channel open 300 ms too long waiting for a user to finish, no amount of ASR or LLM optimization can win that time back — it’s dead air, spent before the transcript even exists.

**WebRTC VAD** is the default because it ships with the browser and in most telephony stacks: a lightweight Gaussian-mixture classifier over 10/20/30 ms frames of 8 or 16 kHz audio, consuming single-digit percentages of one core with zero load cost. Its weaknesses are well known — it’s trained on clean-ish speech, so it degrades on noise, music, and cross-talk, and its aggressive modes clip soft speech onsets. For a quick “is someone talking” gate it’s fine. For endpointing a real conversation, it’s the wrong tool.

**Silero VAD** is where we land for production. It’s a small ONNX neural model (~2 MB) that scores 512-sample chunks (32 ms at 16 kHz) in well under a millisecond on CPU, and it generalizes far better to noise, reverberation, and varied microphones. Critically for latency, Silero returns a *probability curve*, not a boolean, which enables proper endpointing state:

– **Speech start**: probability crosses `threshold` (we run 0.5) for `min_speech` consecutive chunks — this filters one-off noise spikes.
– **Speech end**: probability stays below threshold for `max_silence_ms` of *hangover* (we use 450–600 ms in conversational mode).
– **Padding**: prepend ~200 ms of pre-roll audio to the ASR segment so the recognizer sees the consonant onset, not a clipped first phoneme.

The implementation detail that matters: run Silero synchronously on the audio thread with a fixed 32 ms hop and no buffering backlog — if VAD processing ever queues, endpointing latency becomes a function of GC pressure instead of physics. In our production stack, VAD decision latency sits at 50–100 ms including hangover, and because Silero scores each chunk as it arrives, the moment the user stops we are *already* emitting the cut audio to ASR rather than waiting to “detect” silence after the fact.

## ASR: Whisper v3, Faster-Whisper, and Groq

ASR is where the accuracy-vs-latency tension is sharpest, because the model that transcribes best (Whisper large-v3) is not the one that transcribes fastest, and the ones that are fast are finicky about how you feed them audio.

**Whisper v3** (open weights) is our accuracy reference. Its encoder-decoder architecture with a 30-second context window makes it superb at disambiguating homophones, numbers, and accented speech — but it’s fundamentally a *batch* model. It expects a complete utterance (or a padded window) and emits no partial hypotheses. Running it live means accepting full-utterance latency: the turn must be collected, then transcribed. For short turns (2–5 s) that’s 300–700 ms on a decent GPU — workable, but only if you never re-run it.

**Faster-Whisper** is the same model reimplemented on CTranslate2, and it’s the default drop-in: most of v3’s accuracy at a fraction of the cost. With int8 quantization it runs 3–5× faster than the reference implementation — real-time or better on CPU for small/medium models, multiple-× real-time on GPU for large-v3. It also exposes per-word timestamps, which are gold for voice: compute the *true* end-of-speech from the last word’s timestamp and trim the VAD hangover out of the transcript, shaving another 100–200 ms.

**Groq** is the interesting outlier: Whisper large-v3 on LPU hardware as a hosted API, with an absurdly fast and *stable* encoder — a 10-second turn transcribes in roughly 100–200 ms wall-clock, and the variance is tiny because there’s no shared-GPU neighbor noise. The trade-off is architectural: it’s an HTTP API, so you send a complete audio payload per turn, reintroducing network round-trips. Groq wins when your p99 is dominated by GPU contention; self-hosted Faster-Whisper wins when audio must never leave your VPC.

Our rule of thumb: **partials or nothing.** Don’t block the LLM on the full transcript. If your ASR emits word-level partials, feed the LLM the running hypothesis, let it start drafting against a possibly-incomplete prompt, and only commit the final transcript when the VAD fires end-of-speech. This “warm start” overlaps ASR tail latency with LLM prefill and is worth 150–300 ms per turn — more than any single model swap.

## LLM Inference: First-Token Latency and Speculative Decoding

Once the transcript is ready, the LLM becomes the critical path, and the metric that governs voice is **time-to-first-token (TTFT)**, not tokens/second. Users don’t hear your throughput; they hear the silence before the first word. We target TTFT ≤ 200–300 ms at p99, which forces a very different serving configuration than a chatbot.

Token generation splits into two regimes. **Prefill** processes the whole prompt in parallel and dominates TTFT — its cost scales with prompt length and model size. **Decode** is autoregressive and memory-bandwidth-bound, producing one token at a time; it dominates total response time but is irrelevant to the *first* audible syllable, because TTS can start on the first few tokens (more below). Voice serving is therefore a prefill-optimization problem first: keep prompts short (transcript plus a compact system prompt, no retrieval dump), use an engine with continuous batching (vLLM, TGI) so a prefill never queues behind a long decode, and prefix-cache the immutable system prompt.

**Speculative decoding** attacks the decode regime. The idea: a small, fast *draft* model proposes K tokens; the big model verifies all K in a single parallel forward pass and accepts those matching its own distribution (rejection-sampling the rest). Since verification is parallel and drafting is cheap, you get 1.5–3× decode throughput on compatible model families with *identical* output quality. For voice, the win isn’t raw tokens/sec — it’s that faster decode shrinks the gap between first and last token, so the LLM finishes before the TTS queue starves mid-sentence. Implementations range from n-gram/self-speculative drafts (no extra model) to Medusa- and EAGLE-style auxiliary heads, with built-in support in vLLM and TensorRT-LLM.

Two voice-specific serving rules: stream tokens over the socket as they decode (never buffer a full response), and cap max output tokens for the *spoken* reply. A chatbot can ramble for 1,000 tokens; a voice assistant that talks for three minutes unprompted is a bug. We cap spoken replies at 150–250 tokens, which also bounds worst-case turn time.

## TTS: ElevenLabs, Deepgram Aura, and Bark

TTS is the last model in the chain and the one users are most sensitive to: the first audio frame *must* arrive quickly, and the voice must not sound robotic, or the whole interaction reads as fake. The critical metric is **time-to-first-audio (TTFA)** — the gap between “first LLM tokens available” and “audible speech” — and it’s gated by whether the provider streams audio incrementally.

**ElevenLabs** is our quality benchmark. The Flash models (v2.5) are built for low-latency streaming: audio begins arriving ~75–150 ms after the request, with prosody, emotion, and multilingual quality that remain best in class for customer-facing voices. You pay per character — and you must honor its streaming semantics; waiting for the full clip throws away the entire point.

**Deepgram Aura** is the latency play. Aura-2 streams at very low TTFA (we measure ~100–300 ms to first frame), sounds natural, and prices aggressively for high-volume call traffic. Its voices skew “contact-center neutral” versus ElevenLabs’ characterful range — fine for order-status and support deflection, less ideal for brand voice. If your pipeline is cost-sensitive and your p99 is TTS-dominated, Aura is usually the right default, with ElevenLabs reserved for premium tiers.

**Bark** is the open-source option everyone tries once. It’s a generative (GPT-style) audio model with remarkable expressiveness — laughter, music, paralinguistics — but it is *not* a real-time system: it synthesizes in large chunks and typically needs multiple seconds of GPU time per second of audio, with high variance. For live conversation it’s disqualified on latency alone; we use it only for pre-rendered prompts and IVR menus. If you must self-host streaming TTS, look at neural codec models (the XTTS / CosyVoice family) instead — they emit audio in sub-second streaming frames.

The engineering pattern is identical across providers: send the LLM’s token stream to TTS in *chunks* (one sentence, or ~5–10 tokens at a time — never a whole paragraph), play the first frame the moment it lands, and keep synthesizing while playback runs. And because TTS is now streaming, you need the loop’s final piece — the ability to cut it off.

## Orchestrating the Loop: WebSockets and PCM Buffers

A conversational voice agent is not four models; it’s a state machine that moves audio between them. We implement it as a single duplex WebSocket carrying raw audio frames, control messages, and events. The wire format is deliberately boring: **16 kHz, 16-bit, mono PCM**, framed at 20 ms (640 bytes per frame) — the same frame size as Opus, keeping a clean migration path to compressed transport when bandwidth matters.

The loop, as we run it in production:

1. The client opens one WebSocket and streams 20 ms PCM frames up; the server pushes audio frames and control events down. One socket, no polling, no REST round-trip per turn.
2. Each frame enters a fixed ring buffer; Silero VAD scores it in-band. ~200 ms of pre-roll is retained so nothing before speech onset is lost.
3. On end-of-speech, the buffered segment goes to ASR; partial hypotheses stream to the LLM session as they transcribe.
4. The LLM streams tokens down the socket; a scheduler assembles them into TTS-sized chunks.
5. TTS frames stream down and play. If the VAD hears the *user* start speaking again (barge-in), the server sends a `cancel` event: playback stops on the client, LLM generation aborts server-side, and the ASR context resets for the new turn.

Three details separate systems that feel alive from ones that feel laggy:

– **No cascading buffers.** If each stage buffers “a little for safety,” you’ve added 4× the latency you measured. Buffer only at the transport edge; move frames through VAD → ASR → LLM → TTS with explicit, instrumented handoffs.
– **Barge-in must be instant.** The `cancel` event is a first-class control message, not an afterthought. A user who can’t interrupt the bot will hang up.
– **Backpressure with a dead-letter policy.** If the TTS provider stalls, drop queued audio and restart synthesis from the current text offset — replaying a stale sentence into a conversation that has moved on is worse than a half-second hiccup.

Finally, instrument the *p99 of the whole turn*, not of each stage. Independent stage p99s compose pessimistically; the turn-level p99 is what callers feel, and it’s the number we refuse to let drift past ~1.2 s.

## Build It With Us

A real-time voice pipeline is an exercise in latency archaeology — every millisecond is hiding somewhere, and the p99 punishes the stage you forgot to instrument. At OnlineCloud, we ship voice and chat assistants into production in 6–10 weeks: VAD/ASR/TTS integration, LLM serving tuned for TTFT, WebSocket orchestration, and the latency dashboards that keep the system honest after launch. No offshore handoffs, no decks — senior engineers building evaluated, observable systems.

If you’re building a voice agent — a support helpline, an order-status assistant, a trauma-informed intake line — and you want it to feel like talking to a person rather than a pipeline, let’s talk. Email **contact@onlinecloud.dev** for a free initial consultation; we respond within 24 hours.