How we engineered a conversational voice system that actually works
The Reality Check
Voice AI demos are easy. Production is hard. Real phone calls, real network conditions, real users who don't speak perfectly—everything breaks.
This post shares our journey: architectural patterns, not implementation details.
The Cascade Pipeline
User Audio → STT → LLM → TTS → Speaker Audio
Simple in theory. In reality:
- Latency compounds across each hop
- Context bleeds between stateless services
- State fragments in an inherently stateful conversation
The Six Pain Points We Solved
1. Turn Detection
Problem: When does the user stop talking? Humans pause mid-sentence, use fillers, have background noise.
Solution: Layered approach—Voice Activity Detection for audio signals + semantic turn analyzer for conversational understanding.
2. Multilingual Conversations
Problem: Users code-switch constantly. "My age twenty-four hai" is neither pure English nor Hindi—it's how people actually speak.
Solution:
- Language Manager with sticky language state
- Code-switching ≠ language change request
- Domain-specific pronunciation rules for proper nouns
3. The Latency Monster
Problem: Humans expect 200-400ms response time. We were hitting 1-2 seconds.
Solution:
- Early warmup: Prime LLM before user speaks
- Stream everything: TTS speaks while LLM generates
- Intelligent fillers: "Just a moment..." while processing
Result: First response 400-600ms, subsequent 200-400ms.
4. Context Loops
Problem: Bots get stuck asking the same question. Language switches reset context.
Solution:
- Track collected information persistently
- Language switches preserve conversational state
- Anti-loop guardrails in prompt design
5. Tool Execution
Problem: LLMs sometimes expose internal workings: "Let me call the schedule_callback function..."
Solution: Strict output guardrails. Tools execute silently; bot speaks naturally about results.
6. Regional Audio Quality
Problem: Phone audio with compression, jitter, noise, non-standard codecs.
Solution:
- STT correction layer for common transcription errors
- Confidence-based confirmation
- Graceful recovery with rephrased clarification
The Architecture
Six layers, each with clear responsibilities:
| Layer | Responsibility |
|---|---|
| Telephony | WebSocket, audio serialization, call control |
| Audio Pipeline | VAD, turn detection, buffering |
| Speech | STT with corrections, TTS with pronunciation |
| Conversation | LLM, tools, context management |
| State | Language, transcript, analytics |
| Integration | Agent configs, knowledge base, callbacks |
Key Lessons
- Latency compounds — Shave milliseconds everywhere
- Real speech is messy — Multilingual, noisy, unpredictable
- Embrace statefulness — Don't fight it, manage it explicitly
- Guardrails are features — Preventing bad outputs matters
- Warmup is mandatory — Cold starts kill conversations
What's Next
- Semantic VAD with conversation context
- Zero-shot language switching
- Emotion-aware responses
- Predictive response generation
The gap between demo and production is closing. The engineering still matters more than the model.