We Built a Full Real Time Voice AI Operating System and Discovered the LLM Was Never the Real Problem

Most people think voice AI fails because the model is dumb.

After building a production scale outbound and inbound telephony stack, I’m starting to think the opposite is true:

The AI is often smart enough.

The telecom pipeline destroys user intent before the AI ever sees it.

Our stack today:

Telephony 8kHz µ law
PCM decode
VAD
Silence thresholds
Echo suppression and AEC
STT with Deepgram Groq Sarvam
Validation and hallucination filters
State machine orchestration
Groq LLaMA
Grok TTS
Playback

Features:

Real time Hindi and Hinglish
Sales and support agents
Silero VAD
Deepgram Nova 3
Barge in
Sentence streaming
Carrier suppression
Hallucination filtering
Hindi grammar optimization
FAISS RAG
Pipecat style orchestration

And yet users still say things like:

“The AI forgot what I said”

“It suddenly stopped responding”

“It heard me but replied weirdly”

But the logs tell a different story.

The LLM is usually fine.

What’s actually breaking:

Short Hindi acknowledgements like “haan”, “ji”, “hmm” vanish during suppression windows

Brand names fail under narrowband telephony

Code switching breaks segmentation

Tiny utterances never become canonical transcripts

Speech gets clipped before memory even forms

The assistant appears forgetful because the user’s intent literally never survives the pipeline.

And now I’m wondering something uncomfortable:

Did we over engineer reliability so hard that we destroyed conversational fidelity?

We stacked:

VAD
RMS gates
Endpointing
Carrier suppression
Echo suppression
Confidence filters
Hallucination filters
State transitions
Barge in logic

At what point does “production safety” become “signal destruction”?

Biggest bottleneck right now is TTS:

1.1 to 2.4 second latency

Premature response cutoffs

Long Hindi promotional speech degrades badly

I’m seriously considering prioritizing:

Faster uglier speech

Smaller interruptible chunks

Lower latency over voice beauty

Questions for engineers who’ve shipped real production voice systems:

Where do advanced telephony systems usually lose conversational fidelity?

VAD?

Endpointing?

Suppression windows?

State machines?

Confidence gates?

For Hindi and Hinglish specifically:

How are you handling:

Short acknowledgements

Code switching

Brand names

8kHz telecom degradation

Would you simplify this architecture?

Does this stack sound production grade or fundamentally fragile?

I’d genuinely love brutal criticism from:

Voice AI engineers

Telecom DSP people

Deepgram and Whisper builders

Contact center architects

Pipecat teams

Hindi ASR and TTS researchers

Because right now this feels less like an AI problem and more like trying to preserve human intent through a hostile audio pipeline.