[P] Forked PersonaPlex to route domain queries to DeepSeek via TTS injection — detailed write-up on what worked and what didn't

We forked NVIDIA's PersonaPlex to experiment with augmenting full-duplex speech models with external knowledge. The use case: a voice assistant that handles conversation naturally (PersonaPlex) but routes domain-specific questions to DeepSeek for accurate answers.

What worked: TTS injection via forced text-token generation through the depformer produces natural speech in the model's established voice. The binary protocol extension (new 0x07 message type) integrates cleanly. The browser audio pipeline (Opus capture, AudioWorklet jitter buffering) achieves acceptable latency.

What didn't work: the 7B Helium backbone cannot reliably follow system prompt instructions to signal when it should defer. This isn't a prompt engineering problem — the model was trained for conversational dynamics, not instruction following. We tried explicit markers (!!!) and natural phrase detection ("let me check"), both unreliable.

The deeper finding: even with perfect detection, full-duplex models generate continuously at 12.5 Hz. There's no natural pause point to consult an external system. Fine-tuning could improve detection but doesn't solve the timing problem. The real solution likely requires architectural changes — a routing head that runs ahead of audio generation, or a learned hold behavior.

Full write-ups with architecture details, code, and analysis of open directions: https://github.com/dosht/personaplex

Medium article version: https://medium.com/@mou.abdelhamid/smart-routing-for-full-duplex-speech-models-augmenting-personaplex-with-external-llm-knowledge-09abaccd1d70