We spent a year building voice AI. Every existing tool failed us. So we built our own and open-sourced it.
About 18 months ago we started what seemed like a simple project: help small medical clinics automate their front-desk calls.
Walk into any independent clinic and you'll find the same scene. One person answering phones, scheduling appointments, confirming insurance, handling refill requests — all at once. Phones go unanswered. Patients get put on hold. Callbacks never happen.
Small practices miss 40–60% of incoming calls during peak hours. Of those patients, roughly 80% don't call back.
We figured voice AI was ready. Let's build something real.
The "just use Vapi" phase
Spun up Vapi, wired in a phone number, wrote some prompts. Worked great in a demo.
Then we tried deploying it for an actual clinic.
No audit trail. A patient calls and says "the doctor said I could increase my dose." Three days later there's a complaint. What did the agent say, and why? With Vapi you get a transcript. That's it — no confidence scores, no branching logic, nothing you could show an attorney or compliance officer.
No reliable escalation. We wrote a prompt: "if the patient sounds distressed, transfer to a human." It worked maybe 70% of the time in testing. 70% is not acceptable when someone might be describing chest pain. The system had no structured escalation mechanism — just the LLM deciding in the moment whether something counted as urgent.
Behavior drift. Six weeks into our pilot, Vapi updated their underlying model. The agent started giving slightly different responses to the same questions. In a medical context, even minor behavioral changes can require re-approval from a compliance officer. You can't re-approve something if you didn't know it changed.
The two-tool trap. This is the big one. A call ends — patient wants to reschedule. Now you need to look up their record, find an open slot, book the appointment, send a confirmation. Vapi handles the conversation. It doesn't handle any of that.
The moment your agent needs to actually do something, you reach for a second tool. Most teams bolt on n8n or Make.com:
- Vapi for the voice layer
- n8n for the action layer
- A fragile webhook bridge holding it together
When that bridge breaks (and it does), debugging a missed appointment becomes a three-system investigation. We spent more time troubleshooting the glue than building the product.
Building from scratch
Fine. Let's own the full stack.
We evaluated Pipecat and LiveKit — both solid, both designed for developers who want to assemble pieces, not operators who want to ship in an afternoon.
- Basic call flow: 1 week
- Handling real-world complexity (rambling patients, mid-call topic changes): 3 more weeks
- Getting the compliance layer in (logging, escalation, audit trails): 6 more weeks
Three months in, one engineer had 1,200 lines of Python that almost nobody else could maintain. Every new clinic meant rebuilding the same architecture from scratch. The "platform" was a collection of clinic-specific scripts held together by hope.
What we actually needed
We sat down and described the system we actually wanted.
1 — One place for voice and workflow. Describe it in plain English: "Verify the patient, check their next appointment, offer to reschedule if it's more than 3 days out, send a confirmation." That should produce an agent that handles the conversation and queries your scheduling system, books the slot, and logs the outcome — no separate n8n flow bolted on, no webhooks to babysit. Everything Vapi + n8n does together, but built as a coherent whole.
2 — Safety in the architecture, not the prompt. Escalation as a first-class runtime concept. Guardrails evaluated on every turn. Audit trails as structural outputs, not optional logging.
3 — Stress-test before you deploy. An AI caller simulates hundreds of scenarios and flags failures before the agent goes live. You shouldn't find out about edge cases from real patients.
4 — Self-hostable. For anyone handling sensitive data, "your data goes to our cloud" is often a dealbreaker.
So we built it. We called it Feros.
Low-latency Rust voice runtime · Python control plane · AI workflow builder · built-in eval + guardrails · 19 prebuilt integrations · Apache 2.0
Voice-to-voice latency: ~340ms (below the 500ms threshold where humans start noticing the pause). The eval engine runs pre-deployment stress tests and produces a scored audit artifact you can hand to a compliance officer.
Why open-source
No compliance officer is going to deploy a black-box system for patient interactions without auditing the code. Open source isn't a giveaway here — it's the only distribution strategy that actually works in regulated verticals.
Also: the eval engine got meaningfully better after people started filing issues. The self-hosting path got cleaner because real users told us what broke.
Honest notes
A few things we got wrong:
- Voice AI is harder than it looks even with good tooling. The latency is solvable. Handling callers who give you three problems in one sentence — that takes real iteration.
- "Works" and "auditable" are different bars. Demos are easy. Edge cases only surface with real users.
- Small businesses have no IT capacity. "Self-hostable" means something very different when the person deploying it is a front desk manager.
Where it is now
Feros is in active use. Core runtime is stable. Eval engine works. Self-hosting is functional but still rough in places.
We started with clinics, but the architecture fits anywhere a voice agent needs to actually do things, not just talk:
- HVAC dispatchers routing emergency calls and updating job queues
- Real estate offices qualifying leads and scheduling showings into their CRM
- Law firms handling intake calls and creating case records
- Insurance agents confirming policy details with a full audit trail
Same pattern everywhere: voice + workflow + auditability, in one place instead of three.
GitHub: https://github.com/ferosai/feros Website: https://feros.ai
Happy to answer questions about what worked, what didn't, or what you're building.
Posted May 8, 2026, 3:44 PM UTC