Used GPT-4 to build an AI that responds to messages on behalf of employees — here's what we learned

Full disclosure: I'm one of the founders of Dolly (https://getdolly.ai). Sharing what we actually built and learned.

The premise: the average employee spends ~3 hours a day reading and replying to messages. Most of those replies follow predictable patterns they've already established. So we asked: what if an AI could handle that load on their behalf?

That's Dolly. One AI agent per employee — not a shared team bot. Each person's Dolly is trained on their own communication style and has access to their personal knowledge base.

How GPT-4 fits in:

We use GPT-4 for the confidence scoring layer — the part that determines whether to auto-send a response or surface a draft for human review. This turned out to be the hardest part of the whole system.

We tried smaller models first. They weren't reliable enough on edge cases — specifically, they'd sometimes miss nuance that would make an auto-sent reply feel off or wrong. GPT-4's reasoning ability on ambiguous inputs was the deciding factor.

The persona layer (learning each employee's voice and tone) uses a fine-tuned smaller model running locally per-employee for privacy reasons.

Biggest lessons from v1:

- Users trust the auto-send feature far less than expected early on. It takes about 2-3 weeks of supervised use before they're comfortable letting Dolly send without review.

- The RAG scope matters enormously. 90 days of recent comms + active docs outperformed larger context windows for precision.

- Context window management is non-trivial when you have 6 months of Slack history to work with.

Limited rollout right now — 20 orgs, 17 spots left.

https://getdolly.ai

Happy to go deep on any of the technical choices.