Built Dolly: a per-employee LLM agent that handles workplace messaging on behalf of each individual — architecture discussion

Wanted to share a project and some of the interesting architecture decisions we had to make — curious what this community thinks.

**The problem:** employees spend ~3 hours/day on async messages. The vast majority are patterned responses that don't need the person's full attention. We wanted to automate those.

**What we built:** Dolly — a per-employee AI agent. Not a shared org bot. One agent per person, each with:

- Fine-tuning on that employee's communication history (tone, style, recurring answers)

- RAG layer over their personal knowledge base (docs, past replies, internal wikis)

- LangChain orchestration for tool routing across email and Slack APIs

- A confidence scoring system that determines whether to auto-respond or surface a draft

**Some decisions worth discussing:**

1. **Fine-tune vs. prompt-engineer the persona**: We initially tried heavy system prompting for persona. It worked okay but degraded on edge cases. Per-user fine-tuning produced much more consistent voice fidelity, at the cost of more infra complexity.

2. **Confidence gating**: We use a combination of semantic similarity to past responses + LLM self-assessment to determine confidence. Still not perfect — curious if anyone has better approaches.

3. **RAG scope per employee**: How much context is too much? We found that scoping RAG to the last 90 days of their communications + their active docs gave the best precision/recall tradeoff.

We're in early rollout — 20 orgs, 17 spots left.

https://getdolly.ai

Happy to go deep on any part of the stack.