We over-engineered our LLM data-agent into uselessness — here's what we learned (and where we're stuck)

We built a system where users ask natural-language questions about Excel data (prices, quantities, rankings, etc.) and an LLM answers them.

**V1** was basically an LLM with SQL access in a notebook: it wrote a query, ran it, looked at the result, and decided whether to try again or answer. Simple ReAct-style loop. It handled genuinely complex, multi-step questions surprisingly well.

**V2** "productionized" this by decomposing the problem into a dozen-plus hardcoded "capabilities" (row_count, column_list, filtered_ranking, etc.), each with its own input parsing, retry chain, and escalation rules — plus several stacked review/verification loops meant to catch hallucinations.

Result: it got *worse* at complex questions, not better. Every time a specific query broke, we patched it with a narrow special-case rule for that one capability. Over months, the system became a pile of reactive patches that don't talk to each other — one module doesn't know what another already confirmed, so it re-asks the user things they already answered. Worse, some of these deterministic modules can silently compute a *wrong* answer (e.g. return the highest price when asked for the lowest) and none of the guardrails catch it, because those guardrails were built to catch fabrication, not logic errors.

The core lesson: the failure wasn't "the LLM is bad at this," it was that **we removed the LLM's feedback loop**. Once you pre-decide the decision tree instead of letting the model reason step-by-step with tools and self-check before answering, complexity stops scaling — it just compounds.

We're now redesigning around a small set of general tools (schema lookup, query, calculate, ask-user) plus a mandatory self-verification step before the final answer, instead of dozens of narrow pre-built "capabilities."

**Where we could use input:**

- Anyone gone through this same "decompose into capabilities → regret it → go back to a general loop" cycle? What did the rebuilt version look like?

- For the query tool: structured params (safer, easier to verify) vs letting the model write its own read-only SQL (more flexible, harder to verify) — did you pick one, or run both and route between them?

- How are people catching *logic* errors (right data, wrong computation/direction) rather than just fabrication? Our guardrails only ever checked "is this grounded in real data," never "is this actually correct."

Would appreciate hearing how others have approached this, especially if you've shipped something past the "cool notebook demo" stage.