A/B test of codebase-memory-mcp against plain grep on a real production codebase

Saw codebase-memory-mcp doing the rounds (the "indexes your repo into a knowledge graph, 158 languages, sub-ms queries" one) and wanted actual numbers instead of "it saves 99% tokens" because I was skeptical about the claims, so I ran a controlled test on our production TypeScript monorepo.

Setup

• Two identical Claude Code subagents per task, each in an isolated git worktree, same model (Fable), same spec.
• One agent restricted to standard tools (grep/glob/read). One told to use the knowledge graph as its primary discovery tool.
• Before launching anything, I built ground truth by hand (every file/line that must change, plus "trap" sites that look like they need changes but don't) so I could score recall and precision instead of eyeballing.
• Indexing the whole monorepo took 1.5 seconds (~9k nodes, ~21k edges), so index cost is basically irrelevant.

Task 1: small cross-stack feature (thread a new optional field from 5 frontend call sites through the API contract into a lib function — 8 required edit sites, 3 traps)

The diffs were functionally identical — every meaningful line the same, only comment wording differed. The graph added zero accuracy and cost 40% more time.

Task 2: designed to favor the graph — blast-radius analysis. "Add an env kill switch to our Claude LLM wrapper AND produce a complete inventory of every endpoint/webhook/script that can transitively reach it." The caller tree is ~25 symbols deep across 4 hops, including one path that goes script → migration fn → comparison fn → test bench → LLM wrapper.

Both agents also independently found the same off-spec gotcha (two functions bypass the wrapper and call the SDK directly, so the kill switch doesn't cover all spend). Both perfect again.

The interesting bits

1. Both approaches got everything right, every time. Four runs, zero mistakes on either side. Claude with plain old text search already finds everything it needs in our codebase — so the fancy graph had no mistakes to fix. It was solving a problem that didn't exist.

2. The graph's map had a hole in it. It was missing one connection — a function that directly calls our AI wrapper just wasn't linked up. The agent using the graph noticed something was off and double-checked everything with normal search anyway. So instead of being "the answer," the graph became "a rough map you still have to verify yourself." That still saved a bit of time, but it means you can't rely on the graph alone when missing something would really hurt.

3. It's great when you ask it questions, less great when an AI agent uses it to write code. I asked "if I change this function, what's everything in the app that could be affected?" and it answered in one shot — tracing from buttons in the frontend, across the network boundary, all the way down to the backend function. It even caught one connection that normal search misses, because the code there uses a different name. Doing that by hand means searching, reading, searching again, over and over. But when an agent is building something, it ends up re-checking the graph's answers with normal search anyway, so half the benefit evaporates.

4. Whether it helps depends entirely on the type of task. If the job is mostly editing code, the graph makes things slower — typing out the changes costs the same either way, so faster searching barely matters. If the job is mostly finding things, the graph wins — but by 13%, not the "game changer" the hype suggests.

Caveats: n=1 per condition, one codebase.