A/B Test: Claude Code Agents with Knowledge Graph vs. Grep for Codebase Tasks

Workflow value: 85/100
Status: active · Freshness: 70/100 · Confidence: 0.90 · Level: advanced
Categories: Quality Control, Token Saving, Context & Memory, Debugging, Shipping, Hooks, MCP, Subagents, Multi-Agent
Original source: r/ClaudeAI post/comment

What problem this solves

Evaluating the effectiveness and efficiency of knowledge graph tools versus traditional text search (grep) for Claude Code agents performing code modification and analysis tasks in large codebases. It helps users decide when and how to integrate such tools into their agent workflows.

Summary

This post details a controlled A/B test comparing Claude Code subagents using a knowledge graph tool ('codebase-memory-mcp') against agents using standard text search (grep/glob/read) on a production TypeScript monorepo. The experiment evaluates performance for code modification and blast-radius analysis tasks, providing concrete metrics (recall, wall time, tokens) and nuanced insights into when each approach is more effective. The findings suggest that while both achieve high accuracy, the knowledge graph can be slower for editing tasks and may require agents to double-check its findings with traditional search, making it more valuable for direct human queries than for agent-driven code changes.

Why it is useful

This workflow is highly valuable because it provides a concrete, data-driven A/B test comparing two fundamental approaches for Claude Code agents interacting with codebases: traditional text search (grep) and a knowledge graph tool. It offers nuanced, evidence-based insights into their respective strengths and weaknesses for different types of tasks (code editing vs. discovery). The methodology for setting up such a controlled experiment is itself a reusable workflow for evaluating AI agent performance. It helps users make informed decisions about tool selection and agent strategy, potentially saving time and resources by avoiding over-reliance on hyped solutions where simpler tools suffice, and understanding the limitations of advanced tools.

Workflow

1. Define a specific code modification or analysis task for the AI agent.
2. Set up two identical Claude Code subagents, each in an isolated git worktree, using the same model and specification.
3. Configure one agent to use standard text search tools (grep/glob/read) for codebase interaction.
4. Configure the second agent to use a knowledge graph tool (e.g., codebase-memory-mcp) as its primary discovery mechanism.
5. Manually establish ground truth for the task, identifying all required changes and 'trap' sites (locations that appear relevant but are not), to enable objective scoring of recall and precision.
6. Run both agents on the defined task.
7. Record key metrics for each agent: recall, number of unneeded edits, wall time, and token usage.
8. Compare the results to determine the relative effectiveness and efficiency of the knowledge graph approach versus traditional text search for the given task type.
9. Analyze 'interesting bits' such as unexpected behaviors, limitations of the tools, or agent strategies for verification.

Tools / artifacts

• Claude Code subagents
• Git worktrees
• Grep/glob/read (standard text search tools)
• Codebase-memory-mcp (knowledge graph tool)
• Production TypeScript monorepo (or similar large codebase)
• Ground truth (manually identified correct changes/analysis points)

Validation signals

• Controlled test on a real production TypeScript monorepo.
• Ground truth built by hand for objective scoring (recall and precision).
• Identical Claude Code subagents, isolated git worktrees, same model and spec.
• Quantitative metrics collected: Recall, Unneeded edits, Wall time, Tokens.
• Functional equivalence of diffs noted.
• Both agents independently found the same off-spec gotcha.
• Four runs, zero mistakes on either side (high accuracy observed).

Limitations

• The experiment was conducted with n=1 per condition on a single codebase, as acknowledged by the author. More diverse codebases and runs would strengthen the generalizability of the conclusions.
• The specific knowledge graph tool ('codebase-memory-mcp') may evolve, potentially altering its performance characteristics over time, though the general comparison remains relevant.
• The post is an evaluation rather than a direct 'how-to' guide for a specific coding task, requiring users to interpret the findings for their own workflow design.

Rate this workflow

Upvote this post if the workflow is useful, reproducible, or worth recommending.

Downvote if it is vague, outdated, unsafe, overhyped, or not reproducible.

Reply if it worked for you, failed, is outdated, or has a better alternative.

---

This post was generated automatically from the workflow library database.