Why do GraphRAGs perform worser than standard vector-based RAGs?

I recently came across a study (RAG vs. GraphRAG: A Systematic Evaluation and Key Insights) comparing retrieval quality between standard vector-based RAG and GraphRAG. You'd expect GraphRAG to win, right? Graphs capture relationships. Relationships are context. More context should mean better answers.

Except… that's not what they found. In several tests, GraphRAG actually degraded retrieval quality compared to plain old vector search.

Because I've also seen production systems where knowledge graphs and graph neural networks massively improve retrieval. We're talking significant gains in precision and recall, with measurably fewer hallucinations.

So which is it? Do graphs help or not?
The answer, I think, reveals something important about how we build AI systems. And it comes down to a fundamental confusion between two very different mindsets.

Here's my thought on this: GraphRAG, as it's commonly implemented, is a developer's solution to a machine learning problem. And that mismatch explains everything.

In software engineering, success is about implementing functionality correctly. You take requirements, you write code, you write tests that verify the code does what the requirements say. If the tests pass, you ship. The goal is a direct, errorless projection from requirements to working software.
And that's great! That's how you build reliable systems. But it assumes the problem is well-specified. Input A should produce output B. If it does, you're done.

Machine learning doesn't work that way. In ML, you start with a hypothesis. "I think this model architecture will predict customer churn better than the baseline". Then you define a measurement framework, evaluation sets, and targets. You run experiments, look at the number, iterate and improve.
Success isn't binary. It's probabilistic. And the work is never really "done". It's "good enough for now, and here's how we'll make it better".

So what does a typical GraphRAG implementation actually look like?
You take your documents. You chunk them. You push each chunk through an LLM with a prompt like "extract entities and relationships from this text". The LLM spits out some triples: subject, predicate, object. You store those triples in a graph database. Done. Feature shipped.

Notice what's missing. There's no evaluation of extraction quality. Did the LLM actually extract the right entities? Did it hallucinate relationships that aren't in the source? Nobody checked.
There's no entity resolution. If one document mentions "Hilton Hotels" and another mentions "Hilton Worldwide Holdings," are those the same entity? The system doesn't know. It just created two nodes.
There's no schema alignment. One triple might say "located_in" while another says "headquartered_at" for semantically identical relationships. Now your graph is inconsistent.
And critically, there's no measurement framework. No precision metric. No recall metric. No target to hit. No iteration loop to improve.

You've shipped a feature. But you haven't solved the ML problem.