GraphRAG Benchmarks: What They Really Say vs Vector RAG

Everyone wants the same thing: a single number. “GraphRAG is 47% better than vector RAG” — something to put in a slide and end the argument. The most useful finding from a year of GraphRAG benchmarks is that this number does not exist, and anyone who quotes you one is selling something. When you decompose by question type, a real and repeatable pattern emerges — graphs pull ahead on multi-hop and synthesis, vectors hold their own (or win on cost) for simple lookups — but the headline “win rates” that circulate in blog posts are softer than they look, and some evaporate entirely once you check who did the judging.

A note on how to read this piece: we didn’t run these benchmarks. Every figure below is vendor- or academically-published, on someone else’s corpus and setup. That’s exactly why the useful skill here isn’t memorizing a multiplier — it’s knowing how to read a retrieval benchmark critically. This is the Measure spoke of our knowledge-graph hub; the conceptual companion is GraphRAG vs Vector RAG: Knowledge Graphs for AI, Explained, and the vector-side mirror is Postgres as a Vector Database: What the Benchmarks Say.

GraphRAG benchmarks — a scoreboard where graph, vector, and hybrid each win different question types

📊 There Is No Scalar “Better” — It’s Task-Dependent

The single most important result across the 2026 literature is that “which is better” is a category error until you say at what. The clearest demonstration is GraphRAG-Bench (an ICLR’26 effort), which scores retrieval across four task tiers — fact retrieval, complex reasoning, contextual summarization, and creative generation. The pattern is consistent: the graph advantage grows with reasoning complexity and shrinks toward zero on simple fact retrieval. A systematic RAG-vs-GraphRAG evaluation across HotpotQA, NovelQA, and QMSum found the same shape — GraphRAG ahead on multi-hop synthesis and cross-section reasoning, behind on simple single-document factual queries — with graph quality (how well entities were extracted) directly gating the results.

So the honest scorecard isn’t one row. It’s a table where the winner changes by question shape:

Multi-hop / relational / whole-corpus synthesis → graph or hybrid wins, sometimes decisively.
Simple single-hop fact lookup → vector RAG ties or wins, at a fraction of the indexing cost.
Everything in between → it depends, which is why you benchmark your own workload.

If your queries are all “find the doc that mentions X,” a graph will cost you more and win you nothing. If they’re “how does X connect to Y three steps away,” vectors structurally cannot see the answer.

A GraphRAG vs vector RAG scorecard by task type: graph wins multi-hop, vector wins simple lookups, hybrid often wins overall

🔬 What the Peer-Reviewed Wins Actually Measured

The flashiest GraphRAG numbers come from Microsoft’s From Local to Global paper: roughly 72–83% comprehensiveness and 62–82% diversity win rates over vector RAG. Real result — but read the fine print. Those are LLM-judged pairwise preferences on global sensemaking questions (“what are the main themes across this whole corpus?”), not factual accuracy on retrieval. Comprehensiveness and diversity are qualities of a good summary, and summarization over a whole dataset is exactly the task vector RAG can’t do because no single chunk contains a corpus-wide theme. It’s a genuine win on the question class graphs are built for — not evidence that graphs answer every question better.

When you move to a real domain corpus and measure factual correctness, the gap narrows sharply. A 2026 benchmark on O-RAN telecom docs reported Faithfulness of 0.59 for GraphRAG and Hybrid vs 0.55 for Vector — and Factual Correctness of 0.58 (Hybrid) > 0.50 (GraphRAG) > 0.48 (Vector), with Answer Relevance essentially tied around 0.72–0.74 across all three. Two things fall out of that: hybrid took the top factual score (about 8% over vector), and pure GraphRAG’s edge over vector on this task was real but modest, while it produced more concise, lower-latency answers. That’s a much more sober picture than “72–83%,” and it’s the one to keep in mind: the big win rates are task-specific, and hybrid is the frequent overall winner.

⚖️ The Judge Is Part of the Result

Here’s the uncomfortable finding that should change how you read every GraphRAG number. Most of those pairwise “win rates” are produced by an LLM-as-judge — you ask a model which of two answers is better. A 2026 audit of that methodology found it carries serious, measurable biases: position bias (swapping which answer is shown first can swing the win rate by more than 30 points), length bias (a ~25-token difference on a ~200-token answer can move it more than 50 points), and trial bias (identical comparisons flip across runs).

The punchline is brutal: after controlling for those biases, one system’s reported 66.7% win rate over naïve RAG corrected to 39.06% — i.e., from “clearly better” to below even. That doesn’t mean GraphRAG is fake; it means a win rate reported without a bias-control note is not evidence. If you take one habit from this article: when you see a GraphRAG “X% win,” ask whether the judge’s order and answer length were controlled. If the paper doesn’t say, discount the number.

And two figures you’ll see everywhere deserve a specific callout: the widely-quoted “GraphRAG 86% vs vector 32% on multi-hop” and “0% vs 90% on aggregation” numbers trace to blog posts with no primary-source attribution — one author openly concedes they’re cherry-picked worst cases. Use them only as “some analyses illustrate,” never as benchmark results.

LLM-as-judge bias: a GraphRAG win rate swinging dramatically when answer order and length are controlled

💸 The Cost Lands Upfront (the Part the Decks Skip)

Accuracy is only half a benchmark; the other half is what it cost to get there, and graphs move that cost to a place vector RAG doesn’t. Building a knowledge graph means an LLM extraction pass over your whole corpus before you answer a single question. Using public benchmark ingestion data, that’s roughly 17,600 tokens per document to index (about half a cent per doc on a small model, ~$5–6 per 1,000 docs), and a million-token corpus can run $20–50 in API fees plus hours of compute — Microsoft’s default pipeline makes ~4–6 LLM calls per 512-token chunk. In early 2024 one widely-cited figure put a single dataset’s indexing at ~$33,000; the category’s own leader answered that with LazyGraphRAG, which defers the expensive summarization to query time and brought indexing cost down to roughly vector-RAG levels.

The payoff runs the other way: once built, querying a compact graph is cheap. Graphify reports a 71.5× token reduction from querying its graph instead of re-reading raw files — but that’s a single 52-file-corpus, vendor-reported figure; the direction (query-time savings amortize the build cost) is credible, the exact multiplier is not portable to your repo. The honest model is a break-even curve: graphs win once build_cost < queries × per-query_savings. For a corpus you’ll query a handful of times, vector RAG is simply cheaper. For one you’ll hammer with millions of structured queries, the graph pays for itself. Never price a graph as a free accuracy upgrade.

The knowledge-graph break-even: high upfront build cost vs vector RAG's per-query cost, crossing at high query volume

🧪 What You Should Actually Measure

Because there’s no portable number, the real deliverable is a method you can run on your task. A defensible GraphRAG evaluation:

Tier your eval set by question type — single-hop fact, multi-hop reasoning, aggregation/structural, global summary — and score each bucket separately. A blended average hides the exact signal you’re paying the graph for.
Benchmark all three: graph, vector, and hybrid. The literature keeps finding hybrid wins factual correctness, so leaving it out biases your own test.
Debias your judge. If you use LLM-as-judge, randomize answer order, control for length, and run multiple trials. Otherwise your win rate is noise dressed as signal.
Measure the metrics vectors can’t express. Percentage of answers backed by extracted (ground-truth) vs inferred evidence, entity-resolution F1 (retrieval precision can’t exceed extraction quality — one study found 89.2% F1 with good chunking vs 74.4% with naïve chunking), and stale-edge rate after updates — a graph that’s a few percent stale silently poisons multi-hop answers.
Track cost per successful task and time-to-first-correct-answer, not raw latency. A cheaper query that’s wrong is not cheaper.

Do that and you’ll have something no vendor benchmark can give you: the answer for your corpus, your query mix, and your budget.

❓ Frequently Asked Questions

Is GraphRAG actually more accurate than vector RAG? On multi-hop, relational, and whole-corpus-synthesis questions, yes — repeatably. On simple fact lookups, it ties or loses, at higher cost. There is no single accuracy number; it’s task-dependent, and hybrid frequently beats both.

Where do the “72–83% win rate” figures come from? Microsoft’s From Local to Global paper — but those measure comprehensiveness and diversity of summaries on global sensemaking questions, judged by an LLM, not factual accuracy on retrieval. They’re a real win on the task graphs are built for, not proof graphs win everything.

Why should I distrust GraphRAG win rates? Because many are produced by LLM-as-judge, which carries position, length, and trial biases large enough to swing a result by tens of points. In one audited case a 66.7% win rate corrected to 39.06% after debiasing. Trust numbers that report bias controls; discount the ones that don’t.

Isn’t building a knowledge graph really expensive? The indexing (LLM extraction) is a real upfront cost — roughly $5–6 per 1,000 docs, more at scale — though LazyGraphRAG cut it toward vector-RAG levels. Query-time is cheap once built. It’s a break-even: worth it at high query volume or when you need structure and provenance, not for a corpus you’ll touch a few times.

What single metric should I track? There isn’t one — track a small panel: per-tier accuracy, extracted-vs-inferred evidence share, entity-resolution F1, stale-edge rate, and cost per successful task. The panel is the point; any single number is gameable.

📚 Further Reading

GraphRAG vs Vector RAG: Knowledge Graphs for AI, Explained — the conceptual cornerstone this benchmark piece measures.
Postgres as a Vector Database: What the Benchmarks Say — the vector-side mirror, on reading pgvector benchmarks with the same skepticism.
What Is an Embedding? — the foundation for the “vector” half of every comparison here.
Primary sources: Microsoft From Local to Global, GraphRAG-Bench (ICLR’26), the O-RAN hybrid-RAG benchmark (arXiv:2507.03608), and the LLM-as-judge debiasing study (arXiv:2506.06331). The numbers are each paper’s own, on their own setups.

📚 Recommended Resources

Disclosure: some links below are Amazon affiliate links. If you buy through them, I earn a small commission — at no extra cost to you — which helps keep this site running.