Retrieval, not the model.

We asked one hard question of a fixed body of source text, and answered it ten different ways — the AITruthX framework, and three flavours of Microsoft GraphRAG, each driven by three different large language models. Every answer below is a real, live output from the actual engine. Then a neutral judge counted, for each one, exactly how many claims were not supported by the source. This note reports what happened and what it means for anyone who needs data they can trust.

What AITruthX is — the framework searches, the model only speaks

AITruthX uses a language model for one job only: putting what the framework already found into clear, correct language — the wording and the grammar. The model does not search, and it does not decide what the answer is. That work is done by the framework itself, over approved source material. Because the model is only writing, it can be a cheap, open-source-class base model instead of an expensive frontier one.

That difference is the whole subject of this study, and it is why AITruthX can run on the cheapest model in the comparison and still be the only engine to reach perfect grounding: correctness comes from the retrieval, not from the model's judgement.

What's next for the framework

We're building AITruthX into a platform where anyone can upload their own data and get grounded, LLM-quality answers from it — at the cost of the cheapest model, with output quality rivaling the most expensive.

The rest of this note is the evidence behind that claim — the experiment, the numbers, and every answer and flagged claim in full.

01What Test 2 adds — the rest of the RAG world

Reviewers of Test 1 rightly asked: Microsoft GraphRAG is one family — what about the standard fixes? Two in particular are the textbook rebuttals: cross-encoder reranking (precise top-k instead of noisy stuffing) and parent-document retrieval (resolve to a unit, return its bounded neighbourhood — structurally the closest rival to MDF). Test 2 runs both, plus every other major family:

02A new metric: coverage — so “clean by saying less” can't hide

A fair criticism of grounding alone: a method could score 1.00 by returning one narrow sentence. Test 2 therefore adds coverage — each answer is scored against 35 frozen source facts (fixed before any run): how many does the answer actually convey? A clean-but-empty answer now shows up as exactly that. As the table shows, AITruthX's 1.00 grounding comes at 0.90 coverage — it says as much of the real material as the stuffing methods do.

How to read the table — what each number means

Grounding is the fraction of an answer's claims actually supported by the source (1.00 = nothing invented). Flags is the raw count of claims a judge marked unsupported — lenient counts only clear hallucinations, strict counts every claim not explicitly backed, however small. Coverage is completeness: how many of the 35 frozen source facts the answer conveys. Calls is the number of LLM calls the method needed. A single number means the judges agreed; a range is the min–max across judges, modes, and runs.

Read the coverage column first: almost everyone covers 0.8–0.94 of the source facts, including AITruthX at 0.90 — so no row is winning by saying less. Then read grounding: the only 1.00 in either test is still the first row. The best chunk-based methods (RAPTOR, HyDE, the agent, hybrid) get close — 0.90-something — but close means several invented claims per answer, every time, in every run.

03Premium models on the same retrievals — the Test 1 result, repeated harder

We re-generated the four decisive configurations with Claude Opus 4.8 and Claude Fable 5 — the exact same retrieved context, only the writer changes. If hallucination were a model-quality problem, the expensive models should clean it up.

04Where the hallucinations actually come from — a claim-by-claim audit

Counting flags says how much; it doesn't say why. So we audited every flagged claim from every Test 2 configuration against the exact context its model was given, and classified each one: was it actually in the retrieved text? Stitched across chunk borders? Or invented outright — no basis in the context at all?

05Findings of Test 2

iNo retrieval family produced a clean answer

Fourteen configurations, seven families, two full runs each — and not one clean answer anywhere. The best (RAPTOR, HyDE, the LangGraph agent) reached 0.92–0.94, which sounds close until you read it as what it is: several invented claims in every answer, every time. For a system whose product is trust, "close" is the failing grade.

iiBounding retrieval does not bound generation

The textbook rebuttal to Test 1 was that a top-1 reranker is "as bounded as MDF." It isn't: given one 395-character chunk, the model wrote 1,000+-token answers — ~70 claims manufactured around 4 supported ones. Boundedness of input says nothing about boundedness of output; only completeness of the retrieved unit does that.

iiiLoops and trees pay more for the same problem

The agentic loop spent 6 LLM calls and 64 seconds (vs 1 call, ~8s) and mostly re-retrieved the same chunks. RAPTOR spent 5 extra summarisation calls building its tree. Both still flagged 12–29 claims under strict judging. Extra machinery multiplied cost, not integrity.

06What we actually tested

All ten runs read from the same source corpus (an encyclopedia-level bucket about early nuclear-weapon design, used purely as a fixed test document). All ten answered the same broad question:

“Explain the complete Teller–Ulam thermonuclear design step by step, from the fission primary to the final total yield, including every stage and mechanism involved.”

We chose a deliberately big concept — a whole design, not a single fact — because breadth is exactly where retrieval methods either hold together or start inventing. What differs between the ten runs is only two things: how the source is retrieved, and which model writes the final answer.

07The four retrieval methods, in plain terms

Three of these are Microsoft GraphRAG's real search modes — Microsoft's open-source (MIT-licensed) graph-RAG library, which is exactly why we could run the genuine pipelines rather than a re-implementation; the fourth is the AITruthX framework. Same corpus, same judge — only the retrieval logic changes.

08How the experiment ran — the full process

Nothing here is a paraphrase or a remembered answer. For each cell in the results table we executed the real engine prompts against the real database, captured the model's actual output, and measured tokens and calls directly. Here is the exact pipeline and where each credential came from.

Which keys, which plans, ran what

• Google Gemini (base plan API key). Ran the AITruthX base model, the Gemini-driven Microsoft GraphRAG and flat answers, all text embeddings (gemini-embedding-001), and the neutral judge — model gemini-2.5-flash.
• Claude Opus 4.8 & Claude Fable 5 (Claude Max plan, via Claude Code CLI). After a direct Anthropic-API purchase failed, we authenticated the Claude Code command-line tool on the Max plan and ran each Claude model as a fresh, headless process with zero knowledge of this project — fed byte-identical engine prompts. This is what makes the Claude columns real rather than assistant guesses.
• Same prompts, same data, same judge across every model. The only variable inside a Microsoft GraphRAG row is the model; the retrieval context is held fixed.

09Results

Ten real runs, one broad question. Grounding is the fraction of the answer's claims that the source actually supports; flagged claims is the raw count of claims it does not; depth is how completely the answer walks the full design (0 = one-liner, 1 = every stage).

Read the table top-down and a pattern jumps out. The only perfectly-grounded row is AITruthX on a cheap model; flat stuffing on Fable 5 comes next at 0.94 — but AITruthX gets there at roughly a quarter of the tokens. Everything in the Microsoft GraphRAG block sits near the bottom regardless of which model wrote it: swapping Gemini for Opus for Fable barely moves grounding, because the context those methods retrieved was thin or tangential to begin with.

Isolating the model: the flat-stuffing rows — and why the count is noisy

The flat rows saw the same rich context, so they should isolate pure model quality. When we tried to rank the three models by flag count, we found we couldn't — the number moves too much between judge passes to trust. That instability is itself the finding:

10What “accuracy” means here — and an honest note

In our earlier in-product comparison screen, the single “accuracy %” figure was a blended display metric: accuracy = grounding × 0.6 + relevance × 0.4. It is convenient for a dashboard, but it mixes two different things. In this note we report the components separately and treat grounding — the fraction of claims the source supports — as the real integrity number, because that is what “does it hallucinate?” actually measures. Relevance and depth are reported alongside, never folded in silently.

11Three findings

iRetrieval is the bottleneck, not the model

This is the headline. Microsoft GraphRAG local and global stayed near the floor whether Gemini, Opus, or Fable wrote the answer. The framework that retrieved the right passage — AITruthX — reached top grounding on the cheapest model available. You cannot buy your way out of bad retrieval with a bigger model; you fix retrieval.

iiBigger concepts break stuffing first

The broad “explain the whole design” question is precisely where flat stuffing bled: flat-Opus went from a handful of unsupported claims on a narrow question to 19 on this one. The wider the ask, the more a permissive method fills gaps with plausible-sounding invention. AITruthX held at zero because it never left the bounded entry.

iiiA stronger model does not mean “reasoning wins” — answering your question directly

You asked whether the results show Gemini's base model is best for reasoning, even against Claude's biggest models. They do not — and the honest answer is stronger than a leaderboard. When we tried to rank the three big models by how many unsupported claims they made, the number would not hold still: repeated strict passes put flat-Fable anywhere from 4 to 27, and the ranking among the models flipped between passes. The reason is that the count is confounded by answer length — the more thorough the answer, the more specific details a strict pass can mark as “outside the exact reference,” even when those details are real and came from the retrieved source. So no model here is cleanly “best for reasoning” on this evidence; that framing doesn't survive contact with the data. What does hold across every pass is two things: all the big models keep grounding high (0.88–0.96), and only AITruthX reaches a perfect 1.00 — on the cheapest model, because its answer is bounded to exactly what the source supports, so there is almost nothing for any judge to flag. The grounding comes from retrieval structure, not from hoping a bigger model behaves. That stability is the product.

For the record, the Gemini model used throughout was gemini-2.5-flash for all answers and judging, with gemini-embedding-001 for embeddings.

12One more thing we saw: the refusal

On this broad, weapons-adjacent question, the real Claude Opus 4.8 global run partially declined to reconstruct the full staging — a safety response. AITruthX answered fully, because its context is a bounded, pre-approved slice of encyclopedia-level source text rather than an open-ended “reconstruct the weapon” request. Controlled retrieval doesn't only reduce hallucination; it keeps legitimate answers deliverable.

13Why this becomes a product: grounded training data

Here is the commercial point the experiment sets up. Teams building their own models increasingly harvest answers from top LLMs to assemble training data for a target question — but that harvested data can hallucinate, and providers increasingly throttle or refuse bulk, repetitive querying (often suspected to be model-training use). AITruthX offers the alternative: training pairs that are grounded in the customer's own corpus by construction, so they don't invent.

Each pair is delivered in two modes:

Prompts come either from a short list the customer supplies, or are generated internally per MDF entry. The customer submits their data through the ordinary ingestion flow and sees it land correctly; the training-data job itself is run from the admin side and only the finished, downloadable data file is returned to their profile. It is priced as a data product, separate from ordinary usage cost.

14How the judging was done — two judges, two strictness levels

Because a single count can mislead, we did not judge once. We scored the same four full-depth answers four ways: with Google Gemini 2.5 Flash and, independently, with Claude Fable 5 — and each judge in two modes. A normal pass counts only clear hallucinations (statements that plainly go beyond the source, ignoring reasonable rephrasing). A strict pass enumerates every claim, however small, that isn't word-for-word in the reference. The earlier figures in this note began with the normal, in-depth reading; this section adds the strict reading and a second, independent judge on top.

Running two separate model families as judges is the safeguard: if the conclusion only held under one judge, it wouldn't be trustworthy. Here is every count — and the one row that stays near the floor no matter who judges or how strictly.

AAppendix — every answer, every judge, every flagged claim

Complete transparency behind Figure 6. For each of the four answers: first the exact text the engine returned, then — listed separately — every claim flagged by each of the four judge passes (Gemini normal, Gemini strict, Claude Fable 5 normal, Claude Fable 5 strict), each with the reason it was flagged. Read these and you can see for yourself why the counts move so much: nearly every flagged item is a real detail drawn from the retrieved source that simply falls outside the narrow structured reference — not an invention. That is why longer answers accrue more flags, and why grounding (a fraction) is the metric to trust over any single raw count.