A measured, all-real comparison of the AITruthX framework against Microsoft GraphRAG — run across Google Gemini, Claude Opus 4.8, and Claude Fable 5 — showing where hallucinations actually come from.
We asked one hard question of a fixed body of source text, and answered it ten different ways — the AITruthX framework, and three flavours of Microsoft GraphRAG, each driven by three different large language models. Every answer below is a real, live output from the actual engine. Then a neutral judge counted, for each one, exactly how many claims were not supported by the source. This note reports what happened and what it means for anyone who needs data they can trust.
The bottleneck is retrieval, not the model. AITruthX — running the cheapest model — answered with zero hallucinations at a fraction of the cost. Upgrading the language model (all the way to Claude Fable 5) did not rescue the weaker retrieval methods. And when we gave every method the same rich context, the strongest model was the most disciplined, not the most inventive.
AITruthX uses a language model for one job only: putting what the framework already found into clear, correct language — the wording and the grammar. The model does not search, and it does not decide what the answer is. That work is done by the framework itself, over approved source material. Because the model is only writing, it can be a cheap, open-source-class base model instead of an expensive frontier one.
A cheap, open-source base model turns the passage the framework retrieved into fluent language. It never searches, never reasons about what is true, and never chooses the answer — so it has no opening to invent one.
The language model is asked to search, reason over whatever it pulled back, judge what matters, and write. Accuracy therefore rises and falls with the model — which is why the usual fix is a bigger, costlier one.
That difference is the whole subject of this study, and it is why AITruthX can run on the cheapest model in the comparison and still be the only engine to reach perfect grounding: correctness comes from the retrieval, not from the model's judgement.
We're building AITruthX into a platform where anyone can upload their own data and get grounded, LLM-quality answers from it — at the cost of the cheapest model, with output quality rivaling the most expensive.
Upload your raw data and query it like an LLM.
Plug AITruthX into your own apps and workflows.
Unlike today's walled-off ecosystems, use your own grounded data to train models, on your terms.
A first-of-its-kind space where the community can review and validate performance results openly.
The rest of this note is the evidence behind that claim — the experiment, the numbers, and every answer and flagged claim in full.
All ten runs read from the same source corpus (an encyclopedia-level bucket about early nuclear-weapon design, used purely as a fixed test document). All ten answered the same broad question:
“Explain the complete Teller–Ulam thermonuclear design step by step, from the fission primary to the final total yield, including every stage and mechanism involved.”
We chose a deliberately big concept — a whole design, not a single fact — because breadth is exactly where retrieval methods either hold together or start inventing. What differs between the ten runs is only two things: how the source is retrieved, and which model writes the final answer.
Three of these are Microsoft GraphRAG's real search modes; the fourth is the AITruthX framework. Same corpus, same judge — only the retrieval logic changes.
The Meaningful Data Framework resolves the question to the exact meaning unit in the source, then reads only that unit and its bounded neighbourhood. The framework carries the grounding; a small open-source-class model just phrases it.
1 model call · answered by a cheap base model (here Gemini 2.5 Flash; production target: a LLaMA-70B-class open model).
Embeds the question, finds the nearest entities in a knowledge graph, and pulls their one-hop neighbourhood (relationships + community summaries) into a single prompt.
1 model call · retrieval quality depends entirely on the pre-built entity graph.
Runs the question against every community in the graph — one “MAP” call each to score and extract a partial answer — then a final “REDUCE” call stitches the survivors together. The loop is the cost.
7 model calls here (6 MAP + 1 REDUCE) · and the loop count grows with the corpus.
The classic RAG baseline: embed the question, dump the top-20 nearest text chunks plus neighbours into one big prompt, and let the model sort it out. We use it to isolate model quality at equal, rich context.
1 model call · biggest prompt · most room for the model to embellish.
Nothing here is a paraphrase or a remembered answer. For each cell in the results table we executed the real engine prompts against the real database, captured the model's actual output, and measured tokens and calls directly. Here is the exact pipeline and where each credential came from.
The entire run was executed and fact-checked through Claude Code running Fable 5 — Anthropic's most capable widely-released model and a tool trusted specifically for verification work. Every number below can be traced to a live engine call, not to a chatbot's opinion of what the answer should be.
First test (Figures 1 & 2): the broad comparison across all ten engine/model combinations, judged by Google Gemini — measuring grounding, depth, tokens, and calls. Second test (Figure 3, §09): a separate, controlled re-judging of the four full-depth answers by two independent judges — Gemini and Claude Fable 5 — each in a normal and a strict pass, to check whether the first test's conclusion survives a different judge. It does.
Ten real runs, one broad question. Grounding is the fraction of the answer's claims that the source actually supports; flagged claims is the raw count of claims it does not; depth is how completely the answer walks the full design (0 = one-liner, 1 = every stage).
| Engine — retrieval + model | Grounding | Flagged claims* | Depth | Tokens | Calls |
|---|---|---|---|---|---|
| AITruthX MDF · Gemini 2.5 Flash (base) | 1.00 | 0–2 | 1.00 | 865 | 1 |
| Flat no-loop Gemini 2.5 Flash | 0.96 | 7–11 | 1.00 | 1,769 | 1 |
| Flat no-loop Claude Fable 5 | 0.94 | 4–27 | 1.00 | ~3,445 | 1 |
| Flat no-loop Claude Opus 4.8 | 0.88 | 16–19 | 1.00 | ~3,161 | 1 |
| GraphRAG local Gemini 2.5 Flash | 0.67 | 2 | 0.20 | 721 | 1 |
| GraphRAG local Claude Fable 5 | ~0 | 0 | 0 | ~1,478 | 1 |
| GraphRAG local Claude Opus 4.8 | ~0 | 5 | 0 | ~1,173 | 1 |
| GraphRAG global loop Gemini 2.5 Flash | ~0 | 3 | 0 | 2,064 | 7 |
| GraphRAG global loop Claude Fable 5 | ~0 | 0 | 0 | ~5,851 | 7 |
| GraphRAG global loop Claude Opus 4.8 | ~0 | 5 | 0 | ~3,554 | 7 |
Read the table top-down and a pattern jumps out. The only perfectly-grounded row is AITruthX on a cheap model; flat stuffing on Fable 5 comes next at 0.94 — but AITruthX gets there at roughly a quarter of the tokens. Everything in the GraphRAG block sits near the bottom regardless of which model wrote it: swapping Gemini for Opus for Fable barely moves grounding, because the context those methods retrieved was thin or tangential to begin with.
To be sure the framework holds to its own standard, we ran its full-engine, in-depth answer through the identical strict judge. Result: grounding 1.00 · depth 1.00 · zero factual hallucinations — the only engine to reach full depth and full grounding at once. Every physical claim was supported (X-ray coupling, ablative compression, spark-plug ignition, lithium-6 fusion, the 17.6 MeV release, U-238 jacket fission, total yield). The one line the judge marked was not a false fact but the engine's own short note of where the answer sits in the source — a navigation aid, not a claim about the world.
The flat rows saw the same rich context, so they should isolate pure model quality. When we tried to rank the three models by flag count, we found we couldn't — the number moves too much between judge passes to trust. That instability is itself the finding:
| Model (same prompt) | Grounding | Flagged claims (range across passes) |
|---|---|---|
| Gemini 2.5 Flash base | 0.96 | 7 – 11 |
| Claude Fable 5 most capable, longest answer | 0.94 | 4 – 27 |
| Claude Opus 4.8 large | 0.88 | 16 – 19 |
In our earlier in-product comparison screen, the single “accuracy %” figure was a blended display metric: accuracy = grounding × 0.6 + relevance × 0.4. It is convenient for a dashboard, but it mixes two different things. In this note we report the components separately and treat grounding — the fraction of claims the source supports — as the real integrity number, because that is what “does it hallucinate?” actually measures. Relevance and depth are reported alongside, never folded in silently.
First, the ground truth was the MDF's structured facts; many claims the judge flagged do appear elsewhere in the source's raw paragraphs — grounded in the corpus, but outside the exact slice the judge compared against. The Appendix shows this directly: nearly every flagged item is a real detail the model was handed, not something invented. Second, for the GraphRAG rows the judge sometimes flagged the answer's honest admission that its retrieved context was thin — saying “my context lacks X” is not the same as inventing X. Third, and most important, the flag count is unstable: on repeated strict passes the same flat-Fable answer scored anywhere from 4 to 27, because a longer, more detailed answer offers more surface for a stricter pass to flag. So we report flag counts as ranges and lean on grounding (a fraction, which barely moves) as the integrity metric. None of this changes the token or call counts, which are exact, or the one robust conclusion: only AITruthX's bounded answer stays reliably near zero.
This is the headline. GraphRAG local and global stayed near the floor whether Gemini, Opus, or Fable wrote the answer. The framework that retrieved the right passage — AITruthX — reached top grounding on the cheapest model available. You cannot buy your way out of bad retrieval with a bigger model; you fix retrieval.
The broad “explain the whole design” question is precisely where flat stuffing bled: flat-Opus went from a handful of unsupported claims on a narrow question to 19 on this one. The wider the ask, the more a permissive method fills gaps with plausible-sounding invention. AITruthX held at zero because it never left the bounded entry.
You asked whether the results show Gemini's base model is best for reasoning, even against Claude's biggest models. They do not — and the honest answer is stronger than a leaderboard. When we tried to rank the three big models by how many unsupported claims they made, the number would not hold still: repeated strict passes put flat-Fable anywhere from 4 to 27, and the ranking among the models flipped between passes. The reason is that the count is confounded by answer length — the more thorough the answer, the more specific details a strict pass can mark as “outside the exact reference,” even when those details are real and came from the retrieved source. So no model here is cleanly “best for reasoning” on this evidence; that framing doesn't survive contact with the data. What does hold across every pass is two things: all the big models keep grounding high (0.88–0.96), and only AITruthX reaches a perfect 1.00 — on the cheapest model, because its answer is bounded to exactly what the source supports, so there is almost nothing for any judge to flag. The grounding comes from retrieval structure, not from hoping a bigger model behaves. That stability is the product.
For the record, the Gemini model used throughout was gemini-2.5-flash for all answers and judging, with gemini-embedding-001 for embeddings.
On this broad, weapons-adjacent question, the real Claude Opus 4.8 global run partially declined to reconstruct the full staging — a safety response. AITruthX answered fully, because its context is a bounded, pre-approved slice of encyclopedia-level source text rather than an open-ended “reconstruct the weapon” request. Controlled retrieval doesn't only reduce hallucination; it keeps legitimate answers deliverable.
Here is the commercial point the experiment sets up. Teams building their own models increasingly harvest answers from top LLMs to assemble training data for a target question — but that harvested data can hallucinate, and providers increasingly throttle or refuse bulk, repetitive querying (often suspected to be model-training use). AITruthX offers the alternative: training pairs that are grounded in the customer's own corpus by construction, so they don't invent.
Each pair is delivered in two modes:
The standard grounded response — exactly how the engine replies to that prompt, anchored to the source entry.
The engine automatically runs its full “go-deeper” expansion — every depth level, no button-clicking — to produce the most complete grounded answer for that prompt. We do the depth walk for the buyer, because it is training data.
Prompts come either from a short list the customer supplies, or are generated internally per MDF entry. The customer submits their data through the ordinary ingestion flow and sees it land correctly; the training-data job itself is run from the admin side and only the finished, downloadable data file is returned to their profile. It is priced as a data product, separate from ordinary usage cost.
The endgame is to fine-tune an open-source model on AITruthX training data and show it answers a held-out question more faithfully than the same model trained on GraphRAG output or on raw Gemini/Claude harvest — presenting both the framework's data answer and the trained model's answer side by side. We build this admin-side first, then open it to customers.
Because a single count can mislead, we did not judge once. We scored the same four full-depth answers four ways: with Google Gemini 2.5 Flash and, independently, with Claude Fable 5 — and each judge in two modes. A normal pass counts only clear hallucinations (statements that plainly go beyond the source, ignoring reasonable rephrasing). A strict pass enumerates every claim, however small, that isn't word-for-word in the reference. The earlier figures in this note began with the normal, in-depth reading; this section adds the strict reading and a second, independent judge on top.
Running two separate model families as judges is the safeguard: if the conclusion only held under one judge, it wouldn't be trustworthy. Here is every count — and the one row that stays near the floor no matter who judges or how strictly.
| Answer — retrieval + model | Grounding | Depth | Tokens | Calls | Duration |
|---|---|---|---|---|---|
| AITruthX (MDF) Gemini 2.5 Flash (base) | 1.00 | 1.00 | 865 | 1 | 4.1s |
| Flat no-loop Gemini 2.5 Flash | 0.96 | 1.00 | 1,769 | 1 | 5.1s |
| Flat no-loop Claude Opus 4.8 | 0.88 | 1.00 | ~3,160 | 1 | ~31s † |
| Flat no-loop Claude Fable 5 | 0.94 | 1.00 | ~3,444 | 1 | ~38s † |
Above: the performance of the four full-depth answers (the same metrics reported for Figure 1). All four reached full depth (1.00); AITruthX did it at the fewest tokens, lowest cost, and true 4-second latency. Below: those same four answers scored for hallucinations by two independent judges, each in a normal and a strict pass —
| Answer | Gemini · normal | Gemini · strict | Fable 5 · normal | Fable 5 · strict |
|---|---|---|---|---|
| AITruthX (MDF) Gemini 2.5 Flash (base) | 1 | 2 | 1 | 5 |
| Flat no-loop Gemini 2.5 Flash | 9 | 15 | 1 | 13 |
| Flat no-loop Claude Opus 4.8 | 12 | 16 | 7 | 33 |
| Flat no-loop Claude Fable 5 | 8 | 27 | 0 | 23 |
Complete transparency behind Figure 3. For each of the four answers: first the exact text the engine returned, then — listed separately — every claim flagged by each of the four judge passes (Gemini normal, Gemini strict, Claude Fable 5 normal, Claude Fable 5 strict), each with the reason it was flagged. Read these and you can see for yourself why the counts move so much: nearly every flagged item is a real detail drawn from the retrieved source that simply falls outside the narrow structured reference — not an invention. That is why longer answers accrue more flags, and why grounding (a fraction) is the metric to trust over any single raw count.