Articles·News & press·18 Sept 2026·7 min read

Why RAG still hallucinates — and what to change first

You built a retrieval-augmented generation (RAG) system. It finds the right documents. It still gives wrong answers, sometimes with a citation attached. This is common, and it is usually not one problem but several. This guide helps you work out which ones you have.

Retrieval and generation fail differently

RAG has two steps. First it retrieves passages that look relevant to the question. Then a language model generates an answer with those passages in front of it. Either step can fail, and the fixes are different, so start by separating them.

Take a sample of wrong answers. For each one, ask: was the passage containing the correct answer among the retrieved results?

No — a retrieval failure. The model never saw what it needed.
Yes — a generation failure. The model had the evidence and still got it wrong.

Teams often spend months tuning retrieval when most of their remaining errors are generation failures. Count before you tune.

Common retrieval failures

The answer is split across chunks. Documents are cut into pieces for indexing. A condition in one piece and its exception in another may never be retrieved together.
Similar is not the same as relevant. Search by meaning finds text that sounds like the question. A superseded policy sounds exactly like the current one.
Out-of-date content. If old versions stay in the index, they will be retrieved.
The answer needs several documents. Questions such as “which suppliers are affected by this change?” need many records joined together, not the top few passages.

Common generation failures

Blending. The model mixes retrieved text with what it absorbed in training, and the result reads smoothly.
Over-reading. The passage says “may”; the answer says “will”.
Filling gaps. When the passages do not quite answer the question, a fluent model tends to complete the picture rather than say so.
Lost in the middle. With many passages supplied, material in the middle is more easily overlooked.
Inconsistency. Because generation involves sampling, the same question can produce different answers on different runs.

Why a citation can mislead

A citation feels like proof. It is worth being precise about what it shows. These are three different checks:

The citation resolves — the source exists and opens.
The source supports the claim — it actually says what the answer says.
The answer is correct — the claim is true, complete and current.

A system can score perfectly on the first and poorly on the second. If you report one “citation accuracy” figure, make sure you know which of these it measures.

What to change first

Before changing architecture, try the cheaper fixes. Many teams get most of the way with these:

Build a test set with unanswerable questions. Include questions whose answers are not in your documents. You cannot improve refusal if you never measure it.
Add a way to decline. Make “the documents do not answer this” an allowed, rewarded outcome.
Remove superseded content from the index, or label versions and filter on them.
Check claims, not just citations. Verify that each cited passage supports the sentence it is attached to.
Use fixed rules where the policy is fixed. If the decision is “orders over this amount need approval”, write a rule. Do not ask a model to rediscover it each time.

Well-built RAG systems do many of these things. “RAG” describes a pattern, not a quality level — what matters is the whole workflow and how it performs on your questions.

When the fixes are not enough

Some requirements are hard to meet while a language model is still the step that decides the answer:

The same situation must produce the same decision every time.
Every answer must point to a record a person can inspect.
The system must decline reliably when support is missing.
Errors are costly — financially, legally or to safety.

For those questions, it is worth looking at approaches that move more of the work ahead of the question and reduce what is left to generation. RAG vs Compile-Time Inference compares the two, including what the alternative costs. If a knowledge graph is on your shortlist, see GraphRAG vs Compile-Time Inference.

Whatever you choose, measure it the same way: the evaluation guide lists the separate measurements worth reporting.

FAQ

Why does RAG hallucinate even when it retrieves the right document?

Because retrieval and generation are separate steps. Retrieval puts relevant text in front of a language model, but the model still writes the answer. It can overlook the passage, misread it, blend it with what it learned in training, or add detail that is not there.

Does a citation mean the answer is grounded?

No. A citation shows that a source exists and was available. It does not show that the source supports the specific claim. Check claim support separately from whether the citation resolves.

How do I tell a retrieval failure from a generation failure?

For each wrong answer, check whether the passage containing the right answer was among the retrieved results. If it was not, it is a retrieval failure. If it was there and the answer is still wrong, it is a generation failure. The fixes are different.

Should I replace my RAG system?

Not necessarily. Many problems are fixed by better retrieval, a refusal path and claim-level checks. Consider a different approach for the questions where answers must be checkable and repeatable, and where errors are expensive.

Bring a question your AI needs to get right.

Compile-Time Inference prepares your records before questions arrive, so supported answers and decisions can be checked against their evidence. Start with the questions where a wrong answer costs the most.

See Compile-Time Inference Download the brochure (PDF)Talk to us