Most people think better retrieval fixes hallucination. It doesn't.
I've shipped RAG systems at BCG and built them at TUM, and the pattern is always the same: retrieval finds the right documents, and the model still invents an answer those documents don't support. The retrieval step did its job. The generation step didn't care.
The fix isn't retrieval. It's three things almost nobody actually does.
1. Bounded generation
Telling a model "don't hallucinate" does nothing — it's not an instruction, it's a wish. What works is telling it what it's allowed to do: only use facts from these documents, and make that verifiable. In some of our systems we do this post-hoc — after generation, check whether the output actually cites something, and whether that citation holds up. If it doesn't, reject and regenerate. It's expensive. It also works.
2. Confidence thresholds, on both sides
Most RAG pipelines pull top-k documents and quietly assume they're relevant. They're often not. Score retrieval confidence and generation confidence separately. If either one falls below threshold, the answer should be "I don't know," not a best guess dressed up as fact. This sounds obvious written down. Almost no production system actually does it.
3. An explicit fallback chain
When the model can't answer from context, what happens next? If the answer is "it hallucinates anyway," the system is broken by design. We chain fallbacks: retrieval fails, try a different retrieval strategy; that fails, route to a different model or a human. The model is never allowed to just make something up because nothing else was configured.
The hard part
None of this is hard to build. The hard part is admitting your system needs it in the first place. RAG demos work beautifully in a lab because nobody's testing the edges. Production RAG hallucinates constantly — and the systems that don't are the ones built assuming they would.
See also: Why hallucination mitigation matters more than model size