0 % vs 50 %: making a RAG agent refuse to hallucinate
2026-05-31 · LLM / RAG
A retrieval-augmented agent is only as trustworthy as its behaviour on questions whose answer
isn't in the corpus. The failure mode is quiet: instead of saying "I don't know," the model
invents a confident, well-formed, wrong answer. This post shows a single guardrail that takes
that from common to never — and, crucially, measures it.
Reference architecture:
nim-agent-blueprint — agentic RAG on
the NVIDIA NIM stack with a built-in eval harness.
The ablation
The agent loop is plan → retrieve → generate → validate. The interesting variable is the
generation prompt's contract with the retrieved context:
| Configuration | Out-of-corpus hallucination rate |
|---|---|
| Generate freely from context | ~50 % |
| Guarded prompt (answer only from context; otherwise abstain) | 0 % |
Same model, same retriever, same questions. The only change is a prompt that makes "I can't
answer that from the provided sources" a first-class, rewarded output — plus a validate
step that checks the answer is grounded in retrieved spans before returning it. On in-corpus
questions, retrieval recall@3 stayed at 94–100 %, so the guardrail buys safety without
costing coverage.
Why "just prompt better" isn't the lesson
The lesson isn't the prompt — it's that the difference between 50 % and 0 % is invisible
without an eval harness. A demo that only asks in-corpus questions looks perfect in both
configurations. You only see the 50 % when you deliberately ask things the corpus can't
answer and score groundedness. So the blueprint ships with:
- retrieval hit-rate (is the answer even retrievable?),
- answer groundedness via LLM-as-judge (is the answer supported by what was retrieved?),
- latency, and OpenTelemetry traces per agent step.
That's the difference between "it works on my five questions" and "here is the number a
partner can hold me to."
Takeaway
For enterprise RAG, abstention is a feature, not a failure. Make "I don't know" a rewarded
output, validate groundedness before returning, and measure the out-of-corpus rate — it's
the number that separates a demo from something you'd put in front of a customer.
→ Runnable blueprint + eval harness:
github.com/waynehacking8/nim-agent-blueprint