The Problem

You have a 400‑page internal handbook includes compliance rules, HR policies, engineering runbooks. You ask an LLM: “What’s the approval chain for a budget over $50k?”

Without RAG, the model hallucinates: “The VP of Finance and the CTO must both approve.” But your real policy says: “Only the CFO for >$50k, plus a board note if >$200k.”

The core problem: LLMs are frozen at training time. They don’t know your private documents. Fine‑tuning is expensive, lags behind updates, and still suffers from parametric knowledge bleed. RAG solves the specific problem of grounding generation in fresh, proprietary, or long‑tail facts without retraining.

But naïve RAG (chunk → embed → retrieve → stuff into prompt) breaks in surprising ways. This article walks through one real failure, three common failure modes, and the guardrails we built to make RAG production‑ready.

The Dry‑Run: Answering an Employee’s Parental Leave Question

Scenario: An employee asks a Slack bot: “How many weeks of paid parental leave do I get, and do I need to notify HR before birth?”

The source is a 50‑page PDF Parental Leave Policy v4.2, last updated 3 months ago.

Step 1 – Chunking

Step 2 – Embedding & Indexing

Each chunk is passed through text-embedding-3-small (1536 dimensions). We store vectors in a pgvector index together with metadata (page number, section title, last update date).

Step 3 – Query Embedding

Step 4 – Retrieval

Vector similarity (cosine) returns top‑5 chunks. Example chunks retrieved:

1. “Eligible employees receive 12 weeks of fully paid parental leave.” (score 0.92)
2. “Birth mothers may take an additional 4 weeks of paid medical recovery leave, distinct from parental leave.” (score 0.89)
3. “Notification: Employee must submit a leave request in Workday at least 30 days before the expected birth date.” (score 0.87)
4. “Adoptive parents receive the same 12 weeks but no medical recovery weeks.” (score 0.76)
5. “Leave can be taken intermittently with manager approval.” (score 0.68)

Step 5 – Generation Prompt

We assemble a prompt:

You are an HR assistant. Use ONLY the following context to answer the question.
If the answer is not in the context, say "I don't know."

Context:
[chunk1] [chunk2] [chunk3]

Question: How many weeks of paid parental leave do I get, and do I need to notify HR before birth?

Answer in a clear, bulleted list.

Step 6 – LLM Response

The model correctly outputs:

• 12 weeks of fully paid parental leave for all eligible employees.
• Birth mothers get an additional 4 weeks of paid medical recovery leave.
• You must notify HR via Workday at least 30 days before the expected birth date.

Success – no hallucination about a “CTO approval”.

Failure Modes (Where RAG Secretly Fails)

Even with the above, we see three catastrophic failure patterns in production.

Failure 1 – The “Lost in the Middle” Problem

Failure 2 – Low‑Relevance Retrieval (But High Cosine Score)

Failure 3 – Contradictory Chunks

Two chunks in the same document:

• Chunk A: “You may use PTO during parental leave to top up pay.” (old version)
• Chunk B: “As of Jan 2025, PTO cannot be used to top up parental leave pay.”

Guardrails (Engineering Fixes for Each Failure)

We implemented five explicit guardrails on top of the basic RAG pipeline.

Guardrail 1 – Reranking with Cross‑Encoder

Guardrail 2 – Chunk Positioning Weighting

Guardrail 3 – Contradiction Detector

Before sending chunks to the LLM, we run a lightweight entailment model (roberta-large-mnli) to check for contradictions. If two chunks have CONTRADICTION score > 0.8, we include both but add a system instruction: “The following two sources contradict each other. Explain the discrepancy and default to the newer one based on document version.”

Guardrail 4 – Forced Citation

We require the LLM to output citations like [src: page 12]. We parse the response. If any statement lacks a citation, we reject and retry with a stricter prompt.

Guardrail 5 – Hybrid Search

We augment vector search with BM25 keyword matching. For queries with rare terms (e.g., “Workday notification”), BM25 finds the exact phrase that embedding might smooth over. Final score = 0.6 * vector + 0.4 * BM25.

Architecture Diagram

Explanation of the diagram:

• User query is processed in parallel (embedding + BM25).
• Vector + BM25 results are fused (not shown for simplicity, but it’s inside the DB step).
• Reranker reduces to top‑5 most relevant chunks.
• Contradiction detector adds metadata before prompting.
• LLM generates, then citation validator enforces groundedness. Retry loop prevents hallucinated claims.

Conclusion

RAG is not “just glue code”. Without reranking, contradiction detection, and forced citations, your bot will confidently produce wrong answers from the same document. The guardrails above have reduced hallucination rate on our internal HR dataset from 23% to 4.7% (measured by human‑evaluated citation correctness).