Retrieval-Augmented Generation (RAG) — a pattern where a language model answers questions by first pulling relevant document chunks from a search index, then generating a response grounded in those chunks — is not magic. It is an engineering discipline, and it fails in predictable ways when teams skip the architecture decisions that make retrieval honest. This article covers those decisions: where keyword search breaks down, how to design an ingestion pipeline that holds up under real enterprise corpora, and how to audit retrieval accuracy before you ship anything to users.

---

Why Keyword Search Fails Enterprise Knowledge Bases

Keyword search matches tokens. A query for "equipment return policy after contract termination" will miss a document titled "offboarding asset collection procedures" even if both describe the same process. For small corpora — a hundred documents, stable vocabulary — this gap is tolerable. For an enterprise knowledge base with hundreds of contributors, inconsistent terminology, and documents spanning five years of policy drift, it becomes a structural failure mode.

Vector search solves this by encoding meaning, not tokens. A dense vector embedding maps a sentence into a high-dimensional space where semantically similar text lands close together, regardless of surface wording. The tradeoff: vector search can over-retrieve. It will surface plausible-sounding documents that are not actually relevant because the embedding model generalizes too aggressively.

The production-grade answer is a hybrid retrieval layer: run both keyword (sparse) and vector (dense) retrieval in parallel, then merge the ranked lists using a reciprocal rank fusion algorithm before passing candidates to the language model. This is not a novel idea — the IR research community has used fusion techniques for years under the label "hybrid retrieval." What is new is that most managed vector databases (Pinecone, Qdrant, Weaviate, and others) now expose this as a first-class option, which removes the excuse for deploying pure vector search alone.

One honest caveat: hybrid retrieval adds operational complexity. You are now maintaining two index types, and tuning their relative weights for your specific corpus takes real evaluation work. Do not add it reflexively. Add it when you have measured evidence that either mode alone is failing.

---

How Do You Design an Ingestion Pipeline That Preserves Document Structure?

The ingestion pipeline is where most enterprise RAG systems quietly break. Teams chunk by fixed token count, generate embeddings, write to a vector store, and call it done. Then retrieval returns fragments that begin mid-sentence and end before the relevant clause, and the language model hallucinates to fill the gap.

A better ingestion design makes three deliberate choices:

Chunking strategy. Fixed-size chunking with overlap (e.g., 512 tokens with a 64-token overlap) is a reasonable baseline but not always correct. For structured documents — policy manuals, HR handbooks, technical runbooks — hierarchical chunking works better: keep the parent section intact as a "parent chunk" for context retrieval, and index smaller "child chunks" for precision. The LlamaIndex documentation calls this "small-to-big retrieval" and it is worth reading directly. When a child chunk retrieves, you pass the parent chunk to the model — you get precision on the search side and coherence on the generation side.

Embedding model selection. Do not default to whatever the vector database's hosted embedding suggests. Evaluate on your domain. A general-purpose embedding model trained on web text will underperform on dense technical or legal language. MTEB (Massive Text Embedding Benchmark) publishes ranked evaluations across domain types and is a legitimate starting point for shortlisting candidates. Pick two or three, run them against a held-out sample of your actual documents, and measure recall at k=5 before committing.

Vector database schema. Each chunk record needs more than the text and its vector. At minimum: source document ID, page or section reference, document creation date, content type (policy vs. procedure vs. FAQ), and access tier if your organization has document-level permissions. This metadata is not cosmetic — it is what enables the filtering and auditing steps that follow.

For teams concerned about long-term flexibility here, model neutrality in your AI infrastructure design is worth thinking through before you standardize on a single embedding provider.

---

Metadata Tagging Strategies for Enterprise Corpora

Metadata is retrieval infrastructure, not administrative overhead. A vector similarity score tells you a chunk is semantically close to a query; metadata filters tell you whether that chunk is applicable — current, from the right department, visible to the requesting user.

The tagging taxonomy for most enterprise corpora should cover at least these dimensions:

The practical challenge is populating this metadata at ingestion time. For well-managed document systems (SharePoint with enforced metadata, a structured CMS), you can extract most fields programmatically. For the more common case — a sprawling mix of PDFs, Word files, and wiki exports with inconsistent naming — you need a classification step in the ingestion pipeline. A lightweight classifier (even a small fine-tuned model or a structured prompt against a capable model) can assign doc_type and department with enough reliability to be useful, as long as you build a review queue for low-confidence classifications rather than auto-publishing them.

Never treat metadata as a set-and-forget step. As documents are updated, the metadata must be versioned alongside them. A stale effective_date on an archived policy is not a minor inconvenience — it is a liability if a user receives outdated guidance presented with apparent confidence.

---

How Do You Run Verifiable Retrieval Audits Before Deploying to Users?

Evaluating a RAG system is not the same as evaluating a language model. The model's generation quality is downstream of retrieval quality: if the wrong chunks are retrieved, no amount of prompt engineering fixes the answer. Retrieval audits must be a distinct, structured step before any deployment decision.

A practical audit process looks like this:

Step 1: Build a ground-truth evaluation set. Take 50–100 real questions that your knowledge base should answer — drawn from support tickets, HR inquiry logs, or stakeholder interviews — and manually identify the correct source documents for each. This is the only reliable way to know what "correct retrieval" looks like for your corpus.

Step 2: Run retrieval and score recall@k. For each evaluation question, run your retrieval pipeline and check whether the correct source document appears in the top k results (k=3 and k=5 are standard cutoffs). Recall@5 of around 0.80 or above is a reasonable minimum threshold before moving to generation evaluation. Below that, the generation quality does not matter — fix the retrieval first.

Step 3: Audit failure modes by category. Low recall failures cluster in recognizable patterns: vocabulary mismatch (keyword search is failing), out-of-scope documents flooding results (metadata filtering is missing), or very short documents with low embedding signal (chunking is too aggressive). Categorizing failures before fixing them prevents you from solving the wrong problem.

Step 4: Evaluate answer grounding. Once retrieval passes your threshold, evaluate whether the model's answers are actually grounded in the retrieved chunks or are fabricating. The RAGAS framework (an open-source evaluation library) provides structured metrics for this — specifically "faithfulness" and "answer relevance" — without requiring you to build custom evaluation tooling from scratch.

This is where RAG for an enterprise knowledge base earns its credibility or loses it. The evaluation step is not a QA formality; it is the mechanism that separates a working product from a demo that felt good in the boardroom. For a broader look at how this kind of discipline applies to operational AI workflows, the piece on AI workflow automation for operations teams covers the same principle across other automation surfaces.

If you are building this from scratch and want a reference implementation for the internal knowledge retrieval layer, the internal knowledge AI service overview describes the stack OpenCraft uses in production engagements.

---

FAQ

What is the difference between RAG and fine-tuning for an enterprise knowledge base?

RAG retrieves relevant documents at query time and passes them to the model as context. Fine-tuning bakes information into model weights during training. For enterprise knowledge bases, RAG is almost always the right choice: documents change, fine-tuning every update is impractical, and RAG provides source citations that make answers auditable. Fine-tuning is better suited to adjusting response style or domain-specific reasoning patterns.

How large does a document corpus need to be before RAG is worth the complexity?

There is no fixed threshold, but as a practical guide: if your knowledge base has more than a few hundred documents, inconsistent terminology across authors, or content that updates frequently, RAG pays for its complexity. Below that, a well-structured keyword search with a good UI often delivers more value with less operational overhead.

Which vector database should an enterprise team choose?

The decision depends on hosting constraints, existing infrastructure, and whether you need managed scaling. Qdrant and Weaviate both offer strong hybrid retrieval support and self-hosted options, which matters for data residency requirements. Pinecone is strong for managed, serverless deployments. Evaluate on your own query and corpus scale — benchmark numbers from vendor marketing are not a substitute for testing on your actual data.

How do you handle document permissions in a RAG system?

Enforce permissions at the retrieval layer through metadata filtering, not at the generation layer. If a user is not authorized to see a document, that document's chunks must be excluded from the retrieval candidate set before anything reaches the language model. Relying on the model to "not mention" restricted content is not a security control.

What should an enterprise do when retrieval quality degrades over time?

Corpus drift — new documents, updated policies, changing terminology — erodes retrieval performance incrementally. Build a scheduled re-evaluation job that runs your ground-truth query set against the live index at regular intervals. When recall@k drops below your threshold, trigger a re-chunking and re-embedding run on the affected document segments rather than waiting for user complaints.

---

A working RAG system for an enterprise knowledge base is not a product you install — it is an engineering decision stack you maintain. Get the ingestion pipeline right before you optimize prompts. Build the evaluation set before you demo to stakeholders. Every reliable knowledge base AI in production is built on retrieval discipline, not model capability alone. If you want a structured approach to that evaluation process, reach out to OpenCraft and we can walk through the audit framework with your actual corpus.

---

More from ocraft.id