The first few times a RAG system gave me a bad answer, I did what I think everyone does: I went and fiddled with the prompt. Made it stricter. Added a "only answer from the context" line. It barely moved the needle.

What finally fixed things was looking one step earlier. Nine times out of ten the model wasn't the problem at all — the right passage just never showed up in the context window, so there was nothing to ground the answer on. You can't prompt your way out of missing evidence.

So here's what I now reach for when retrieval is the weak link, roughly in the order I'd try them.

The short version

The three I'd reach for first

Hybrid search, so you stop losing exact terms

This one bit me directly. We had a support bot that could happily explain what a connection error was but couldn't find the doc for ERR_CONN_REFUSED specifically, because dense embeddings smear that exact token into "something about connections." BM25 finds it instantly. The fix is to run both retrievers and merge the two ranked lists with Reciprocal Rank Fusion — no score calibration needed, which is the nice part:

dense_hits  = vector_store.search(query, k=20)
sparse_hits = bm25.search(query, k=20)

def rrf(rank, k=60):
    return 1 / (k + rank)

scores = {}
for rank, d in enumerate(dense_hits, start=1):  scores[d.id] = scores.get(d.id, 0) + rrf(rank)
for rank, d in enumerate(sparse_hits, start=1): scores[d.id] = scores.get(d.id, 0) + rrf(rank)

top_ids = sorted(scores, key=scores.get, reverse=True)[:10]   # doc IDs, best first

Reranking, so the best chunk is actually on top

First-pass retrieval optimizes for recall, which means it grabs a wide net and the genuinely-best passage often sits at rank 7, not rank 1. A reranker takes that shortlist and scores each pair more carefully. It costs a bit more than the initial search but it's pocket change compared to the generation call, so there's not much reason to skip it. Pull a wide shortlist, rerank, keep the top few:

candidates = retriever.invoke(query)          # grab 20–50, recall first
scores     = cross_encoder.predict([(query, d.page_content) for d in candidates])
top_k      = [d for _, d in sorted(zip(scores, candidates), key=lambda x: x[0], reverse=True)][:5]

Contextual chunking, so a chunk knows what it's about

Picture a chunk that just says "the limit is 5,000 requests per minute." Great — for which API? Which plan? On its own it's nearly unretrievable. The trick is to have an LLM write one line of context for each chunk and prepend it before embedding:

context = llm(f"Briefly situate this chunk within the document:\n{doc}\n\nCHUNK:\n{chunk}")
embed(f"{context}\n\n{chunk}")   # index the contextualized version

Yes, it costs an extra LLM call per chunk at index time. You pay it once, and it's the single change that moved our numbers the most.

Things I'd tell my past self not to do

• Don't start with the prompt. If the evidence isn't in the context window, the prompt is irrelevant. Log what retrieval actually returned before you change anything else — half the time the bug is obvious the moment you look.
• Don't set chunk size once and forget it. It's the biggest lever you've got. Try 256, 512, 1024 on your own data and measure; the "right" default depends entirely on your docs.
• Don't go dense-only. Pure vector search will keep missing codes, IDs, and rare names. Bolting on BM25 is the cheapest real win after chunking.
• Don't skip reranking and just stuff 20 chunks in. More context isn't better context — it's slower, pricier, and the model gets distracted. Retrieve wide, rerank, send five.
• Don't crank top_k to paper over bad ranking. Past a handful of good chunks you're mostly adding noise. Fix the ordering, then trim.

Wrapping up

Retrieval is where most of your RAG quality lives or dies, and the nice thing is these stack — cleaner chunks make the reranker's job easier, hybrid search feeds it better candidates, and so on. If you want the full version of this (embeddings, vector DBs, the query-rewriting tricks, agentic patterns, evaluation — way more than fits here), I keep a sorted reference over at CheatGrid's RAG cheat sheet. It's free and there's no signup.