• Book: RAG Pocket Guide
• Also by me: Database Playbook
• My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
• Me: xgabriel.com | GitHub

You ship a RAG bot to a 4,000-employee company. The first demo question is "Who reports to the VP of Engineering's manager?" The retriever returns three pages from the org-chart deck and a 2024 all-hands transcript. The model answers confidently and gets the name wrong, because no single page in the deck contains the answer. The answer was a two-hop traversal that lived in nobody's chunk.

The team blames the chunk size. They tune from 800 to 1200, add overlap, swap the embedding model. The number on Ragas crawls up by two points. The wrong-name failure stays.

This is the wall every team running pure vector RAG hits eventually, and it is why almost every serious RAG vendor in 2026 (Microsoft, Neo4j, and the broader GraphRAG ecosystem on top) has been adding a graph layer underneath. This post is about why.

The three failure modes vector RAG cannot fix with chunking

Pure vector retrieval over chunked text is good at one thing: surface a passage that talks about something similar to your query. It is bad at three things, and chunk-size tuning will not save you from any of them.

Entity disambiguation. Three companies named "Acme Corp" appear in your corpus: a customer, a competitor, a 1990s subsidiary that got spun off. The chunks containing each one look semantically similar. The retriever returns a mix and the model conflates them. You do not have a "name" problem; you have an "identity" problem. Identity needs nodes, not strings.

Multi-hop questions. "Which contracts signed by anyone reporting to Sarah's old team are up for renewal in Q3?" That requires resolving Sarah, walking her org tree as of a date, joining to a contracts table, filtering by a renewal predicate. No chunk contains that answer. Chunks contain pieces. Vector retrieval gives you pieces. Synthesis fails because the model never sees the join.

Relationship reasoning. "Show me companies in our portfolio that had board members in common with Stripe before 2022." Embeddings can find passages about board overlaps if they exist verbatim. They cannot compose typed edges. The corpus may have all the information and your retriever still cannot answer, because the answer is a graph query, not a similarity match.

These three are the dominant ceiling on enterprise RAG, and the response across the field has been the same: add a graph layer.

What "adding a graph layer" actually means

A graph layer in 2026 RAG looks like this. Nodes for entities: people, companies, products, projects, documents. Typed edges between them: EMPLOYS, OWNED_BY, CITED_IN, SIGNED_BY. Properties on both: names, dates, identifiers. Each node also carries pointers back to the chunks it was extracted from, so traversal results can be hydrated into LLM context.

The graph is not a replacement for the vector store. It is a parallel index over the same corpus. At query time, you run two retrieval passes and fuse the results.

The pattern shows up in three lineages.

Microsoft GraphRAG, open-sourced via Microsoft Research and now widely forked, builds a hierarchical community graph from extracted entities and uses Leiden clustering to summarize subgraphs. It is heavy at index time and excellent at global-summary questions across a corpus.

LightRAG, presented at EMNLP 2025 and available on GitHub, keeps the graph layer but drops the community-summarization step. It runs dual retrieval (graph traversal and vector similarity) and fuses results. According to Neo4j's benchmark writeup, indexing a 500-page corpus runs in the order of minutes and a fraction of a dollar in LLM calls, with quality close to GraphRAG's at roughly two orders of magnitude lower cost.

Neo4j's hybrid vector-and-graph approach treats the graph database itself as a hybrid store: native vector indexes alongside native traversal, with the same nodes carrying both an embedding and typed edges. The retrieval pattern is one query that combines db.index.vector.queryNodes with a Cypher traversal.

Three different teams, arriving at the same insight from three different architectures: vector for proximity, graph for structure, fuse the two.

A 40-line hybrid retrieval example

Here is the smallest non-toy version that runs. It uses NetworkX for the graph (replace with Neo4j in production) and pgvector for embeddings.

import networkx as nx
import psycopg
from openai import OpenAI

client = OpenAI()
db = psycopg.connect("postgresql://localhost/rag")

# Graph: entity nodes + typed edges, built at ingest time.
G = nx.DiGraph()
# G.add_node("emp:42", name="Sarah Kim", role="VP Eng")
# G.add_edge("emp:88", "emp:42", type="REPORTS_TO")
# G.nodes["emp:42"]["chunk_ids"] = [120, 873]

def embed(text: str) -> list[float]:
    r = client.embeddings.create(
        model="text-embedding-3-large", input=text
    )
    return r.data[0].embedding

def vector_topk(query: str, k: int = 8) -> list[int]:
    q = embed(query)
    rows = db.execute(
        "SELECT id FROM chunks "
        "ORDER BY embedding <=> %s::vector LIMIT %s",
        (q, k),
    ).fetchall()
    return [r[0] for r in rows]

def graph_neighbors(seed_entities: list[str], hops: int = 2) -> set:
    visited = set(seed_entities)
    frontier = set(seed_entities)
    for _ in range(hops):
        nxt = set()
        for n in frontier:
            nxt.update(G.successors(n))
            nxt.update(G.predecessors(n))
        frontier = nxt - visited
        visited |= frontier
    return visited

def hybrid_retrieve(query: str, seeds: list[str]) -> list[int]:
    vec_ids = set(vector_topk(query, k=8))
    nbrs = graph_neighbors(seeds, hops=2)
    graph_chunks = set()
    for n in nbrs:
        graph_chunks.update(G.nodes[n].get("chunk_ids", []))
    return list(vec_ids | graph_chunks)

# entity linking on the query is its own step, often a small classifier
# or a tool call: "extract entities present in our graph"
ids = hybrid_retrieve(
    "Which direct reports of Sarah's manager joined after 2024?",
    seeds=["emp:42"],
)

The shape: vector retrieval brings back semantically relevant chunks; graph traversal brings back chunks anchored to the entities the question is about. The union goes to the reranker. Reciprocal Rank Fusion is the standard trick if you want a single ranked list rather than a union, and the arXiv paper on practical GraphRAG covers the variants.

Replace NetworkX with Neo4j when you outgrow process memory. The retrieval shape does not change.

Where the graph earns its keep

Three workloads make the graph layer pay for itself almost immediately.

Org-aware QA. Anything that involves "who manages whom," "who works on what," or "what team owns this." The vector store can find the right document; only the graph can answer the relational question.

Contract and clause cross-reference. "List all MSAs where the data-processing addendum references Section 7.3." Section references are edges. Walking them is a traversal. A reranker over chunks will get half of them and miss the long-tail.

Multi-document synthesis. Knowledge-base questions that require facts from three or four documents to compose an answer. Pure vector retrieval treats each as an independent passage. The graph treats them as connected via shared entities, and the connectedness is the signal you want.

A useful rule of thumb: if your evaluator's failure log contains "the answer is correct but combines facts from documents the retriever returned only one of," you are looking at a graph problem, not a chunk problem.

What the graph does not give you for free

Honesty about the cost.

Entity extraction is an LLM call per passage at ingest time. Even at LightRAG prices, a 100k-document corpus runs into real money — and you re-pay on every corpus refresh unless your incremental ingest is careful.

Schema drift is real. Nobody designs an entity schema correctly the first time. You will redo extraction. Plan for it.

Entity linking at query time is its own subproblem. The user typed "Sarah." Which Sarah? You need a small classifier, a fuzzy lookup, or a tool call that asks the LLM to enumerate the entities in your graph that match. Get this wrong and traversal seeds the wrong nodes; the graph then confidently returns the wrong answer.

The graph is also a moving target. People change roles. Companies merge. Time-travel queries ("who was on the board in March 2023?") require versioned edges. That is doable but it is not free.

Why this is happening now and not in 2023

Two things changed.

LLM-driven entity extraction got cheap enough to run at corpus scale. In 2023, decomposing a 10k-document corpus into typed entities and edges cost thousands of dollars and took days. In 2026, with smaller models doing structured extraction at a fraction of the price, the same job is a Saturday-afternoon batch.

Hybrid stores caught up. Neo4j shipped vector indexes that sit next to graph data. pgvector matured. Qdrant added named graph collections. You no longer maintain two separate systems with their own ops profiles; the graph and the vectors live in the same database, and a single retrieval call hits both. Recent writeups like Qdrant's GraphRAG with Neo4j guide and the paperclipped 2026 production comparison are decent reads on the operational side.

The result is that the architecture cost dropped past the engineering benefit, and now everyone serious is building it.

Picking the layer for your stack

A short version.

If your corpus is under 10k documents and your questions are mostly single-hop fact retrieval, vector RAG with parent-document retrieval is enough. Do not add complexity you do not need.

If you have hit the multi-hop wall, start with LightRAG. By "multi-hop wall" I mean questions that obviously have answers in your corpus but the retriever cannot compose them. The cost-quality ratio is the best in the category, the dual-retrieval pattern is the simplest one to reason about, and the operational shape (one process, two retrieval calls, fuse) is small enough to review in an afternoon.

If you need global summaries (questions like "what are the top themes across this 50,000-document corpus?"), Microsoft GraphRAG's community-summarization layer is genuinely doing work the lighter alternatives cannot. Pay the index cost; it is worth it.

If your retrieval is already running on Neo4j or you have a real graph problem (heavy entity reconciliation, time-travel queries, deep traversal), do the hybrid in one store rather than gluing two together. The operational savings dominate the architectural debate within a quarter.

The thing the field figured out in 2026 is that retrieval was never just a similarity-search problem. It was a "find the right things and understand how they connect" problem, and the second half needs structure that pure embeddings do not have. Add the graph layer before you tune your chunk size again.

If this was useful

The graph-layer pattern, the parent-document split, the chunking and reranking choices that sit underneath them: those are the spine of RAG Pocket Guide. Picking the store that hosts both your vectors and your graph without becoming an ops nightmare is the spine of the Database Playbook. If you are building toward the architecture above, both books speak to the same Monday-morning decisions.