RAG, and the many things people mean by it

The short version: RAG means fetching relevant text at query time and putting it in the prompt. The phrase now covers a dozen very different techniques — vector search, keyword search, hybrid, rerankers, graph, agentic — that fail in different ways. The retrieval step, not the model, is where accuracy is won or lost.

Retrieval-augmented generation began as one idea: instead of relying only on what a model absorbed during training, fetch relevant text at query time and put it in the prompt. Everything downstream — accuracy, freshness, the ability to cite a source — follows from that. But "RAG" has since become an umbrella over a dozen techniques that work differently and break differently. When you hear "we use RAG" and "RAG is dead" in the same week, it's because the speakers are describing systems that share little beyond the acronym.

Here is what's actually inside them.

The baseline, and why it's harder than it looks

The version most people picture has two phases.

Indexing, done ahead of time. You take your documents, split them into chunks, run each chunk through an embedding model to turn it into a vector — a list of a few hundred to a few thousand numbers that encodes the chunk's meaning — and store those vectors.

Retrieval, at query time. You embed the user's question the same way, find the stored chunks whose vectors are closest to the question's vector, paste the top few into the prompt, and ask the model to answer from them.

Two design decisions inside that sketch quietly determine whether the whole thing works.

The first is chunking. You can't embed a whole document as one vector — it would blur everything together — so you split it. But where? Fixed-size splits (say, every 500 tokens) are simple and cut sentences in half. Recursive splitting respects paragraph and sentence boundaries. Structural splitting follows the document's own headings and sections. Semantic splitting tries to break where the topic shifts. The size matters as much as the method: small chunks are precise but strip away surrounding context, so a retrieved sentence can lose the qualifier that changes its meaning; large chunks keep context but dilute the signal, so the one relevant line is buried among nine irrelevant ones and the similarity score drops. Most retrieval failures that look like "the model is dumb" are actually a chunk that severed a fact from what it depended on.

The second is search at scale. Comparing the query vector against every stored vector exactly is too slow once you have millions of them, so vector databases use approximate nearest-neighbor search — algorithms like HNSW that build a navigable graph of vectors and trade a small amount of recall for an enormous speedup. This is fine, but it means even the "find the closest vectors" step is an approximation with its own tuning knobs.

And then there's the failure that no amount of tuning fixes: semantic similarity is not the same as containing the answer. The nearest vectors to "what's our refund policy" are chunks that are about refunds — which may or may not be the chunk that states the actual rule. The embedding captures topic, not answerhood. Most of the more advanced variants below exist to close that gap.

Dense, sparse, and hybrid

The vector approach is dense retrieval: it matches on meaning. Its blind spot is literals. Ask for error code E-4021 or a customer named "Knaus" and dense retrieval flounders, because those tokens carry meaning the embedding can't generalize — there's nothing semantically "near" a specific code or surname.

The older, pre-neural approach is sparse retrieval, of which BM25 is the workhorse. It matches actual words, weighting them by how rare they are across the corpus (a word that appears in every document tells you nothing; a word that appears in three documents is a strong signal) and how often they appear in a given document. It is exactly good at what dense retrieval is bad at: finding the document that contains this specific term.

The two have complementary blind spots, which is why hybrid retrieval — running both and merging the results, often with a method called Reciprocal Rank Fusion that combines the two ranked lists — is one of the highest-return upgrades over naive RAG. Real queries routinely mix a concept ("how do we handle a chargeback") with a literal ("on order 44812"), and you want a system that can match on both.

Rerankers, and why they're a different kind of model

Retrieval is built for speed over millions of candidates, which means it casts a wide, slightly sloppy net. A reranker is a second, slower, more accurate model that re-scores a short list of candidates — say the top 50 — and keeps the best handful.

The reason it can be more accurate comes down to architecture. The embedding model used for retrieval is a bi-encoder: it encodes the query and each document separately, into independent vectors, and compares them. That separation is what makes it fast (you can embed all your documents in advance) but also what limits it — the query and document never actually "see" each other during encoding. A reranker is typically a cross-encoder: it feeds the query and a candidate document through the model together, so every word of the query can attend to every word of the document. That's far more accurate at judging relevance, and far too slow to run against your whole corpus — which is exactly why you run it only on the top candidates that retrieval already narrowed down. Retrieve broadly, then rerank precisely. The generator ends up seeing a short list that was actually judged for relevance, not just for vector proximity.

Fixing the chunk, not just the search

A 2024 technique from Anthropic, contextual retrieval, attacks the chunking problem head-on. Before embedding each chunk, it prepends a short, model-generated description of where that chunk sits in its parent document — what it's about, what came before. A chunk that originally read "The figure rose 15%" becomes something the index can actually match, because it now carries "In the Q3 revenue section, discussing the North American segment: the figure rose 15%." It's a simple idea that meaningfully cuts the rate at which the right chunk fails to be retrieved, precisely because it restores the context that splitting tore away.

Graph RAG

Everything so far retrieves chunks in isolation, then hopes the model can stitch them together. That works for "look up a fact" and struggles with "synthesize across everything." If you ask "how is this person connected to that project" or "what are the main themes across these 400 documents," top-k chunk retrieval returns a scattered handful and no structure.

Graph RAG — Microsoft published a well-known implementation in 2024 — restructures the problem. In an indexing pass, it uses a model to extract entities and the relationships between them and assembles them into a graph; it can also pre-compute summaries of clusters ("communities") within that graph. Retrieval then traverses structure instead of grabbing nearest neighbors. For aggregation and multi-document questions this is a categorical improvement, because the connections between facts are stored explicitly rather than left for the model to reconstruct from fragments. The cost is real: extracting and maintaining the graph is significant up-front work, and it only pays off for question types that actually need it.

When the query itself is the problem

Sometimes retrieval fails not because the index is bad but because the user's raw question is a poor search key.

• Query rewriting and expansion rephrase or decompose the question into better search terms before retrieving — turning a vague question into one or several sharper ones.
• HyDE (Hypothetical Document Embeddings) does something counterintuitive: it asks the model to write a hypothetical answer first, then searches using that fake answer's embedding. The intuition is that an answer, even an imperfect one, looks more like the documents you're trying to find than the question does.

Agentic and iterative retrieval

Every variant above retrieves once. Agentic RAG retrieves in a loop: search, read what came back, decide whether it's enough, search again with what you just learned, and stop when you can answer. This is what genuine multi-hop questions require — where you can't even form the second query until you've seen the answer to the first ("who manages the person who approved this budget?").

It's strictly more capable and strictly more expensive. Each hop is another model call and another retrieval; latency stacks up; and a loop without a good stopping condition burns tokens chasing diminishing returns. Used with discipline it answers questions a single retrieval simply cannot. Used carelessly it's a slow, costly way to get the same answer.

A decision guide

Doesn't a long context window make all this obsolete?

This is the live debate behind "RAG is dead." If a model can take a million tokens, why not skip retrieval and paste in everything?

Three reasons it doesn't go away. Scale: your corpus is almost always larger than any window — a company's documents don't fit in a million tokens, let alone an individual's lifetime of context. Cost and latency: as covered elsewhere on this blog, filling a giant window is expensive on every request and the model attends unevenly to a full one, so stuffing everything in is both pricier and often less accurate than retrieving the right slice. Freshness: a retrieval index can be updated the moment a document changes; a model's training cannot.

Long context and retrieval aren't competitors. A long window raises the ceiling on how much retrieved context you can usefully include. Retrieval is still what decides what goes in it.

The part that's actually hard

Notice that none of these techniques is about generation. The model writing the answer is, at this point, the most reliable component in the pipeline — and the most dangerous, because a capable model handed the wrong three paragraphs will produce a fluent, confident, well-cited, wrong answer. It will not flag that its sources were irrelevant. The accuracy of a RAG system is set almost entirely by whether the retrieval step surfaced the right context, and almost not at all by how good the model is at writing.

That's the real reason "RAG" is worth understanding past the acronym. The interesting engineering was never the generation. It was always the retrieval — and "we use RAG" tells you which of a dozen quite different retrieval systems someone built, which is the only part that determines whether it works.