Retrieval-augmented generation
A Socratic walk-through of retrieval-augmented generation — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why does handing a model the source document work better than training the document into it?
You have a company handbook and you want a model to answer questions about it. Two roads present themselves. Train the handbook into the model, so that it knows the policy. Or leave the model alone, find the relevant paragraph when a question arrives, and paste it into the prompt.
The first road looks obviously superior — it makes the knowledge part of the system rather than something bolted on each time. The second looks like a workaround. Yet the second is what nearly everyone builds, under the name introduced by Lewis and colleagues in 2020: retrieval-augmented generation.
So something about the first road is worse than it appears. What?
Reasoning it through
REASONING #Begin with what training actually does with a document. It performs many small adjustments to weights that already encode a great deal else, nudging the model toward the observed continuations. The handbook does not arrive anywhere in particular; it is dissolved into the same distributed representation everything else lives in. Ask the practical questions that follow. Where is clause 4.2 now? Nowhere addressable. How do you revise it when the policy changes next quarter? You train again, and hope the old version is displaced rather than left competing. How do you show a reader the source? You cannot — there is no source left, only a disposition.
Now ask what happens when a question arrives about something the handbook never covered. The model does not know that it does not know; it produces the most probable continuation given the pattern of a policy question, and a plausible policy is exactly what a model of policy documents is good at producing. The failure mode of trained-in knowledge is a fluent invention, and it is indistinguishable from the successes.
Contrast the second road. The paragraph sits in the context window as ordinary text. Now think about what the model is being asked to do: not recall, but read. Extracting an answer from a passage in front of you is a much easier task than retrieving one from weights — and it is a task these models are exceptionally good at, largely independent of whether the content was ever in their training data.
Follow the consequences and they are all in the same direction. The answer can carry a citation, because the passage it came from is a known object. The knowledge updates the moment the document does, with no training at all. Access control is possible, because retrieval can be scoped to what this user may see — something a trained-in fact cannot be, since the weights are the same for everyone. And when nothing relevant is retrieved, the system knows it: an empty result is a signal, and the model can be instructed to say it does not know.
Is retrieval-augmented generation therefore a solved technique? No, and the honest version of the picture matters. The system's ceiling is set by the retriever. If the right paragraph is not fetched, no amount of model quality recovers it — the model will answer from its own weights instead, often without any sign that it has done so. Chunking is a real problem: split the handbook into passages and you may sever a clause from the condition that governs it, and each chunk must carry enough context to be independently intelligible. Long contexts crowded with marginally relevant passages measurably degrade answer quality. And grounding is a strong tendency, not a guarantee: models do sometimes contradict the passage they were given, particularly when it conflicts with what they were trained to believe.
The analogy
THE ANALOGY #Think of an open-book examination against a closed-book one. The closed-book candidate has to have absorbed the material in advance, and when memory fails, a confident half-remembered answer is what comes out — with no way for the examiner to see where it came from. The open-book candidate need only find the right page and read carefully, can point to the passage, and works from this year's edition rather than whatever was memorised two years ago.
An open-book candidate can browse, follow a cross-reference, and notice that the index sent them to the wrong chapter, whereas the retrieval step happens once, before the model sees anything — so the model is handed a page it did not choose and cannot go back for another, which is why the retriever, not the reader, sets the ceiling.
Clarifying the model
THE MODEL #Three refinements.
First, retrieval and generation are separate systems with separate failure modes, and treating them as one blurs every diagnosis. When an answer is wrong, the first question is always whether the right passage was retrieved. If it was not, tuning the prompt is wasted effort; if it was, the retriever is fine and the reading step is at fault. Evaluating them separately — retrieval by recall, generation by faithfulness to the passages given — is the difference between engineering and guesswork.
Second, "retrieval" need not mean embedding search. It is whatever gets the right text into the context: a keyword index, a database query, a filter on document metadata, an API call, or several fused together. Vector search is popular because it handles paraphrase, but it is bad at exact identifiers and codes, and hybrid approaches are standard for good reason.
Third, correct a common overstatement. Retrieval-augmented generation does not eliminate hallucination. It changes it from unbounded invention into a bounded and checkable problem: with the source passages recorded, a wrong answer can be compared against what the model was given, and a claim with no support in them can be detected automatically. The value is not that the model stopped being able to make things up — it is that you can now tell when it has.
A picture of it
THE PICTURE #How to readFollow the numbered messages downward in time. Notice that the model is not consulted until step 4, and that everything it will say about the handbook arrived in that one message — there is no arrow from the index to the model, because the application, not the model, decides what it gets to see. The alternative branch at the bottom is the part that trained-in knowledge cannot offer: when retrieval comes back empty, the system has a fact about its own ignorance and can act on it, rather than producing a fluent guess.
What became clearer
WHAT CLEARED #Training a document into a model dissolves it — the text stops being an object with a location, a version, and an owner, and becomes an inclination spread across weights that cannot be cited, revised, or scoped to a reader. Retrieval keeps the document a document and changes the model's job from recall to reading, which is both easier and checkable. The trade is that the system's accuracy now rests on the retriever's ability to find the right passage, and on chunking it in a way that leaves each piece intelligible on its own — so the hard engineering moves out of the model and into the search, which is a much better place for it to be.
Where to go next
ONWARD #- How chunking strategy and passage overlap change what a retriever can find, and why parent-document retrieval exists.
- What faithfulness evaluation looks like: checking each claim in an answer against the passages that were supplied.
Key terms
TERMS #| Term | What it means |
|---|---|
| Retrieval-augmented generation | fetching relevant text at query time and supplying it to the model rather than training it in. |
| Grounding | constraining an answer to content actually present in the supplied passages. |
| Chunking | splitting documents into passages small enough to retrieve and large enough to stand alone. |
| Faithfulness | the degree to which an answer's claims are supported by the passages the model was given. |
| Hybrid retrieval | combining keyword and vector search, since each fails where the other succeeds. |
Every term the collection defines is gathered in the glossary.