Fine-tuning versus retrieval

Why does teaching a model your facts leave them stale, while teaching it your style does not?

Two teams fine-tune a model on the same corpus of company documents. One wanted answers to questions about the documents; six weeks later its answers are quietly wrong, because three policies changed. The other wanted the model to write like those documents — the house voice, the structure, the refusal to over-claim — and it still does, and will next year.

Same technique, same data, opposite durability. That asymmetry is the whole subject, and it is not about how well the training was done. Something about the two targets differs in kind. What?

Ask what each team was trying to install. The style team wanted a tendency: given any input, lean toward these sentence rhythms, this level of hedging, this shape of document. That tendency was present in every one of their training examples. Thousands of documents, one lesson, repeated thousands of times.

The facts team wanted something quite different: a specific association between a specific subject and a specific value. Clause 4.2 says fourteen days. That appears in one document, perhaps once. One example, one lesson, no repetition.

Now recall what training does. It nudges weights toward continuations that were observed, and each nudge is small and spread across a great many parameters. Which of these two lessons does that process suit? The style lesson is reinforced from every direction at once, so the small consistent nudges compound. The fact is a single point that must be memorised, competing against everything else those same weights encode — and models are notoriously inefficient at it: getting a fact to stick reliably typically requires seeing it many times, in many phrasings, or it fails to generalise. Berglund and colleagues documented an especially stark version in 2023, the reversal curse: a model trained that A is B often cannot answer the question with B as the subject, because what was installed was a directional continuation and not a symmetric fact.

So the first asymmetry: style is a distributional property with thousands of supporting examples; a fact is a lookup with one. Gradient descent is very good at the former and clumsy at the latter.

Now the durability question. Does style go stale? Ask what would make it stale — the house voice would have to change, which happens on the timescale of a rebrand. Facts go stale on the timescale of the business: a price, a policy, a personnel list, a version number. And critically, updating a fact in weights is not an edit. There is no address for it. You retrain, and the old association is still in there, competing; the model may now answer inconsistently depending on phrasing, which is worse than being simply out of date.

There is a further cost worth naming. Training hard on a narrow corpus degrades capabilities the model had before — catastrophic forgetting, described by McCloskey and Cohen in 1989 and still a live constraint. The parameters that would encode your policy are the ones holding general reasoning and language. Techniques like LoRA, which freeze the base model and train a small low-rank adapter, mitigate this considerably; they do not make the underlying trade disappear.

Put the pieces together and the rule falls out on its own. Retrieval is right where knowledge is discrete, volatile, verifiable, and owned — because it keeps the fact as an addressable object that can be updated, cited, and permissioned. Fine-tuning is right where the target is diffuse, stable, and demonstrated rather than stated — tone, format, domain vocabulary, task structure, an output schema. You cannot put "write like us" in a context window convincingly; twenty examples get you partway, and thousands of gradient steps get you further. And you cannot put a corpus that changes weekly into weights and expect it to stay true.

They are complements, and a system that needs both should use both: fine-tune the behaviour, retrieve the facts.

Three refinements.

First, fine-tuning genuinely can install facts — this is not a claim that it cannot. It is a claim about efficiency and maintenance. A fact needs substantial repetition and paraphrase to become reliable, generalises unpredictably to reformulations, cannot be cited, and cannot be revised without retraining. Retrieval achieves the same end with an edit to a document. The comparison is about cost of ownership, not capability.

Second, the reverse mistake is just as common: trying to get style out of retrieval. Stuffing twenty exemplar documents into the prompt is expensive on every call, crowds out the space the content needs, and imitates the voice more weakly than a fine-tune would. If the behaviour is required on every request, paying for it once in weights beats paying for it in tokens forever.

Third, the practical order of operations is worth stating plainly, because it is usually skipped. Start with prompting and retrieval, since they are cheap, immediately revisable, and diagnosable. Reach for fine-tuning when you have identified a behavioural gap that prompting cannot close and you have collected enough examples of the behaviour to demonstrate it. Fine-tuning to fix a knowledge gap is the single most common misapplication, and it fails in a particularly unhelpful way — not with an error, but with an answer that was right when it was trained.

• Why LoRA and other parameter-efficient methods reduce forgetting, and what they still cannot avoid.
• How to build an evaluation that separates a retrieval failure from a generation failure before either is blamed.

Every term the collection defines is gathered in the glossary.