Where a fact lives in a model

If a model holds no table of facts, where is anything it knows actually stored?

A model will tell you that the Eiffel Tower is in Paris. Open the file it was loaded from and you find no such sentence, no row in a table, no index. There are only numbers — billions of them, arranged in matrices, and not one of them is labelled "Paris".

Two reactions are available and both are wrong: that the model must secretly contain the text, or that it does not really know anything and merely guesses. The interesting position is between them — something is stored, permanently, in an arrangement with no place to put a fact. Where did it go?

Start from the compression angle, because it is the honest one. Training data runs to trillions of tokens; the model's parameters number in the billions. The model is roughly a hundred times smaller than what it read. Verbatim storage is arithmetically impossible for the bulk of it. So whatever is retained has to be retained in a form that is shared — structure that pays for itself across many examples.

Now ask where in the architecture a fact could plausibly sit. A transformer layer has two halves. Attention moves information between positions; it is about routing. The feed-forward block does not look at other positions at all — it takes one position's vector and transforms it. That is the half with nowhere to look things up from, so it is the half that must contain whatever it contributes. It is also the larger half: in a standard layer the feed-forward weights outnumber the attention weights roughly two to one.

There is a neat way to read that block. Its first matrix compares the incoming vector against many stored directions; its second matrix, for each of those, adds a corresponding stored direction back into the stream. That is exactly the shape of a lookup: match against a pattern, retrieve an associated payload. Geva and colleagues made this argument in 2021, calling feed-forward layers key-value memories.

So we have a candidate: the Eiffel Tower fact lives in feed-forward weights that recognise something like "the subject is the Eiffel Tower and a location is being asked for" and push the representation toward Paris. Meng and colleagues gave this a strong empirical test in 2022 with causal tracing — corrupting the subject token, then restoring individual hidden states to see which restoration revives the correct answer. The signal concentrated in mid-layer feed-forward blocks at the subject's final token. They then edited a single such block and changed the model's belief about where the tower stands.

That looks like a clean answer, and here I have to slow down, because the follow-up work complicated it. Hase and colleagues showed in 2023 that where a fact can be localised does not predict where it can successfully be edited — you can change the model's answer by editing layers that causal tracing did not implicate at all. So localisation and modifiability come apart, and any picture in which the fact sits at one address is too tidy.

Push on the arithmetic and you see why. Even granting that a fact is concentrated in a few blocks, those blocks hold thousands of dimensions and every parameter in them participates in enormously many other facts. There is no subset you could excise to remove Paris while leaving the rest intact. This is what distributed representation means, and it has a direct consequence: a fact is not a stored item but a stable disposition of the whole — a direction the computation reliably takes when a particular pattern arrives.

One more consequence worth drawing. If facts are dispositions rather than entries, then there is no boundary between knowing something and merely being inclined toward it. The same machinery that reliably produces "Paris" produces a confident wrong answer for an obscure name, with nothing internally marking one as retrieved and the other as invented. Hallucination is not a separate failure mode bolted onto a lookup system; it is the same mechanism operating where the training signal was thin.

Three refinements.

First, "in the feed-forward layers" is a claim about weight count and about where causal signal concentrates, not a claim that attention is uninvolved. Attention is what assembles "the Eiffel Tower" from its tokens and carries the subject to the position where the question is being answered. Without that routing, the stored association would never be queried. Retrieval needs both halves.

Second, the token embedding table is a genuine lookup and worth distinguishing. Each token really does have its own row, so a fact about a word — roughly what company it keeps — is stored in something table-shaped. But it is a small fraction of a large model's parameters, and it is not where relational facts about the world sit.

Third, correct a common conclusion. Having no fact table does not make a model's knowledge vague. Facts seen often in training are held very robustly — reproduced across paraphrases, languages, and contexts, which is more than a lookup table would give you. What is missing is not reliability but addressability: no way to enumerate what is known, check it against a source, or update one entry without touching the rest.

• How model editing attempts a surgical change, and why an edited fact often fails to propagate.
• Why a model can state a fact one way and fail to state its reverse.

Every term the collection defines is gathered in the glossary.