Context window

Why does a model with no memory between messages appear to remember the whole conversation?

You are told, correctly, that the model keeps nothing between your messages. Its weights do not change while you chat; when your request finishes, nothing about you persists inside it. And yet you can say "make it shorter" and it knows what it is, or refer back to a name you mentioned twenty exchanges ago.

Both of those cannot be true of the same thing — unless the remembering is happening somewhere other than where you assumed. So where would you put it, if you had to build this and were forbidden from letting the model retain anything?

Think about what the model actually is between requests: a large fixed function. Text goes in, a probability distribution over the next token comes out. Same input, same weights, same output — give or take some floating-point and batching noise that makes strict determinism hard to guarantee in practice, but the shape of the claim holds. A pure function has nowhere to hide a memory.

So if it cannot remember, and it nevertheless behaves as though it does, the only possibility left is that the memory is being handed to it each time. And that is exactly what happens. When you send your fourth message, the application does not send four words; it sends the entire transcript — your first message, the model's first reply, your second, its second, and so on, usually with a system prompt in front — and asks for a continuation of that whole document.

Sit with the consequence for a moment, because it is not intuitive. The model does not experience a conversation at all. It experiences a single long document that happens to be formatted as a script with two named speakers, and it is asked to write the next line. The illusion of continuity is produced entirely by re-reading. Nothing was retained; everything was resupplied.

Now, does that fully account for the feeling of memory? Ask yourself what would break. If the transcript is what carries the state, then anything not in the transcript cannot be known — and that matches experience precisely. Start a new chat and the model has no idea who you are. Tell it something and delete that message from the history, and the knowledge vanishes with it. The memory has exactly the extent of the text being resent, which is the signature of state living in the input rather than in the system.

That immediately raises the limit. The input is finite: the context window is the maximum number of tokens the model can attend to in one request, counting the system prompt, the entire transcript, any attached documents, and the reply being generated. Windows have grown enormously — from a couple of thousand tokens in early deployed models to hundreds of thousands, and in some cases a million or more — but the limit never goes away, and there are real reasons it cannot simply be set to infinity. Standard attention has every token attend to every other, so cost grows roughly with the square of the length; the key-value cache that makes generation fast grows with it too, occupying accelerator memory. Long contexts are expensive in both money and latency, which is why you are charged per input token on every turn, and why a long chat costs more per message than a short one even when your messages are the same size.

What happens when a conversation outgrows the window? The model does not gracefully forget — it cannot accept the request at all. So the application intervenes, and this is where most of the confusion about "memory" lives. It might drop the oldest turns, or summarize the earlier part of the conversation into a compact note and send that instead, or store facts in an external database and retrieve the relevant few. Long-term memory features in chat products are built this way: not a change inside the model, but a retrieval step that puts selected text into the window before the model runs. Recognizing that distinction is most of what makes these systems predictable — when an assistant "forgets" something, the useful question is almost never about the model, it is about what did or did not make it into the context this turn.

Three refinements worth keeping. First, the window is shared. A long system prompt, a pasted document, and a long reply all consume the same budget, so "the window is huge" does not mean the transcript may be huge.

Second, fitting inside the window and being used well are different claims. A fact can sit comfortably within the limit and still be relied on less than one placed at the edges of the input — the effect usually called being lost in the middle. Capacity is a precondition for attention, not a guarantee of it.

Third, the key-value cache is worth not misunderstanding. It exists so that a long prompt need not be recomputed from scratch for every generated token, and providers may keep it warm briefly across requests to cut cost. That is a performance optimization over the same resent text — it does not give the model memory of anything you did not send.

And a small correction to the usual mental picture: nothing is being "forgotten" when a chat gets long. Forgetting implies it was held and lost. It was never held.

• Why quadratic attention cost is the binding constraint, and what sparse or linear attention schemes trade away to escape it.
• How retrieval-augmented generation turns an unbounded corpus into a few thousand tokens that fit.
• Why a fact in the middle of a long context is used less reliably than one at either end.

Every term the collection defines is gathered in the glossary.