A context window is not memory

The short version: A context window is working memory — it holds what's in front of the model right now, then vanishes. Real memory persists across sessions and pulls back the right piece on demand. A bigger window gives the model more room to think, not the ability to remember.

When a lab announces a larger context window, the news usually gets read as "the model can remember more now." It can't. A context window and a memory are different mechanisms that solve different problems, and the difference is worth understanding precisely, because nearly every "the AI forgot" complaint traces back to treating one as the other.

Start with what a language model actually is at inference time: a fixed function. The weights are frozen. You pass in a sequence of tokens, the model runs a forward pass, and it returns a probability distribution over the next token. It writes nothing down. There is no variable that persists from one call to the next. The model that answers your second message is byte-for-byte identical to the one that answered your first, and it has no record that the first ever happened.

So why does a chat feel continuous? Because the application re-sends the entire conversation on every turn. When you type your fifth message, the system concatenates the system prompt, your four previous messages, the assistant's four previous replies, and your new message into one long string, and feeds the whole thing back in. The "memory" you're experiencing is the transcript being replayed into the window each time. The model isn't remembering the conversation; it's re-reading it from scratch, every single turn.

The context window is just the ceiling on how long that replayed string can be.

What the window costs

The window is not free space you can fill without consequence. Two mechanisms make it expensive, and both get worse as it grows.

The first is attention. A transformer works by having every token "attend to" every other token in the sequence — each token computes a relevance score against all the others to decide what to incorporate. That's what lets the model connect a pronoun to the noun it refers to forty words earlier. But the number of those pairwise scores grows with the square of the sequence length. Double the context and you roughly quadruple the attention computation. A 100,000-token prompt isn't ten times more work than a 10,000-token one; it's closer to a hundred times.

The second is the KV cache. When the model generates a response token by token, it caches intermediate values (the "keys" and "values") for every token already in the context so it doesn't recompute them for each new token it emits. That cache lives in GPU memory and grows linearly with context length. For long contexts it becomes large — gigabytes — and it competes with the model's own weights for the same scarce, expensive memory. This is a real operational ceiling, not a theoretical one: it's a major reason serving very long contexts costs what it does.

Put those together and "just use the biggest window" is a strategy that bills you for it.

Attention is not recall

Even setting cost aside, a fact being inside the window does not mean the model will use it.

The clearest demonstration is the "lost in the middle" effect, documented by Liu and colleagues in 2023. They placed a single relevant fact at different positions inside a long context and measured whether the model could find it. The result was a U-shaped curve: accuracy was high when the fact sat near the beginning or the very end of the input, and noticeably worse when it sat in the middle. The model attends unevenly. Information buried in the center of a long context is the most likely to be effectively ignored — not dropped, just under-weighted to the point of being missed.

This is also why "needle in a haystack" demos — where a model retrieves one planted sentence from a huge document — are weaker evidence than they look. Finding one distinctive sentence is a retrieval task, and it's the easy case. It tells you almost nothing about whether the model can reason over the whole context: hold ten facts from different sections at once, notice a contradiction between page 3 and page 80, or synthesize a conclusion that no single passage states. Benchmarks built to test that harder skill — RULER from NVIDIA is one — consistently find that a model's effective context length, the length at which it still performs well on real reasoning, is well short of the number on the box. A model advertised at 128,000 tokens often starts degrading on hard tasks long before it gets there.

So the window has two soft limits beneath its hard one: the point where cost becomes unreasonable, and the point where the model stops actually using what you put in. Both arrive before the advertised maximum.

The cost compounds over a conversation

Because the whole transcript is replayed every turn, the cost of a conversation doesn't grow with the length of your latest question — it grows with the length of everything said so far. Turn 50 pays to process the first 49 turns again. Across a long session the total work scales roughly with the square of the conversation length: each turn is linear in the history, and you have a history-length number of turns.

Prompt caching softens this. If the start of your prompt is identical to the previous request, providers can cache the processed form and skip re-computing it, charging a reduced rate to read it back. That genuinely helps. But it doesn't change the shape of the problem — you are still paying, on every turn, to carry forward the entire past. The window-as-memory approach means the cost of remembering rises without bound as the relationship with the user deepens, which is exactly backwards from what you want.

What memory actually is

The useful frame is the one computers have used for decades: working memory versus storage.

A memory has three properties a window doesn't. It persists — it survives the end of a session, a new chat tomorrow, a switch to a different model. It is selective — its whole purpose is that you don't load all of it at once; you retrieve the small relevant slice. And it requires retrieval — a separate step that decides, for this specific request, which few facts out of everything stored should move into the window.

That retrieval step is the actual hard part, and it's the part a bigger window does nothing to address. A larger window gives the model a roomier desk. It does not build the filing cabinet behind the desk, and it does not decide which folder to pull. Those remain entirely your problem.

What this means if you're building

If your plan for memory is "use a long-context model and paste in the history," you have bought time, not a solution, and you've bought it at a price that climbs with usage. It will look great in a demo, where the history is short. It will degrade in production, where real users accumulate months of context that is too large to replay, too expensive to replay every turn, and too unevenly attended-to to trust even when it fits.

The durable architecture treats the window as scarce and deliberate. Keep a real store of what's known. On each request, retrieve only the fraction that's relevant. Spend the window on reasoning, not on re-reading the user's entire past. Context length is one input to that system — a useful one, and a bigger one is genuinely better — but it was never the system itself.

So when a window doubles, be precise about what improved. You got a bigger desk. You still have to build the filing cabinet, and you still have to decide what to take out of it.