The easy and hard problems of AI memory

The short version: Recalling a recent fact is the easy case, and it's nearly solved. The hard, open problems are reasoning across many sessions, tracking when things happened, honoring corrections, and admitting what it doesn't know — which is exactly what benchmarks like LongMemEval and LoCoMo test.

If you evaluate an AI's memory by telling it your name and asking for it back, it passes, and you've learned almost nothing. Recalling a single fact from recent context is the easy case — the one that gets easier for free every time context windows grow. The hard cases are what make long-term memory an open problem, and they're precisely the ones a quick test never reaches.

Two benchmarks were built to test the hard cases deliberately: LongMemEval and LoCoMo. They're a good lens for separating what's effectively solved from what isn't.

The easy problem, stated precisely

Within a single recent stretch of conversation, where the relevant fact is still inside the context window, retrieval is close to solved. State something a few turns ago and a capable model reads it back reliably. Most of what gets demoed as impressive memory is this: the information is physically present in the window, and the model is just attending to it.

This is real and worth acknowledging — but it's the case that improves automatically as windows get longer and attention gets better. It tells you nothing about the cases below, all of which involve information that is not sitting conveniently in the current window, and most of which involve reasoning rather than lookup.

The hard problems

Synthesis across sessions. The answer isn't in any one place — it's distributed across a conversation in January, another in March, and one last week, and producing it means gathering several pieces and combining them. Retrieval that finds a relevant chunk fails here; the system has to find all the relevant chunks and integrate them into a single answer. This is the multi-hop problem, and it's where naive top-k retrieval (grab the nearest few, hope they're enough) reliably comes up short.

Temporal reasoning. A memory isn't a bag of facts; it's a timeline. "What was I working on before the reorg?" "Which of these two did I mention first?" "How long have I been using this approach?" Answering correctly requires not just what was said but when, and the ability to reason about order and duration. Most storage schemes record facts and discard the temporal structure, which makes these questions unanswerable no matter how good retrieval is.

Knowledge updates — supersession. You said you were vegetarian. Six months later you mention you've started eating fish. A naive system now holds both statements and, asked about your diet, may surface either — including the stale one, presented with full confidence. Recognizing that a newer statement overrides an older one rather than coexisting with it is one of the most common and most damaging failures in practice, because the system gives no signal that it's wrong. It's the entire difference between a memory and a pile of every claim you've ever made. And it's genuinely hard: the system has to detect that two statements are about the same thing, determine which is newer, and decide whether the new one replaces, refines, or merely adds to the old — a judgment, not a lookup.

Abstention — knowing what you don't know. A robust memory has to recognize when the answer was never stored and say so, instead of confabulating something plausible. This is the hardest instinct to instill, because the incentives point the wrong way. To a user, a confident wrong answer and an honest "I don't have that" feel very different; to a naive accuracy metric, the wrong answer and the abstention can score identically as "not the gold answer," so optimizing for the metric can actively train the system to guess. A memory that always produces an answer is not a feature; it's a system with no concept of its own boundaries.

What the benchmarks measure

LongMemEval (introduced in 2024) is organized around exactly these abilities. It tests information extraction from long histories, multi-session reasoning, temporal reasoning, knowledge updates, and abstention, across roughly 500 carefully constructed questions. The histories are deliberately long enough that you can't simply paste everything into the context and let the model sort it out — the system has to genuinely store and retrieve. The headline finding was sobering: systems that looked strong on short interactions dropped substantially — by a wide margin — once memory had to span sessions. The benchmark's value is that it reports performance per ability, so you can see that a system is fine at extraction and falling over on temporal reasoning, rather than averaging the two into a number that hides both.

LoCoMo ("Long-term Conversational Memory," 2024) pushes on length and realism. Its conversations are very long — on the order of 300 turns and roughly 9,000 tokens each, spanning up to dozens of sessions — generated with persona-driven agents and then verified by humans so the long arcs stay coherent. The questions span single-hop recall, multi-hop reasoning, temporal questions, and open-domain knowledge, plus an event-summarization task. It's built to surface the degradation that only appears at scale, and it does: models lag human performance markedly, with the gap widest on exactly the temporal and multi-hop categories that the easy case never touches.

The shared message of both, and the single most useful thing to internalize: performance on a short conversation predicts almost nothing about performance on a long one. The curve bends downward as history accumulates, and where it bends is exactly where real users live after a few months of use. A system evaluated only on fresh conversations is being graded on the easy case and shipped into the hard one.

Why a single recall number misleads

A high recall score is easy to over-read, because two systems can post nearly identical recall and behave completely differently in the cases that actually hurt:

• One recalls facts but ignores that they were later corrected. High recall, fails supersession, confidently tells your new colleague you still work at your old job.
• One finds individual facts but never combines them. High recall on single-hop, fails multi-hop, can't answer anything that requires two facts at once.
• One answers everything, including the questions it has no basis to answer. High recall, zero abstention, fabricates fluently whenever it doesn't know.

This is why the credible evaluations decompose by ability rather than report one headline figure, and why an honest claim of improvement names the specific hard case it moved — temporal, supersession, multi-session synthesis — rather than an average that can quietly hide a regression in the case that matters most. "Our recall went up" is compatible with "we got worse at the thing users actually complain about."

The takeaway

The easy problem — recall from recent context — is largely handled by capable models and big windows, and it will keep getting handled for free. The hard problems — synthesizing across sessions, reasoning over time, honoring updates, and abstaining gracefully — are where memory systems genuinely differ, and they're what LongMemEval and LoCoMo are pointed at.

If you're evaluating an AI memory system, your test should be pointed there too. "Did it remember the fact?" is the wrong question — it's a test of the easy case. The question that separates a demo from something you can build on is: did it remember the fact, recognize that it was later superseded, place it correctly in time, and admit the things it has no record of? The first is a parlor trick. The rest is the actual problem.