Why models hallucinate
A language model has no separate sense of what's true. It predicts plausible text. Truth and plausibility usually overlap — until they don't, and then you get a confident, fluent, wrong answer. The mechanism, not the hand-wave.
By Jackdaw Team
The short version: A model predicts plausible text, not true text — it has no separate sense of what's true. A hallucination is the same machinery that produces every correct answer, running on a question where plausible and true happen to diverge.
"Hallucination" is a bad word for the thing it describes, because it implies a glitch — a system briefly malfunctioning. The reality is less comfortable: a hallucination is the model working exactly as designed, on a question where its design and the truth point in different directions. Understanding why means understanding what the model is actually optimizing for, which is not what most people assume.
The model predicts plausible text, not true text
A language model is trained to do one thing: given a sequence of tokens, predict the next one. It learns this from a vast amount of human text, and what it absorbs is the statistical shape of language — which words tend to follow which, what a well-formed answer looks like, how a citation is structured, how a confident explanation flows. At no point does it learn a separate function for "is this true?" There is no truth signal in next-token prediction. There is only "what is a likely continuation."
Most of the time this is fine, because in the training data, true statements are more common than any particular false one. "The capital of France is ___" is followed by "Paris" far more often than by anything else, so "Paris" is both the plausible completion and the correct one. Truth and plausibility overlap, and the model looks like it knows facts.
But it doesn't know facts. It knows likelihoods. And when plausibility and truth come apart — when the most natural-sounding continuation isn't the true one — the model follows plausibility, every time, because that's the only thing it's tracking. A made-up court case with a realistic name and citation format is an extremely plausible continuation of "a relevant precedent is." A function that sounds like it should exist in a library is a plausible continuation of "you can call." The model generates these for the same reason it generates correct answers: they fit the pattern. It has no way to notice that one of them corresponds to reality and the other doesn't.
This is why the framing matters. The model is not sometimes-retrieving-facts and sometimes-malfunctioning. It is always generating the most plausible-sounding text, and a hallucination is just an instance where plausible-sounding and true diverged. The same mechanism produced your correct answer a moment ago.
Why it's so confident
The thing that makes hallucinations dangerous is that they arrive in the same fluent, assured prose as everything else. A wrong citation isn't hedged or shaky; it's formatted perfectly and stated plainly.
That follows from the mechanism too. The model generates fluent text whether or not the content is grounded, because fluency is what it learned — the surface form of a confident answer is independent of whether the facts underneath are real. There is some uncertainty information inside the model (the probabilities it assigns to each next token), but that signal is weak, poorly calibrated for factual correctness, and mostly invisible in the final text. The model doesn't have a reliable internal "I'm not sure about this" that it surfaces to you. So you get the same tone for "Paris" and for a fabricated case name.
Why training can make it worse
There's a subtler contributor that recent research has put a name to: the way models are trained and evaluated often rewards guessing. Think of a multiple-choice exam where wrong answers and blank answers both score zero. A student who guesses will, on average, beat a student who leaves hard questions blank. Standard benchmarks frequently work the same way — a confident wrong answer and an honest "I don't know" are both simply "not the right answer," scored identically — so the training process learns that producing an answer is better than abstaining. The incentive structure quietly teaches the model that bluffing is optimal.
That reframes the problem usefully. Some hallucination is intrinsic to predicting plausible text. But some of it is a learned behavior we accidentally selected for, by grading models in a way that punishes saying "I'm not sure." Which means part of the fix isn't a better model — it's better incentives: evaluations and training that reward calibrated uncertainty and stop treating an honest abstention as equivalent to a confident error.
Why retrieval helps (and doesn't fully solve it)
The most effective practical mitigation is grounding: don't ask the model to recall a fact from its parameters, hand it the source text and ask it to answer from that. This is what retrieval-augmented generation does. It works because it changes the task from "recall whether this is true" — which the model can't do — to "read this passage and report what it says," which it's good at. The plausible continuation is now anchored to provided text rather than to the model's diffuse statistical memory.
But it's a reduction, not a cure, for two reasons. First, the model can still hallucinate over the provided context — misread it, blend it with its parametric memory, or confidently answer a question the passage doesn't actually address. Second, grounding only helps if retrieval surfaced the right text in the first place; give the model the wrong passage and it will fluently answer from the wrong passage. Grounding moves the failure point from the model to the retrieval layer. That's a real improvement — the retrieval layer is something you can inspect and fix — but the underlying tendency to produce plausible text regardless of truth is still there underneath.
What this means
Three things follow if you take the mechanism seriously.
Stop expecting the model to know what it doesn't know. It has no reliable internal truth meter. Designing around AI means building the verification outside the model — grounding, citations you can check, a second pass, a human gate on anything consequential — rather than hoping the model will flag its own errors. It mostly can't.
Treat fluency as zero evidence of accuracy. The confidence in the prose is a property of the language, not of the facts. A perfectly formatted citation and a real one are indistinguishable from the inside; the only way to tell them apart is to look the thing up.
Match the trust to the task. For tasks where plausible and true almost always coincide — rephrasing, summarizing text you provided, drafting from given material — hallucination is rare and low-stakes. For tasks that ride on specific facts the model has to recall — names, numbers, citations, APIs, case law — assume it will sometimes invent them, and build the check. The word "hallucination" makes this sound like a rare edge case. It's better understood as the normal behavior of a plausibility engine, showing through wherever plausibility and truth happen to part ways.