Why Models Confabulate

Lesson 1 of 4 in Hallucination.

Everything this site has taught so far converges on one uncomfortable conclusion, and this lesson assembles it.

Start with the objective. An Large language model (LLM) is trained on Next-token prediction: given the tokens so far, score every Token in the vocabulary by how plausibly it comes next (the foundations module built this up in detail). Plausible is doing all the work in that sentence. The loss rewards continuations that look like the training distribution — and true statements are merely one highly represented kind of plausible text. Where the model’s weights encode a fact well, the most plausible continuation and the true one coincide, and the model looks like it “knows.” Where they don’t — a niche API, a person with three search results to their name, an event after the Knowledge cutoff — the model still owes the decoder a probability for every token. Something will be emitted. The something that scores highest is whatever resembles a correct answer: a fluent, well-formatted, confident fabrication.

The word hallucination stuck, but confabulation is the better clinical metaphor: not perceiving something false, but fluently filling a gap in memory without any signal that a gap was filled. Nothing malfunctioned. The machine that completes text completed text.

Key terms: Hallucination, Next-token prediction, Grounding, Abstention, Sampling

From objective to confabulation

  1. Objective: predict the next token

    Pre-training optimizes plausibility under the training distribution. Truth is correlated with plausibility, never identical to it.

  2. Is the fact well-encoded in the weights?

    Facts seen often and consistently in training data are encoded reliably; rare, contested, or post-cutoff facts are weakly encoded or absent.

  3. Plausible ≈ true

    The highest-probability continuation happens to be the correct one. The output looks like knowledge — and here, it effectively is.

  4. Plausible ≠ true, but tokens must flow

    The model has no null output. Every step ends in a probability over the vocabulary, and decoding picks from it — over a gap, high-probability text is “what an answer would look like.”

  5. Post-training adds confident form

    SFT teaches the assistant style — direct, structured, assured — everywhere, including over gaps. The delivery does not vary with the reliability of the content.

  6. No fact/guess flag anywhere

    No token in the output stream is marked “retrieved” vs “improvised.” The reader gets fluency as the only signal, and fluency is uniform.

  7. Confident confabulation

    A well-formed, assured, false answer — indistinguishable in style from a correct one.

Post-training makes the problem presentable. Supervised fine-tuning teaches the model the form of a good answer — direct, organized, confident — by imitating demonstrations (the SFT module covers the recipe and its limits of imitation). But demonstrations teach style everywhere at once: the model learns to sound authoritative whether or not the underlying knowledge is there. An Instruction-tuned model that would once have trailed off into incoherent Base model text now wraps the same missing knowledge in clean prose with headers and a summary. The confidence you read is a property of the format the model was tuned toward, not a report on its epistemic state.

And there is no epistemic state to report. At no point in the forward pass does the model consult a store of verified facts and fall back to generation when the lookup misses — there is no lookup. Recall of a memorized fact and improvisation of a plausible one are the same computation: attention and matrix multiplies producing a distribution over next tokens. That is why no reliable “hallucination bit” exists to surface, and why every mitigation in lesson three works around the model — changing its inputs, its training, or checking its outputs — rather than reading a flag inside it.

Why “I don’t know” is so hard to train in

If the mechanism is a knowledge gap plus mandatory output, the obvious fix is teaching the model to say “I don’t know.” That turns out to be genuinely hard, for reasons worth understanding before lesson three hedges about Abstention.

The incentive problem. Training and evaluation mostly reward answering. In pre-training, refusing to continue text is simply never an option the loss contemplates. In post-training, demonstrations and preference data reward being helpful, and a benchmark scored on accuracy treats an abstention exactly like a wrong answer — so, like a test-taker who never leaves a multiple-choice question blank, a model tuned toward those rewards is pushed to guess. Researchers have argued this scoring asymmetry is a core reason confident guessing persists; the framing is discussed across the literature, though how much of the observed rate it explains remains debated.

The self-knowledge problem. Abstaining well requires the model’s stated uncertainty to track its actual error rate — calibration. Reported behavior here is mixed and model-specific: token-level probabilities can be informative in narrow setups, but verbalized confidence (“I’m quite sure…”) is commonly reported to be poorly calibrated, and post-training is often observed to distort whatever calibration pre-training left behind. Treat any claim that a model “knows what it doesn’t know” as an empirical question about a specific model on your task — measurable with the tools in lesson four, not assumable.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.