Lost in the Middle

Lesson 3 of 4 in Context Windows and Long Context.

The advertised window tells you how many tokens the model accepts. It says nothing about whether the model uses every position equally well. In 2023, Liu et al. tested exactly that with a beautifully simple design: give the model a stack of documents and a question whose answer sits in exactly one of them — then move that one document through the stack and watch what happens to accuracy.

For the models they tested (open and commercial, mid-2023), the reported result was a U-shaped curve: accuracy was highest when the relevant document sat at the very beginning or very end of the context, and dropped — sometimes far — when it sat in the middle. On a synthetic key–value retrieval task, several models showed the same signature. The paper also reported that performance tended to degrade as contexts grew longer overall, including for the explicitly long-context variants they evaluated: accepting more tokens and using them proved to be different capabilities.

Line chart, schematic only. X axis: position of the relevant document in the prompt, from 1st to 20th. Y axis: retrieval accuracy in percent. The line starts high around 75 percent at position 1, sags to a low near 52 percent around position 10, and rises again to about 70 percent at position 20, forming a U shape that illustrates weaker use of information placed in the middle of a long context.

Schematic of the U-shaped position effect reported by Liu et al. (2023): accuracy by position of the one relevant document among twenty. The curve is drawn to show the shape of the finding, not their measurements — the paper’s real curves differ by model, task, and context length, and newer models often show flatter ones. The only trustworthy curve is the one you measure on your own model. (illustrative — source: Liu et al. (2023) — Lost in the Middle, arXiv:2307.03172)

Until you have measured, position strategy is hypothesis management — and the hypotheses are cheap to act on. Put the instructions the model must not miss at the start, and the decisive document or the restated task at the end, nearest the answer. If one retrieved chunk matters most, don’t bury it at position 10 of 20 — many retrieval stacks order chunks best-first or best-last for exactly this reason. And when a long-context request misbehaves, add position to your suspect list: the fact the model “ignored” may simply have sat in a bad neighborhood.

Then replace the hypotheses with data. The probe below takes an afternoon: plant one verifiable fact at a controlled position among distractor text, sweep the position and the total length, and score exact-match accuracy. The output is the only curve that matters — your model’s, on your kind of documents.

A minimal position-sensitivity probe

You will be given {{N}} documents, then a question. Answer the question using only the documents. Reply with the answer alone.

{{DISTRACTOR_DOCUMENTS_BEFORE}}

Document {{K}}: {{DOCUMENT_CONTAINING_THE_PLANTED_FACT}}

{{DISTRACTOR_DOCUMENTS_AFTER}}

Question: {{QUESTION_ANSWERED_ONLY_BY_THE_PLANTED_FACT}}
Answer:

Run the identical fact and question with K swept across positions (first, quarter, middle, three-quarters, last) and at several total context lengths. Use distractors from your real document domain, score exact match, and plot accuracy against K per length. Flat curve: place freely. U-shaped or worse: order your retrieval output accordingly — and rerun the probe when you change models, because the curve belongs to the model, not to you.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.