Hard Limits, By Construction

Lesson 2 of 4 in Capabilities and Hard Limits.

Some LLM failures are bugs of the moment — a model too small, a prompt too vague, a missing tool. Those get fixed. This lesson is about the other kind: limits that follow from what the machine is. Each one below traces to a specific mechanical fact, which is exactly why no prompt phrasing, no retry, and no amount of politeness removes them. Knowing the difference is the most practical skill in this module: hard limits you design around; everything else you engineer away.

Limit one: there is no ground truth inside the machine. The model produces a probability for every token in the Vocabulary and samples a continuation. Nothing in that loop consults a fact store, because there is none — only Weights tuned to make training text likely. So when the weights hold a fact strongly (it recurred consistently across the corpus), the model states it correctly; when they hold it weakly, the model produces the most plausible-sounding completion instead, with identical fluency. That failure mode is called Hallucination — fluent confabulation, delivered in the same confident voice as everything else. What it means for your systems, and how injection and leakage compound it, is the Security & Risk domain’s territory; here, the point is mechanical: a plausibility engine cannot check itself against a truth it never stored.

Limit two: the model never sees your letters. Before anything reaches the model, a Tokenizer chops text into multi-character tokens. Ask how many times a letter appears in a word, and you are asking about characters inside chunks the model receives as opaque IDs — it can only answer from memorized spellings, and it often answers wrong. The same mechanism undermines arithmetic on long numbers, which fragment into uneven digit chunks that misalign place value. These feel like reasoning failures; they are input-representation failures. The full mechanics — and the debugging checklist — live in Tokenization, and you can watch words shatter yourself in the Tokenizer Playground.

Limit three: knowledge stops at the training cutoff. Training ends on a date; the Weights freeze; the model ships. Everything after that date simply is not in there — not blurry, not partial: absent. That boundary is the Knowledge cutoff. A deployed model does not learn from your questions either; each call runs the same frozen weights. When a model seems to know last week’s news, engineering put that text into the prompt — the weights did not change.

Limit four: the context window is a hard budget. The model attends over a finite number of tokens — the Context window. Whatever does not fit is not “deprioritized”; it is invisible, exactly as if it were never sent. And the budget is finite for a mechanical reason: Attention compares positions pairwise, so cost grows steeply with length — the bill for that is itemized in the Transformer domain’s production module.

Limit five: nothing persists between calls. The model is a pure function from input tokens to output probabilities. Call it twice with the same input and nothing carries over; there is no session inside the weights. Every “remembers our conversation” experience you have ever had with a chatbot is an application storing history and re-sending the relevant parts inside the context window — engineering around statelessness, never memory inside the model.

Interactive sorting exercise: Sort each item: is it a hard limit of the model itself — something no prompt, retry, or model update removes — or a symptom that engineers routinely fix with scaffolding around the model?

Key terms: Hallucination, Knowledge cutoff, Context window, Tokenizer, Sampling

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.