What They Do Well
Lesson 1 of 4 in Capabilities and Hard Limits.
You know from the previous module that an Large language model (LLM) does exactly one thing: given the tokens so far, produce a probability for every possible next token, over and over. That sounds too narrow to be useful. The surprise of the past few years is how much fits inside it — and the pattern of what fits is not random. The tasks LLMs do best share one property: most of the answer is already sitting in the prompt.
Transformation is the cleanest case. Translate this paragraph, summarize this report, rewrite this email to be friendlier, convert this table to JSON. The information arrives in the input; the model’s job is to re-emit it in a different shape. Next-token prediction conditioned on your text is extremely good at this, because the continuation is tightly constrained — the model is reshaping, not inventing.
Drafting is the objective wearing work clothes. A first draft of an email, a project outline, a product description: “produce fluent text that plausibly continues from these instructions” is not an approximation of the training objective, it is the training objective. That is why drafting feels effortless — and why everything a draft asserts beyond your prompt needs checking.
Code looks like a different skill but is the same one on stricter text. Programming languages have rigid grammar, enormous public corpora, and conventions that repeat millions of times — ideal conditions for a predictor. Better still, code is checkable: it compiles or it does not, tests pass or fail. The strength is real; the verification loop is what makes it usable.
Extraction and classification flip the shape: the output is small and the input constrains it almost completely. Pull the invoice number, label the sentiment, route the ticket. These were whole product categories before LLMs; now one well-prompted model handles many of them at once.
Conversation and explanation ride on post-training. An Instruction-tuned model has been further trained on dialogue and helpful responses, so adjusting register — explain this to a new hire, now to an executive — is a text transformation it has seen countless versions of.
| Task class | Why next-token prediction fits | Watch out for |
|---|---|---|
Transformation — translate, summarize, rewrite | Nearly all the information is already in the prompt; the completion is tightly constrained by the text it must reshape | Silent omissions and shifted emphasis — a summary can be fluent and still drop the one number that mattered |
Drafting — emails, docs, outlines | Fluent continuation of instructions is the training objective, not an approximation of it | Every specific fact beyond your prompt is a prediction, not a lookup — treat unverified specifics as unchecked |
Code | Rigid grammar, huge public corpora, endlessly repeated conventions — and output you can verify by running it | Plausible-looking APIs that do not exist; correctness comes from tests and review, never from confidence |
Extraction & classification | Small output space, heavily constrained by the input: label it, route it, pull the fields | Edge cases that resemble nothing in training data; define categories precisely and measure on your own inputs |
Conversation & explanation | Instruction tuning trains directly on dialogue; changing register and level is a text transformation | Explanations are optimized to sound right — the confident tone carries no information about accuracy |
Run your eye down that middle column and the pattern condenses into a rule you can use before any benchmark: ask how much of the answer is in the prompt, and how much must come from the model’s Weights? Summarizing a document you provided sits at the safe end — the facts travel with the request. Answering a niche factual question from memory sits at the dangerous end — every specific is a prediction about what text usually says, sampled from a distribution, not retrieved from a store. The middle column is also why engineers push tasks toward the safe end deliberately: pasting the relevant policy into the prompt turns “recall our refund policy” into “transform this text”, and reliability jumps accordingly.
Why one objective buys so many skills
The economics of this list deserve a closer look, because none of these task classes was explicitly trained. The training signal was only “predict the next token” over an enormous corpus — yet translation, summarization, and classification fell out. Why?
Predicting text well is a harder job than it sounds. To keep Loss low across billions of documents, a model cannot memorize its way through; it must internalize the regularities that generate the text — grammar, style, facts that recur consistently, the structure of arguments, the format of a bug report followed by its fix. Skills are compressed into the Weights because they are the cheapest way to predict.
Brown et al. (2020) demonstrated the practical consequence with GPT-3: a large enough model performs new tasks from instructions and a few examples in the prompt, with no additional training — what the paper called few-shot, in-context learning. The examples do not teach the model anything new at inference time; they locate a capability it already compressed during training. Your prompt is less like programming and more like addressing: you are pointing into the library of behaviors prediction forced the model to build.
That framing predicts both halves of this module. Tasks whose regularities saturate the training distribution (transforming, drafting, coding) get strong entries in the library. Tasks the distribution never forces the model to master — checking truth, counting letters it cannot see, knowing events after training stopped — get no entry at all, no matter how politely you ask. Those are the next lesson.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.