Scale and the LLM Era
Lesson 4 of 4 in From N-grams to Transformers.
The transformer paper was about translation. Within about a year, two research lines pulled the architecture apart and kept different halves.
The GPT lineage kept the decoder — the half that generates. The recipe: pre-train a decoder-only Transformer on plain text with pure Next-token prediction, then adapt it to tasks. GPT (2018) showed the recipe worked. GPT-2 (2019) showed one such model could handle many tasks nobody explicitly trained it for. GPT-3 (Brown et al., 2020, arXiv:2005.14165) pushed the same recipe to 175 billion parameters and showed few-shot prompting: describe a task in the prompt, give a couple of examples, and the model performs it with no retraining at all. Abilities nobody targeted — translation, arithmetic, rudimentary code — surfaced as side effects of scale, the pattern since named Emergence. Every chat assistant you have used descends from this branch.
The BERT branch kept the encoder — the half that reads. BERT (Devlin et al., 2018, arXiv:1810.04805) trains by masking words in the middle of sentences and predicting them from both directions at once. That makes it a superb reader — it swept the era’s understanding benchmarks — and a non-writer: there is no natural way for it to generate text. Encoder models never became chatbots; they became infrastructure. Search ranking, classification, and the Embedding models behind retrieval systems are where this branch still quietly runs.
| GPT lineage (decoder-only) | BERT branch (encoder-only) | |
|---|---|---|
Training objective | Predict the next token, left to right | Predict masked tokens using both directions |
Native strength | Generating text, one token at a time | Reading and representing text it is given |
Can it chat? | Yes — generation is its native act | Not natively — it has no generation loop |
Where it lives today | Chat assistants, code tools, LLM APIs | Search, classification, embeddings for retrieval |
Why did the generative branch keep growing? Because scaling stopped being a gamble and became a strategy: Kaplan et al. (2020, arXiv:2001.08361) measured Loss falling smoothly and predictably as model size, data, and compute grow — spend more, reliably get better. That is the one-line version; the Pre-training domain owns what those curves really say and how they were later revised.
A scaled-up Base model is still just a text predictor — prompt it with a question and it may continue with more questions, because that is a statistically plausible continuation. The step that made assistants: fine-tune on demonstrations of helpful behavior, then optimize against human preference judgments — instruction tuning plus RLHF (Ouyang et al., 2022, arXiv:2203.02155), which in one line means make the model prefer the responses people prefer. That turns a Base model into an Instruction-tuned model; the Post-training & Alignment domain tells that story properly.
On November 30, 2022, OpenAI released ChatGPT as a free research preview, and the public met the instruction-tuned transformer in a chat box. The architecture was not new that day — the tuning, the packaging, and the interface were what changed. Adoption exploded, and the industry reorganized around the result.
One more fork completes today’s map. In February 2023 Meta released LLaMA — transformer Weights researchers could download and run — and in July 2023, Llama 2, under a license broad enough for most commercial use. Open-weights models put the artifact itself in your hands: run it on your own hardware, fine-tune it, inspect it. Closed-weights models stay behind APIs. Both descend from the same 2017 architecture; choosing between them is now a routine engineering decision, and the Model Landscape domain treats it in depth.
The scale era
- 2019-02-14 — GPT-2 and staged release:
A 1.5B-parameter GPT trained on web text showed surprising zero-shot ability across tasks, and OpenAI staged its release over misuse concerns. It was the first clear signal that scaling one recipe broadens capability — and the first mainstream release-policy debate.
- 2020-05-01 — GPT-3: in-context learning:
At 175B parameters, GPT-3 exposed a new interface: in-context learning, where the model performs tasks from instructions and a few examples in the prompt, with no gradient updates. Prompting began to replace fine-tuning as the primary way to program a model.
- 2022-11-30 — ChatGPT launches:
A free chat interface over an RLHF-tuned GPT-3.5 model reached an estimated 100 million users within two months. It was the moment LLMs became a consumer product, and it reset the industry’s research and deployment agenda around assistants.
- 2023-02-01 — LLaMA: the open-weights era begins:
Meta trained 7B–65B models Chinchilla-style on trillions of tokens and released the weights to researchers; the 13B model rivaled GPT-3. The weights spread far beyond the research program and ignited the open-weights ecosystem of fine-tunes and local inference.
- 2023-03-14 — GPT-4:
A multimodal model (text and image input) with large capability jumps — top-decile bar-exam performance among them — whose technical report disclosed neither architecture nor training data. It marked both the new capability frontier and the industry’s turn away from openness.
- 2023-07-18 — Llama 2: open weights go commercial:
Meta released retrained 7B–70B models plus RLHF-tuned chat variants under a license permitting most commercial use. Open weights became a viable production choice rather than a research artifact, anchoring the open-versus-closed split that defines the current landscape.
In production
The model you rent from a cloud endpoint today is the survivor of this whole history — and the reasons it won are the reasons the infrastructure around it looks the way it does.
AWS
The transformer’s parallel training is why accelerator fleets became cloud strategy: an architecture whose training saturates thousands of chips at once turned scale into something you purchase rather than a research bet. Amazon Bedrock’s catalog is overwhelmingly decoder-only descendants of the GPT branch, and they are served the way the lineage dictates — generation still happens one token at a time, which is why output tokens, not requests, are what you meter.
Azure
Azure AI Foundry’s model catalog is this module’s family tree turned into a menu: generative decoder models for chat and code sit beside encoder-style embedding models for search and retrieval — the GPT and BERT branches, both still earning their keep. Quotas denominated in tokens per minute reflect the same asymmetry the architecture created: training parallelized, generation did not.
Google Cloud
Google’s TPU line predates the transformer, but the architecture repaid the investment: dense matrix multiplication with no serial dependency across positions is exactly what such accelerators are built to feast on. Vertex AI serves both branches — generative models alongside embedding models — and its per-token economics trace directly back to the decoder’s one-token-at-a-time loop.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.