The Post-training Map

Lesson 2 of 3 in From Base Model to Assistant.

Post-training is not one technique. It is a short pipeline of them, run in sequence on top of the base Checkpoint, and the pipeline’s shape has been remarkably stable since it was first published at scale. The canonical reference is InstructGPT (Ouyang et al. 2022, arXiv:2203.02155): take GPT-3, apply supervised fine-tuning on human-written demonstrations, train a Reward model on human comparisons of model outputs, then optimize the model against that reward with reinforcement learning. Three stages — demonstrate, compare, optimize — and the result was striking enough to reset the field: labelers preferred the outputs of a 1.3B-parameter InstructGPT model over the raw 175B GPT-3. A hundred times fewer parameters, preferred anyway, because it did what people actually asked.

The modern map keeps that skeleton and adds one newer stage. Direct methods like DPO (Rafailov et al. 2023) now often replace the reward-model-plus-RL machinery with a single loss on preference pairs — same signal, less apparatus. And for domains where answers can be checked by a machine — mathematics, code — labs have added RL with verifiable rewards, the openly documented recipe behind reasoning models such as DeepSeek-R1 (DeepSeek-AI 2025). Here is the whole map.

The post-training map: base model to released assistant

  1. Base model

    The pre-trained next-token predictor: broad capability, document-completion behavior, no dialogue interface. Everything downstream reshapes this checkpoint.

  2. Supervised fine-tuning (SFT)

    Train on curated demonstrations of good assistant behavior, rendered in a chat template with role markers. Contributes the interface: turns, format, answering instead of continuing.

  3. Preference optimization

    Collect comparisons between candidate responses, then optimize toward the preferred ones — via a reward model plus RL (RLHF) or directly on the pairs (DPO). Contributes judgment: better versus worse, beyond what one demonstration can express.

  4. Machine-checkable tasks in scope?

    Math, code, and other domains where a checker — unit tests, exact answers — can grade outputs automatically, without a human or a learned reward model.

  5. RL with verifiable rewards

    Optimize directly against automatic checkers. The published recipe behind reasoning models that learn to spend more tokens thinking before answering (e.g. DeepSeek-R1, 2025).

  6. Released assistant

    A checkpoint whose behavior profile — format, tone, refusal boundaries, persona — is the accumulated result of every stage. Safety training is woven through the stages, not bolted on at the end.

One honest sentence on what each stage contributes — the following modules expand each into its own machinery.

  • SFT contributes the interface: after it, the model speaks in turns, follows its Chat template, and answers the question instead of extending it.
  • Preference optimization contributes judgment: a demonstration can only show one acceptable answer, while comparisons teach which of two fluent answers people prefer — the signal that preference-based RL introduced (Christiano et al. 2017) and InstructGPT scaled to language models, whether optimized through a Reward model (RLHF) or directly on preference pairs (DPO).
  • RL with verifiable rewards contributes reasoning stamina where a checker can grade the answer — and only there; it buys accuracy on math and code at the price of longer, costlier outputs, as the Reasoning model module will quantify.
  • Safety training is deliberately not a box on the diagram, because it lives inside the others: Refusal demonstrations in the SFT data, harmlessness comparisons in the preference data, and — in some published recipes — AI-generated critiques guided by an explicit set of principles (constitutional AI; Bai et al. 2022).

When assistants arrived

  • 2022-03-01 — InstructGPT: RLHF at scale:

    Ouyang et al. applied reinforcement learning from human feedback at scale to make GPT-3 follow instructions; labelers preferred a 1.3B aligned model over the raw 175B one. It established RLHF as the standard post-training recipe that turns a text predictor into an assistant.

  • 2022-11-30 — ChatGPT launches:

    A free chat interface over an RLHF-tuned GPT-3.5 model reached an estimated 100 million users within two months. It was the moment LLMs became a consumer product, and it reset the industry’s research and deployment agenda around assistants.

  • 2023-07-18 — Llama 2: open weights go commercial:

    Meta released retrained 7B–70B models plus RLHF-tuned chat variants under a license permitting most commercial use. Open weights became a viable production choice rather than a research artifact, anchoring the open-versus-closed split that defines the current landscape.

Key terms: Supervised fine-tuning (SFT), Reinforcement learning from human feedback (RLHF), Direct preference optimization (DPO), Reward model, Preference data, Alignment

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.