What Actually Changes

Lesson 3 of 3 in From Base Model to Assistant.

Anchor the whole domain on one contrast: post-training is small. Pre-training consumes hundreds of billions to trillions of tokens. The InstructGPT pipeline, by contrast, ran on datasets measured in tens of thousands of prompts across its stages (Ouyang et al. 2022) — orders of magnitude less data, and correspondingly little compute. Whatever post-training does, it does not have the budget to re-teach the model the world.

What that little data can do is redistribute. The base model already contains helpful answers, refusals, personas, and formats as low-priority continuations somewhere in its distribution; post-training moves probability mass onto them. That is why this module keeps saying behavior shaping, not knowledge injection: facts, languages, and skills come overwhelmingly from pre-training, and post-training decides which of the model’s existing behaviors show up by default.

Three families of change account for most of what you experience as “the assistant”:

  • Format. The model speaks in turns under its Chat template, defaults to markdown habits and characteristic lengths, and stops when the answer is done instead of running to the token limit.
  • Refusal. A base model declines nothing — it completes whatever text you start. A Refusal is a trained behavior with trained boundaries, put there by safety data in the SFT and preference stages. Where those boundaries sit is a property of the recipe, not of the architecture.
  • Persona. The stable “I am an AI assistant” voice, tone, and self-description — including how strongly the model lets a System prompt steer it.

None of this is free. Ouyang et al. documented what they called an Alignment tax: during RLHF fine-tuning, performance on some public NLP benchmarks regressed, and they mixed pre-training gradients into the RL updates to claw much of it back. How large the trade-off is for any given model and task is an empirical question — measure it rather than assume it — but the idea that shaping behavior can cost some raw capability is documented, not folklore.

One scoping line worth pinning before the drill: post-training, as this domain teaches it, is what labs do to shape general behavior for everyone. Making a model good at your data and your domain — fine-tuning on your corpus, retrieval, prompting strategy — is a different discipline with different tools: the Adapting LLMs domain.

Interactive sorting exercise: Where does each behavior or property primarily come from? Sort each card into the stage that creates it.

Why so little data can change so much

An SFT set of tens of thousands of dialogues cannot compete with trillions of pre-training tokens for gradient volume — and it does not need to, because it is not fighting the pre-trained distribution, it is selecting from it. LIMA (Zhou et al. 2023, arXiv:2305.11206) made the point sharply: fine-tuning a 65B base model on just 1,000 carefully curated prompt–response pairs produced a usable assistant. The authors’ reading — their “superficial alignment hypothesis” — is that almost all capability is learned in pre-training, and post-training chiefly teaches the format and style with which to surface it. Treat it as a hypothesis with striking evidence, not settled law: later modules show where imitation alone runs out (hallucinated confidence, no notion of better-vs-worse) and why the preference and RL stages still earn their compute.

The methods are also built to keep the change small. RLHF as published penalizes divergence from a reference model — the KL penalty — so optimization concentrates existing mass instead of wandering to new distributions; DPO bakes the same reference-anchored structure directly into its loss. The engineering consequence you should carry out of this module: the base distribution is still in there. Post-training lowers the probability of unwanted continuations; it does not delete them. That is one mechanism behind jailbreaks, and the reason this domain’s capstone treats Alignment as a training outcome to be tested, never a guarantee to be assumed.

In production

You will probably never touch a base model. Every model behind a hosted endpoint arrives post-trained, and its behavior profile — format habits, refusal boundaries, default persona — is as much a part of the product as its raw capability.

AWS

Amazon Bedrock puts models from many providers behind one API, and each carries its own post-training. Swapping models swaps behavior profiles, not just quality and price: default formats shift, refusal boundaries move, personas change — even between models with similar benchmark scores. Platform guardrails you configure stack on top of the trained-in behavior; they never replace it, so behavioral evaluation on your own traffic belongs in every model-selection decision.

Azure

Azure AI Foundry makes the layering explicit: the model’s trained-in behavior underneath, the service’s configurable content filters on top. When an output is blocked or refused, debugging starts by identifying which layer acted — a trained refusal reads differently and responds to different fixes than a platform filter, and only the platform layer is under your control.

Google Cloud

Vertex AI’s Model Garden gathers models with very different post-training histories behind one surface. The behavior profile is a selection criterion alongside cost and capability: two models that score alike can differ sharply in refusal patterns, verbosity, and formatting defaults, and only an evaluation on your own prompts reveals which profile your product inherits.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.