The Training Recipe

Lesson 2 of 4 in Supervised Fine-Tuning.

The surprise of the SFT recipe is how little of it is new. Same architecture, same Cross-entropy Loss on next-token predictions, same Teacher forcing setup where the target at every position is simply the next Token of the reference text. What changes is the data and the scale: instead of trillions of tokens of web text, a curated set of conversations — thousands to a few million examples — trained for a small number of epochs at a gentle Learning rate, so the model adjusts its behavior without trampling what pre-training built.

Two details carry all the novelty. First, conversations have to become token sequences at all. Second, you do not want the loss applied to every token in them.

Chat templates make the conversation structure literal. A transformer eats one flat token sequence — it has no native concept of “turns”. So a Chat template serializes the conversation: special tokens from the tokenizer’s Vocabulary mark where the System prompt ends, where the user speaks, where the assistant speaks, and — critically — where the assistant’s turn ends. Those delimiters are ordinary token IDs, learned like any others (the mechanics live in Tokenization). The model learns the conversation format the same way it learned English: by predicting it. That includes learning to emit the end-of-turn token when an answer is complete — which is exactly what your inference server watches for to know when to stop generating.

A single-row heatmap over a simplified templated conversation. The columns are tokens: a system marker, a system instruction, a user marker, the user question "Explain DNS briefly", an assistant marker, then the assistant response tokens "DNS maps names to addresses" and an end-of-turn token. The row shows the loss weight per token: 0 for all system and user tokens and the role markers, 1 for every assistant response token and the end-of-turn token.

A templated conversation as the model sees it during SFT, with the loss mask underneath. Only the assistant’s tokens — including the end-of-turn marker — carry loss weight 1 and produce gradient; the system and user tokens are context only. Template tokens simplified and invented for teaching; real templates differ per model family. (illustrative — source: Hugging Face — Chat templates documentation)

One SFT training step

  1. Sample a batch of conversations

    Curated demonstration conversations, drawn from the instruction dataset.

  2. Apply the chat template

    Serialize each conversation into one token sequence, with special tokens marking role boundaries and turn ends.

  3. Forward pass: predict every next token

    Exactly as in pre-training, the model produces a next-token distribution at every position in the sequence.

  4. Mask the loss to assistant tokens

    Positions whose target token belongs to a system or user span get loss weight 0. Only assistant-span targets count.

  5. Backpropagate and update weights

    Cross-entropy averaged over the unmasked positions; a gentle learning rate nudges the whole network toward the demonstrated behavior.

  6. Repeat for a few epochs

    SFT runs are short compared to pre-training — the model is being redirected, not rebuilt.

Masking the loss to response tokens

Why mask at all? Because the objective is “be the assistant”, not “be the whole conversation”. If prompt tokens carried loss, the model would spend capacity learning to generate user messages — imitating typos, vague phrasing, and whatever your users write — and your gradient budget would be dominated by text you never want it to produce. Masking says: condition on everything, imitate only the assistant.

The mechanics are simpler than the idea. The forward pass is untouched — every token, masked or not, flows through Attention and shapes the hidden states that the assistant-span predictions depend on. Prompt tokens are fully present as context; they just contribute no gradient of their own. In common frameworks this is implemented by setting the label at masked positions to an ignore index (conventionally -100 in PyTorch-style APIs), so the cross-entropy simply skips them, and the loss is averaged over the unmasked positions only.

One subtlety repays attention: the mask applies to targets, not inputs. The position whose target is the assistant’s first token sits at the end of the user’s span — so the model is trained, at exactly that boundary, to begin answering. And because the end-of-turn token is an unmasked target too, the model is explicitly trained to stop. A model that rambles past its answer or truncates mid-sentence often traces back to exactly these boundary tokens and how they were masked and templated.

Masking everything but the response is the common default, not a law of nature — recipes exist that keep some loss weight on prompt tokens. The trade-off is the one you would guess: more signal per example versus gradient spent on text the model should never emit.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.