QRS explainer · Large language models

Post-training

How a model that predicts the next word becomes an assistant that answers, follows instructions and reasons.

Pre-training teaches a language model what text looks like. Post-training reshapes what it chooses to say: first by imitating good answers, then by learning which of its own answers people prefer, and most recently by rewarding answers that can be checked.

Read the guide → The pipeline at a glance

A long read in seven parts, from the history to the derivations and the practice, with ten interactive figures.

By Yuhe Sui · Last reviewed: 14 October 2026

中文版指南 →

UserWhat is the capital of France?Probability of each whole replyy1 What is the capital of Germany?0%continues the quizy2 France is a country in Western Europe, bordered by…3%keeps writingy3 Paris.25%answersy4 The capital of France is Paris.61%answers in fully5 Lyon.1%wrongy6 Great question! The capital of France is Paris, a ci…10%flatters
One prompt, six possible replies: the running example of the guide. The base model mostly continues the document; supervised fine-tuning moves probability onto answers; RLHF concentrates it on the reply the reward model scores highest, mostly the complete sentence but partly a flattering reply the reward model over-rates. The facts the model knows do not change. Illustrative probabilities, not measurements from a real model.

The pipeline at a glance

The classic recipe runs left to right. The last column shows the two main departures from it.

  1. Starting point

    Pre-training

    Predict the next token on a very large amount of text. Knowledge and skills, but no notion of answering.

    History →

  2. Imitate

    Supervised fine-tuning

    Train on prompt–response demonstrations with the same next-token loss, counted on the response only.

    SFT →

  3. Compare

    Reward model

    Fit a scalar score to human comparisons with the Bradley–Terry model: the preferred reply should score higher.

    Reward model →

  4. Optimise

    RL with PPO

    Sample replies, score them, and push the policy towards high reward while a KL penalty keeps it near the SFT model.

    PPO →

  5. Departures

    DPO

    Skip the reward model and the RL loop: a classification-style loss on preference pairs, derived from the same objective.

    RLVR and GRPO

    Replace learned rewards with checkable ones (a correct maths answer, passing tests) and the critic with a group of samples.

    DPO →   GRPO →

What you will learn

  1. From task models to pre-train and adapt

    ULMFiT, GPT-1, BERT, GPT-2, T5 and GPT-3, and the question they leave open.

  2. Supervised fine-tuning

    The objective, derived; chat templates and loss masks; why a little data goes a long way.

  3. Learning from preferences

    Why comparisons, the Bradley–Terry reward model, and reward hacking.

  4. Reinforcement learning, from scratch

    MDPs, returns, policy gradients, value functions, Bellman equations, TD and GAE.

  5. RLHF in practice

    Trust regions, PPO’s clipped objective, the InstructGPT pipeline and its limits.

  6. Beyond PPO

    DPO, verifiable rewards, GRPO and DeepSeek-R1.

  7. In practice

    The methods side by side, reported settings, memory costs, failure diagnosis, evaluation and open questions.

An explainer, not a QRS research project. The toy figures are illustrative; every other claim cites its source in the guide’s references.