Definition
Supervised fine-tuning (SFT) is the training stage that takes a raw pretrained language model — a model that can only continue text — and teaches it to respond to instructions the way a person would want. It does this by training on a curated dataset of (prompt, ideal response) pairs, adjusting the model's weights so that, given the prompt, it produces the response. The learning signal is ordinary next-token prediction: the same cross-entropy loss used in pretraining, now measured only against answers a human considered good.
The word supervised is the key. In pretraining, the model learns from raw text with no notion of a "correct" answer — every document is just something to predict. In SFT, each example carries a target that someone judged to be the right output, so the model is being corrected toward specific, endorsed behavior rather than toward the statistical average of the internet. That single change is what turns a model that autocompletes a question into a wall of ten more questions into a model that answers it.
SFT is why a base model and an instruction-tuned model of identical size feel like different products. The result that made this concrete came from OpenAI's 2022 InstructGPT paper: labelers preferred the outputs of a 1.3-billion-parameter model that had been fine-tuned and aligned over those of the original 175-billion-parameter GPT-3, "despite having over 100x fewer parameters." The knowledge was already in the big model; SFT (and the preference optimization after it) is what made a much smaller model actually usable.
How It Works
SFT is the middle stage of a three-part pipeline. First comes pretraining, where a model reads trillions of tokens of text and learns to predict the next one — this is where nearly all of its knowledge and language ability is formed (see Pre-trained Models). Then comes SFT, a short, focused stage on a small curated dataset that shapes how the model uses that knowledge. Finally comes preference optimization — RLHF, or newer variants like direct preference optimization and RLVR — which refines quality using a reward signal instead of fixed target answers.
Within the SFT stage, each training example is a conversation laid out as text: a prompt (often a system message plus a user turn) followed by the response the model should have produced. The whole thing is tokenized into one sequence, and the model is trained with the same next-token objective as pretraining — but with one critical adjustment called loss masking. The tokens belonging to the prompt are excluded from the loss; only the tokens of the response generate a gradient. The model is never rewarded for predicting the user's words, because at inference time it will never have to. It is graded solely on the answer it is supposed to write.
A worked example: cross-entropy on the answer only
Cross-entropy loss for one token is -ln(p), where p is the probability the model assigned to the token that actually appears next. If the model is completely confident and correct — p = 0.99 — the loss for that token is -ln(0.99) ≈ 0.010. If it splits its bet evenly between two plausible words — p = 0.5 — the loss is -ln(0.5) ≈ 0.693. Training pushes the average of this quantity, over every response token, downward.
Now the masking. Suppose one example has a 40-token prompt and a 160-token response, 200 tokens in all. Loss masking zeroes out the 40 prompt tokens, so only 160 tokens — 80% of the sequence — contribute to the gradient. Across a dataset of 10,000 such examples, the model is corrected on roughly 1.6 million answer tokens. Contrast that with pretraining, where a 7-billion-parameter model might see on the order of two trillion tokens. Ten thousand examples at ~500 tokens each is about 5 million tokens; run for 3 epochs, that is 15 million token-updates — roughly one part in 130,000 of the pretraining exposure. SFT is a light touch by design, not by accident.
Why the learning rate is smaller
Because SFT sees so little data relative to pretraining, its danger is not under-fitting but over-writing. A large gradient step tuned for two trillion tokens would, applied to a few million, drag the weights far enough to erase capabilities the model spent its whole pretraining budget acquiring. So SFT uses a much smaller learning rate — typically on the order of 1e-5 to 2e-5, roughly ten to thirty times below a pretraining peak of a few 1e-4 — and only a handful of epochs. The learning rate and epoch count are the two hyperparameters that most determine whether SFT teaches the model to follow instructions or quietly damages it.
The epoch count is genuinely subtle. InstructGPT's authors report that their "SFT models overfit on validation loss after 1 epoch; however, we find that training for more epochs helps both the RM score and human preference ratings, despite this overfitting." They trained for 16 epochs with a cosine learning-rate decay. In other words, the validation-loss number — the thing you would normally watch to stop training — pointed the wrong way, and the behavior people actually cared about kept improving past it. SFT is one of the places where the standard overfitting signal cannot be trusted at face value.
Real-World Applications
Every widely used instruction-following or chat model has an SFT stage; it is not an optional refinement but the step that makes the model a product.
InstructGPT and the GPT-3.5 line. OpenAI's InstructGPT was the reference implementation of this pipeline. Its SFT dataset "contains about 13k training prompts," a mixture of prompts submitted to the API and prompts written by hired labelers, with the ideal responses also written by those labelers. The reward-modeling and PPO stages that followed used 33k and 31k prompts respectively — but SFT came first, and without it the later stages have nothing coherent to improve upon. The InstructGPT recipe is the direct ancestor of the ChatGPT-era assistants.
Open-weight instruct models. The chat or "instruct" variants shipped alongside base models — from families such as Llama, Mistral, Qwen and Gemma — are their base models plus SFT (usually followed by preference optimization). The base and instruct checkpoints have identical architectures and parameter counts; the difference a user feels is almost entirely the fine-tuning. This is also why the open-source community can take a base model it did not train and produce a capable assistant from it: SFT is cheap enough to run on rented hardware, so the expensive pretraining is done once and specialized many times.
Domain and task specialization. Beyond general instruction following, SFT is how teams adapt a general model to a narrow job: fine-tuning on a few thousand labeled examples of medical-coding decisions, legal-clause classification, or a company's own support transcripts, so the model adopts the exact format, vocabulary and judgment that task requires. Here SFT is doing transfer learning — reusing general capability, redirecting it with a small, specific dataset.
Key Concepts
Data quality dominates data quantity. The strongest evidence for this is the LIMA study, whose name stands for "Less Is More for Alignment." Its authors fine-tuned "a 65B parameter LLaMa language model ... on only 1,000 carefully curated prompts and responses, without any reinforcement learning or human preference modeling" and still reached strong instruction-following. Their explanation is the mechanism worth remembering: "almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary to teach models to produce high quality output." SFT is not teaching the model facts — it is teaching it a style of response it already had the raw material to produce. That is why a thousand pristine examples can beat a hundred thousand scraped ones, and why curating the dataset is usually where the real work of SFT lives. Some teams now expand small human-written seed sets with model-generated synthetic data, which raises quality only if the generated examples are filtered as carefully as human ones.
Full SFT versus parameter-efficient SFT. Full SFT updates every weight in the model, which means holding optimizer state for billions of parameters in memory and saving a complete multi-gigabyte copy of the model per fine-tune. Parameter-efficient methods avoid this. The most common, LoRA, freezes the original weights entirely and trains a small pair of low-rank matrices injected alongside them, often well under 1% as many trainable parameters. The forward objective is identical — same prompt-response pairs, same masked cross-entropy — only the set of weights being adjusted changes. For most instruction-tuning tasks the quality is comparable, and the practical wins (fit on a smaller GPU, store a 20 MB adapter instead of a 14 GB model, swap adapters at serving time) are large. "SFT" without qualification usually means full fine-tuning; LoRA and its relatives are parameter-efficient SFT.
SFT is imitation; RLHF is optimization. The two stages differ in what they can teach. SFT can only imitate the answers in its dataset, so it is bounded by the quality of those answers — it cannot make the model better than its best demonstration. Preference-based methods learn from comparisons ("A is better than B") rather than a single gold target, which lets them push into regions no human wrote out in full and to suppress behaviors that are easy to rank against but hard to demonstrate. SFT gets the model into the right neighborhood cheaply; RLHF explores within it. Running RLHF on a model that has not been through SFT rarely works, because the reward signal needs a reasonable starting policy to improve.
Challenges
Catastrophic forgetting. The central risk is that fine-tuning on a narrow dataset degrades capabilities the model had before. Train too hard on 5,000 customer-service transcripts and the model may get better at that voice while losing coding ability or general reasoning it will never see reinforced in this data. This is catastrophic forgetting, and it is the direct reason for the small learning rate, the few epochs, and the popularity of LoRA — freezing the base weights makes forgetting far harder because the original knowledge is literally untouched.
Overfitting to surface form, not just to loss. As the InstructGPT epoch result showed, validation loss is an unreliable stop signal here. Worse than memorizing tokens, a model can overfit to superficial patterns in the dataset: if every example response opens with "Certainly! Here is..." the model will paste that on everything, having learned the format's shell instead of its substance. Small, homogeneous SFT sets are especially prone to this, which is why diversity of the curated examples matters as much as their individual quality.
Exposure bias. SFT trains the model on perfect human-written prefixes — at every step the tokens it conditions on are the ideal ones. At inference it conditions on tokens it generated, which may already contain a small mistake the training distribution never showed it how to recover from, so errors can compound across a long response. This gap between training-time and generation-time conditions is one reason SFT alone leaves quality on the table that later preference optimization recovers.
You inherit the dataset's flaws exactly. Because SFT is direct imitation, every error, bias, formatting quirk or unsafe answer in the curated responses is a target the model is being explicitly trained to reproduce. There is no reward model to catch a bad demonstration — a wrong answer written confidently by a rushed labeler becomes ground truth. The dataset is not a suggestion to the model; it is the definition of correct.
Code Example
The mechanism that distinguishes SFT from plain language-model training is loss masking: computing the loss over the response tokens only. The example below builds one training sequence, masks the prompt, and shows how the same per-token cross-entropy produces very different numbers depending on the model's confidence.
import math
# One SFT example, already tokenized. -100 is PyTorch's
# "ignore this position in the loss" sentinel.
prompt_ids = [8, 3, 51, 9] # 4 prompt tokens
response_ids = [12, 40, 7] # 3 response tokens
input_ids = prompt_ids + response_ids
labels = [-100] * len(prompt_ids) + response_ids # prompt masked out
# Probabilities the model assigned to each *correct* response token.
p_correct = [0.99, 0.50, 0.90]
# Cross-entropy is averaged only over unmasked (response) positions.
losses = [-math.log(p) for p in p_correct]
sft_loss = sum(losses) / len(losses)
print("per-token losses:", [round(l, 3) for l in losses])
print("masked positions:", labels.count(-100), "of", len(labels))
print("SFT loss:", round(sft_loss, 3))
Running it prints:
per-token losses: [0.01, 0.693, 0.105]
masked positions: 4 of 7
SFT loss: 0.27
The four prompt tokens contribute nothing; the model is graded only on the three answer tokens. The confident-and-correct token (p = 0.99) barely registers at 0.01, while the coin-flip token (p = 0.5) dominates the average at 0.693 — so the gradient pushes hardest exactly where the model was most uncertain about its own answer. Scale this from three tokens to millions and from one example to a few thousand, tune the learning rate down an order of magnitude from pretraining, and that is the whole of supervised fine-tuning.