Fine-tuning (FT)

Definition

Fine-tuning continues training a model that already works — usually a pre-trained foundation model — on a much smaller dataset of your own, so a behaviour you currently have to ask for becomes what the weights do by default. What it reliably buys is format, tone, domain vocabulary, tool-use conventions, and the ability to make a small cheap model match a large one on one narrow task. What it does not reliably buy is knowledge, and that is the part most articles skip: Gekhman et al. (2024) found that fine-tuning examples introducing genuinely new facts are learned significantly more slowly than examples consistent with what the model already knows — and that as those examples are eventually learned, they "linearly increase the model's tendency to hallucinate."

The distinction that decides everything downstream is one line: fine-tuning changes the weights, while prompting and RAG change what the model sees at inference time. A prompt is re-sent on every request and can be edited in a second. A fine-tune is paid for once, applies silently to every call afterwards, and is edited by running the whole pipeline again.

So the field's working order is prompt, then retrieve, then fine-tune — and that order is empirical, not fashion. Ovadia et al. (2023) compared knowledge injection by fine-tuning against retrieval directly and reported that "RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge." Reach for fine-tuning when the problem is behaviour — a JSON shape the prompt keeps drifting away from, a house style, a refusal policy, or shaving a 2,000-token instruction block off a latency-critical endpoint. The best production systems use both: a fine-tuned model that behaves, fed by retrieval that supplies current facts.

The token-cost side of that decision — how many requests it takes before a one-off training bill beats re-sending demonstrations forever — is worked out on the few-shot learning page. This page owns the other side: what the training run actually changes inside the model, and what it costs to do.

How It Works

Mechanically, fine-tuning is ordinary supervised training with one change of starting point: the weights begin at the pre-trained values instead of random ones. Each example is run forward through the model, the output is compared against the desired one to produce a loss, and gradient descent walks every trainable parameter a small step downhill. Nothing exotic happens. The interesting question is which parameters are trainable, because that single choice sets the entire cost.

The cost that surprises people is not the model — it is the bookkeeping the optimizer keeps beside it. Train with Adam in the standard mixed-precision setup and every parameter needs five copies of itself: a bf16 weight (2 bytes), its bf16 gradient (2), an fp32 master copy (4), and Adam's two moment estimates, also fp32 (4 + 4). That is 16 bytes per parameter before a single activation is stored. Llama 3.1 8B has exactly 8,030,261,248 parameters, so full fine-tuning needs 8.03e9 x 16 bytes ≈ 128 GB — of which the model itself is only 16 GB and the other 112 GB is gradients and optimizer state. It does not fit on an 80 GB H100.

Freeze the base and the picture inverts. LoRA leaves all 8.03 billion pre-trained weights untouched in bf16 — 16 GB, read-only, no gradient, no moments — and applies the 16-bytes-per-parameter tax only to a few million adapter parameters. A rank-8 adapter on the query and value projections is 3,407,872 trainable parameters, so the entire optimizer footprint is about 55 MB, and the run drops onto a single 24 GB consumer card. QLoRA loads the frozen base in 4-bit quantization instead, taking 16 GB down to roughly 4 GB; the paper reports that this "reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance."

One caveat is worth more than any other number here, because expecting otherwise is how people get burned: the memory saving is far smaller than the parameter saving. The LoRA paper cut trainable parameters on GPT-3 175B by 10,000x, but VRAM only from 1.2 TB to 350 GB — a factor of 3.4 — because the frozen weights still have to sit in memory and the activations still have to be kept for the backward pass. Parameter counts shrink by four orders of magnitude; memory shrinks by one. What does shrink by 10,000x is the artifact you ship: the checkpoint goes from 350 GB to 35 MB, which is why you can host a thousand behaviours against one copy of a base model.

The other half of the recipe is restraint. Fine-tuning runs at a learning rate roughly an order of magnitude below pre-training and for one to three epochs, not one long pass — push either harder and you get catastrophic forgetting, where the model aces your examples and quietly gets worse at everything else. That page covers the mechanism and the fixes; the operational point here is that your training loss will fall smoothly the entire time it is happening, so you must evaluate on capabilities you did not fine-tune for or you will never see it. LoRA is usually run at a learning rate about ten times higher than a full fine-tune, because the adapter starts from zero and has to travel further.

The most consequential decision is not a hyperparameter at all. LIMA (Zhou et al., 2023) fine-tuned a 65B LLaMa on 1,000 curated prompt-response pairs — no reinforcement learning, no preference modelling — and human raters judged its answers equal or better than GPT-4's in 43% of comparisons, Bard's in 58% and DaVinci-003's in 65%. Read the 43% honestly: GPT-4 still won the majority. The claim the result supports is not that a thousand examples beat a frontier lab, but that almost all the capability was already in the pre-trained weights and a thousand consistent examples were enough to surface it. Ten thousand contradictory ones would have taught the model to be contradictory.

Types

Two independent questions sit inside any fine-tune — how much of the model you update, and what signal you train it against — and they combine freely. A LoRA adapter trained with DPO is one point in that grid; a full fine-tune on instruction pairs is another.

How much of the model you update

• Full fine-tuning updates every weight. Maximum headroom, 16 bytes per parameter of memory, and a full-size checkpoint for every task you train.
• LoRA (Low-Rank Adaptation) freezes the base and learns a low-rank update alongside it. The dominant choice for large language models today.
• QLoRA keeps the frozen base in 4-bit and trains LoRA adapters on top in higher precision, cutting the floor by a further 4x.

DoRA, prefix tuning and Houlsby's original adapter modules vary what the small trainable part looks like, and the family is collectively parameter-efficient fine-tuning (PEFT); the memory argument is identical for all of them, because it comes from freezing the base rather than from the shape of what you train. The arithmetic behind LoRA specifically is simple enough to do in your head, which is unusual for anything this consequential. A weight matrix of shape d x k holds d·k parameters. A rank-r LoRA does not touch it; it learns the update as two thin matrices, d x r and r x k, for r(d + k) trainable parameters. On a 4096 x 4096 attention projection at r = 8, that is 8 x (4096 + 4096) = 65,536 against 4096 x 4096 = 16,777,216 — a factor of 256. For a square matrix the ratio collapses to d / 2r, so rank is an unusually cheap knob: doubling to r = 16 still leaves you at 128x fewer trainable parameters.

The bet underneath is that the update a fine-tune needs is intrinsically low-rank — that the difference between "base model" and "base model that always emits your schema" lives in a handful of directions. For format, tone and style that holds up well. For a domain genuinely far from the pre-training distribution it can fail, and a low-rank update cannot express a change it has no room to represent; that is the case where full fine-tuning earns its memory bill. Because the learned matrices can be added into the base weights before deployment, LoRA also costs nothing at inference: the paper notes this "guarantees that we do not introduce any additional latency during inference compared to a fine-tuned model by construction."

What signal you train against

• Supervised fine-tuning (SFT) / instruction tuning trains on input-to-desired-output pairs. This is how a raw base model becomes an instruction-follower, and it is the overwhelming majority of practical fine-tuning.
• Preference tuning aligns the model on pairs of preferred and rejected responses, via RLHF or the simpler DPO.
• Reinforcement fine-tuning optimises against a grader that scores outputs — a unit-test suite, a maths checker — using methods such as GRPO.

Those three are stages, not alternatives: a modern chat model is a base model that has been through SFT and then preference or reinforcement tuning, in that order. The linked pages cover how each optimiser works. What matters for the decision on this page is what each needs from you — SFT needs written answers, preference tuning needs judgements about pairs of answers you did not have to write, and reinforcement fine-tuning needs no answers at all but a grader you trust, which is the hardest of the three to build and the easiest to game.

Real-World Applications

Every instruction-following assistant you have used is a fine-tune. Meta's Llama base checkpoints become the Instruct variants through SFT followed by preference tuning, and OpenAI's InstructGPT (Ouyang et al., 2022) is the paper that established the recipe. The base model is not a worse chat model; it is not a chat model at all, and fine-tuning is the entire difference.

In code, GitHub Copilot's original Codex was GPT-3 fine-tuned on public repositories, and Code Llama is Llama 2 with continued training on code. Teams apply the same shape internally — fine-tuning a model so it reaches for the company's own framework and naming conventions rather than the most popular ones on the public internet. This is a textbook behaviour problem: the model already knows how to write the language, and what you are correcting is which of several valid styles it defaults to.

The reasoning models are the newest branch. DeepSeek-R1 was produced with reinforcement fine-tuning against automatically verifiable maths and code rewards rather than by hand-writing chains of thought, which is what makes the approach scale — nobody has to author the reasoning, only check the answer. On the other side of the spectrum, OpenAI's Whisper is routinely fine-tuned by practitioners on a few hours of audio for one accent, language or noisy acoustic environment it handles weakly out of the box.

The pattern that has quietly become most common is the cheapest one: fine-tune a small model on a large model's outputs so it matches the big model on a single narrow task at a fraction of the serving cost. That is knowledge distillation wearing a fine-tuning implementation, and it is the case where the payoff is measured in unit economics rather than in quality — a smaller model, a shorter prompt, lower latency, and the same output.

Key Concepts

• LoRA rank (r): how many directions the adapter is allowed to move the model in. It is the capacity knob and the overfitting knob at the same time.
• Adapter merging: folding B·A back into the base weights before serving, which is why LoRA adds no inference latency but also why a merged model can no longer be swapped per request.
• Base-model pinning: a fine-tune belongs to one specific checkpoint. New base version, new training run.
• Retain-set evaluation: scoring capabilities you did not train on, before and after. Without it, a forgetting regression ships invisibly.
• Alignment tax: the general-benchmark cost that preference tuning imposes on top of an SFT model, and the reason labs evaluate broadly after every post-training stage.

Challenges

The hardest problem is not training the model, it is knowing whether the model got better. You need an evaluation before you have anything to evaluate, and it has to measure two things at once: the new skill, and the general capability you may have damaged getting it. A single task metric will show a clean improvement while the model has become measurably worse at instruction-following — the loss curve you are watching cannot see it, because the loss is defined entirely over your examples.

The second is that the dataset, not the checkpoint, is the real artifact, and its quality problems are invisible until they are baked in. Contradictory labels do not average out into a compromise; they teach the model that both answers are acceptable, which surfaces as an unpredictable model rather than a wrong one. Nothing downstream recovers from this — not a lower learning rate, not a higher rank, not more epochs. Deduplicating and auditing a thousand examples is genuinely more valuable than collecting nine thousand more.

The third is the one that gets left out of every cost estimate: fine-tuning is not a one-off. Your adapter is pinned to a specific set of base weights, so when the provider deprecates that snapshot or ships a better one, nothing transfers — you re-run the data pipeline, the training, and the evaluation to stand still. On a fast-moving base model that bill arrives every few months, and it is the strongest practical reason to keep facts in a retrieval index (which survives a base swap untouched) and spend fine-tuning only on behaviour that is genuinely yours.

Future Trends

• Reinforcement fine-tuning becomes a product feature. Verifier-graded training is moving out of frontier labs into hosted APIs, which shifts the customer's work from writing answers to writing graders — a different and largely unsolved engineering discipline.
• Multi-adapter serving changes the arithmetic. Runtimes that hold one base model in GPU memory and hot-swap thousands of LoRAs per request make per-customer or per-task fine-tuning economically sane for the first time, because the marginal cost of the ten-thousandth adapter is a few megabytes rather than a GPU.
• The set of problems needing a fine-tune keeps shrinking. As base models get better at following instructions and context windows get cheaper, capability gaps close on their own and the durable reasons to fine-tune narrow to the ones that do not: guaranteed output format, latency, and cost per token.

Code Example

Rather than assert that LoRA trains "under 1% of the parameters", compute it from the real shapes of Llama 3.1 8B. The only inputs are the layer dimensions and the rank:

# Llama 3.1 8B: hidden 4096, 32 heads, 8 KV heads (GQA), FFN 14336, 32 layers.
SHAPES = {
    "q_proj":    (4096, 4096),
    "k_proj":    (4096, 1024),    # 8 KV heads x 128 head dim
    "v_proj":    (4096, 1024),
    "o_proj":    (4096, 4096),
    "gate_proj": (4096, 14336),
    "up_proj":   (4096, 14336),
    "down_proj": (14336, 4096),
}
LAYERS, TOTAL = 32, 8_030_261_248

def lora_params(modules, r):
    # a rank-r adapter on a d x k matrix has r(d + k) trainable parameters
    return LAYERS * sum(r * (d + k) for d, k in (SHAPES[m] for m in modules))

for label, mods, r in [
    ("q,v only   r=8 ", ["q_proj", "v_proj"], 8),
    ("all linear r=8 ", list(SHAPES), 8),
    ("all linear r=64", list(SHAPES), 64),
]:
    n = lora_params(mods, r)
    print(f"{label}: {n:>12,}  ({n / TOTAL:.3%})  optimizer state {n * 16 / 1e6:.0f} MB")

# q,v only   r=8 :    3,407,872  (0.042%)  optimizer state 55 MB
# all linear r=8 :   20,971,520  (0.261%)  optimizer state 336 MB
# all linear r=64:  167,772,160  (2.089%)  optimizer state 2684 MB

Two things fall out of that table. Even the greediest configuration here — every linear layer at rank 64 — is 2.1% of the model, so "under 1%" is a description of the usual defaults rather than a law. And the trade-off is real: adapting only q_proj and v_proj is the original paper's setting, while adapting all linear layers is the setting QLoRA found necessary to match full fine-tuning quality, at six times the trainable parameters. Now the memory picture the choice actually controls:

P, ADAPTER = 8_030_261_248, 3_407_872
GB = 1e9

full_ft = P * 16 / GB                        # weights + grads + fp32 master + Adam m, v
lora    = (P * 2 + ADAPTER * 16) / GB        # base frozen in bf16, adapter trains
qlora   = (P * 0.5 + ADAPTER * 16) / GB      # base frozen in 4-bit

print(f"full {full_ft:.1f} GB | lora {lora:.1f} GB | qlora {qlora:.1f} GB")
# full 128.5 GB | lora 16.1 GB | qlora 4.1 GB     (before activations)

The frozen base weights dominate both LoRA rows, which is exactly why quantizing them is the next lever and why the memory ratio (8x, then 32x) is nothing like the parameter ratio (2,356x).

Academic Sources

• "LoRA: Low-Rank Adaptation of Large Language Models" — Hu et al. (2021) — 10,000x fewer trainable parameters and 3x less GPU memory on GPT-3 175B.
• "QLoRA: Efficient Finetuning of Quantized LLMs" — Dettmers et al. (2023) — 4-bit NormalFloat base weights; a 65B fine-tune on one 48 GB GPU.
• "LIMA: Less Is More for Alignment" — Zhou et al. (2023) — 1,000 curated examples, no RL, and the case for quality over quantity.
• "Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs" — Ovadia et al. (2023) — retrieval beats fine-tuning for injecting facts, new or old.
• "Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?" — Gekhman et al. (2024) — new-knowledge examples are learned slowly and raise hallucination linearly.
• "Training language models to follow instructions with human feedback" — Ouyang et al. (2022) — InstructGPT; the SFT-then-preference-tuning recipe.
• "Parameter-Efficient Transfer Learning for NLP" — Houlsby et al. (2019) — adapter modules, the origin of the PEFT family.