Catastrophic Forgetting

Why fine-tuning makes a model worse at everything it used to do well — the mechanism inside shared weights, and what each fix actually costs.

Published Updated

On this page

Definition

Catastrophic forgetting is what has happened when you fine-tune a model, it gets better at your task, and it comes back measurably worse at everything else — instruction-following, arithmetic, the second language it used to speak. Nothing malfunctioned: gradient descent moved the weights your new loss asked it to move, and that loss contains no examples of the old behaviour and therefore no penalty for destroying it.

If you want the practical answer before the explanation: freeze the base model and train an adapter instead. A rank-8 LoRA attached to the query and value projections of an 8-billion-parameter model trains roughly 3.4 million parameters — about 0.04% of the total — and the other 99.96% receive no gradient at all, so the base weights are byte-for-byte identical when you finish. The rest of this page is why that works, the one thing it does not fix, and what to do when freezing the base is not an option.

How It Works

There is no place where the old knowledge is kept

A neural network does not store "French" or "how to follow instructions" in a location you could route around. Every capability is spread across the same weights, and every weight participates in many capabilities at once — that distributed encoding is exactly what lets the network generalise instead of memorising. It is also what makes it fragile: there is no cell to leave alone, because the cells that hold what you want to keep are the cells your new task will use.

So when training begins, the optimiser has one instruction. It computes the gradient of your loss function with respect to every parameter and steps downhill. That loss is defined entirely over your fine-tuning examples. It has no term for "and do not damage what you already knew", because you did not give it one and there is nowhere in the objective for such a term to live unless you add it deliberately. Forgetting is not a bug in the optimiser; it is the optimiser succeeding at the problem you actually posed.

The scale of the overwrite is worth doing as arithmetic. Fine-tune a 7-billion-parameter model on 1,000 examples at batch size 8 for three epochs and you have taken 375 gradient steps. In a full fine-tune each step writes a new value to all 7 billion parameters — about 2.6 trillion parameter updates, every one of them justified by a loss that has only ever seen your 1,000 examples. The result of a pre-training run over trillions of tokens is being steered by a dataset that fits in a spreadsheet.

Why "catastrophic" and not "gradual"

The word is doing real work. Old-task performance does not decay smoothly in proportion to how far the weights moved; it can collapse. The reason is that a shared representation feeds decisions everywhere downstream, so shifting it slightly moves a large population of inputs across decision boundaries at once. The effect was named in McCloskey and Cohen's 1989 study: a network taught addition facts with 1 as an addend, then taught facts with 2, lost the first set almost entirely after a handful of passes over the second — not degraded, erased.

This is also why a fine-tune can look fine right up until it does not. Loss on the new task falls smoothly and monotonically, which is the curve you are watching, while the capability you were not measuring falls off a cliff somewhere in the middle of epoch two.

The stability–plasticity dilemma, stated properly

Stephen Grossberg's framing from the 1980s is the sharpest way to see why there is no clean fix. The property that lets a network absorb a new task quickly — large, freely-moving weights that respond strongly to new gradients — is the identical property that lets it overwrite an old one quickly. Plasticity and stability are two readings of the same number. You cannot raise both from the same mechanism, so every mitigation below buys one by spending the other, and the useful question about any of them is not "does it work" but "which trade is it making".

The four mitigation families and what each costs

Turn the learning rate down. The crude first line of defence, and the one to try before anything clever. Fine-tuning typically runs at 1e-5 to 5e-5 where pre-training used 1e-4 to 3e-4 — an order of magnitude smaller — with one to three epochs rather than one long pass over trillions of tokens. Smaller steps move shared weights less far, so less is destroyed. The cost is symmetric and unavoidable: less is also learned, and on a genuinely distant task you may not reach your quality bar at any rate low enough to be safe.

Replay the old data. Mix examples of what you want to keep into the new training set, so the loss regains a term for the old behaviour. This is the most direct fix and it works at industrial scale: Code Llama continued training Llama 2 on 500 billion tokens of a mix that was deliberately only 85% code, with 8% code-adjacent text and a further 7% plain natural language held in specifically so the model would keep the language ability it started with. The cost is that you need the old data. A modern base model's pre-training corpus runs to trillions of tokens — Llama 3 used roughly 15 trillion — so replaying even 1% means sourcing 150 billion tokens, and if you are fine-tuning someone else's checkpoint you do not have them at any price. Retention can also be legally impossible: data licensed for one training run, or subject to an erasure request, cannot sit in a replay buffer waiting for the next one.

Penalise moving the weights that mattered. Regularisation approaches keep the old data out of the loss but put a proxy for it in: estimate how important each parameter was to the previous task and add a quadratic penalty for moving the important ones. Elastic weight consolidation (Kirkpatrick et al., 2017) uses the diagonal of the Fisher information matrix as that importance estimate. The cost is bookkeeping and honesty about the approximation: you must store two extra numbers per parameter — the anchor value and its importance — which for a 7B model is 14 billion extra floats, about 28 GB in 16-bit, on top of everything training already needs. And because importance is estimated per task, the errors compound over a long sequence of tasks; EWC slows forgetting, it does not eliminate it.

Isolate the parameters. Freeze the base and give each new task its own small set of weights. Houlsby et al. (2019) showed adapter modules coming within 0.4% of full fine-tuning on GLUE while adding 3.6% of parameters per task, and LoRA pushed the same idea to well under 1%. This is the dominant practical answer for large models, and its guarantee is structural rather than statistical: the frozen weights receive no gradient, so no amount of training can move them, and the original model is recoverable exactly by detaching the adapter. The honest caveat is that what you deploy is base plus adapter, and that combined function can still answer unrelated questions worse — the base cannot forget, but the system can still regress. What you have bought is a perfect undo, cheap per-task storage of a few megabytes, and the ability to serve many behaviours from one un-degraded set of base weights. What you have spent is capacity: a low-rank update cannot express every change a full fine-tune could.

Knowing whether it happened

None of this is actionable without measurement, and measuring only the task you trained on is how the regression ships. Keep a held-out evaluation of the capabilities you care about keeping, score it before and after, and report the delta. The continual-learning literature formalises this as backward transfer: accuracy on an earlier task after all later training, minus accuracy on it immediately after it was learned. Negative backward transfer is forgetting, quantified.

Real-World Applications

  • Alignment tuning at OpenAI: the InstructGPT paper (Ouyang et al., 2022) named the "alignment tax" — RLHF improved instruction-following while regressing performance on several public NLP datasets. The fix, PPO-ptx, mixes pre-training gradients into the reinforcement-learning update: replay, wearing a production hat. Broad post-training evaluation suites exist because of this result.
  • Domain-adaptive continued pre-training: Code Llama's 85/8/7 data mix is a forgetting budget written as a recipe, and the same reasoning governs continued pretraining runs that adapt a general model to medicine, law or a single codebase.
  • Adapter fleets in production: image-generation communities host tens of thousands of LoRAs trained against a handful of shared base checkpoints, and multi-adapter serving stacks swap them per request against one copy of the base in GPU memory. The economics only work because the base is provably unchanged by any of them.
  • Robotic policies: Google DeepMind's RT-2 co-fine-tuned on web vision-language data alongside robot trajectories rather than on trajectories alone, because robot-only fine-tuning cost the generalisation the web data had provided.
  • Sequential agents: the original EWC result trained a single network on ten Atari games in sequence and retained playable performance across them, where the unprotected baseline kept only the most recent — the cleanest demonstration that the phenomenon is about the optimiser, not about language models.

Key Concepts

  • Stability–plasticity dilemma: fast learning and durable memory are the same knob read two ways, which is why every mitigation is a trade rather than a fix.
  • Backward transfer: the signed measurement of what later training did to earlier capabilities; negative means forgetting.
  • Alignment tax: capability lost to a post-training step that was targeting behaviour, not knowledge.
  • Fisher information: EWC's estimate of which parameters the previous task depended on, and the reason the method costs memory proportional to the model.
  • Task-recency bias: a sequentially trained model skews toward whatever it saw last, even when older tasks are formally still in the evaluation.

Challenges

The hardest part is not preventing forgetting but noticing it. "Everything else the model could do" is not a finite set, so there is no evaluation that covers it; you sample, and the capability you failed to sample is the one that quietly breaks in production three weeks later. This asymmetry is why the field's practical advice keeps collapsing into "freeze more, train less" — not because parameter isolation is theoretically elegant, but because a guarantee you can reason about beats a regression suite you have to trust.

The second difficulty is that the mitigations do not compose for free. Replay needs data you may not legally hold. Regularisation needs an importance estimate whose error accumulates across tasks. Adapters cap how much the model can change, which is a real ceiling when the target domain is genuinely far from the base distribution — at which point you are back to full fine-tuning with a replay mix, paying storage and compute. And a longer task sequence makes all of it worse: each new task both adds something to protect and degrades the estimates protecting everything before it.

Finally, forgetting is not always the enemy. Machine unlearning, model editing and right-to-erasure compliance all want targeted, verifiable removal of specific knowledge — the same mechanism, deliberately aimed. The difficulty there is the mirror image of the one above: the distributed encoding that makes accidental forgetting easy makes precise forgetting extremely hard, because you cannot delete the weights holding one fact without touching everything else they hold.

Future Trends

Modularity is the direction with the most momentum. Mixture-of-experts routing, adapter libraries and model merging all attack the problem the same way — by giving new knowledge somewhere to go that is not on top of the old knowledge — and merging in particular has turned adapters into composable objects, where task vectors can be added to or subtracted from a base model arithmetically. Expect the interesting work to be in routing and composition rather than in better regularisers.

The second trend is evaluation catching up. Because a forgetting regression is invisible to the metric a fine-tune optimises, retain-set benchmarking is moving from a research convention into standard release engineering: score the general suite before and after every post-training step, and treat an unexplained drop the way you would treat a failing test. That is a process change more than a technical one, and it is the one most teams doing fine-tuning today are still missing.

Frequently Asked Questions

Because the weights that encoded those abilities are the same weights your fine-tune updated. A neural network stores knowledge across shared parameters, not in separate slots per task, and gradient descent optimises only the loss you gave it — which contains no examples of the old behaviour and therefore no penalty for destroying it. Nothing broke; the optimiser did exactly what it was asked.
It prevents the base model from forgetting, which is not the same thing. The frozen base weights receive no gradient at all, so the original model is recoverable byte-for-byte by detaching the adapter. But what you deploy is base plus adapter, and that combined function can still behave worse on unrelated inputs. LoRA gives you a guaranteed undo and cheap per-task swapping, not immunity.
Keep a held-out evaluation for the old capabilities and score it before and after, then report the drop. In the continual-learning literature this is backward transfer: accuracy on an earlier task after training on later ones, minus accuracy on it right after it was learned. A negative value is forgetting. Measuring only your target task is how the regression ships unnoticed.
Lower the learning rate and cut the number of epochs. Fine-tuning typically runs at 1e-5 to 5e-5 against pre-training rates around 1e-4 to 3e-4, and pushing either the rate or the epoch count up is the most common cause of a fine-tune that wins on your examples and loses everywhere else. It is crude, it trades away some adaptation, and it works.
No. Overfitting is failing to generalise from the data you trained on. Catastrophic forgetting is losing what you could already do before that training started. They often appear together in a small-dataset fine-tune, and a single validation set on the new task will detect the overfitting while missing the forgetting entirely.
Yes, and it has a name: the alignment tax. OpenAI's InstructGPT paper reported that preference tuning regressed performance on several public NLP benchmarks, and mitigated it by mixing pre-training gradients back into the update. This is why labs evaluate broadly after any post-training step rather than only on the behaviour they were trying to change.

Related Terms

Continuous Learning (CL)

Keeping a deployed model current as the world moves — why teams schedule batch retraining instead of true online learning, and what breaks if they don't.

Transfer Learning (TL)

Transfer learning reuses a model trained on one large task as the starting point for a related task, so you need far less labeled data and compute.

Fine-tuning (FT)

What fine-tuning changes inside a model, what it costs in GPU memory, and why it reliably buys format and style but not new facts.

Simple Continual Pretraining (SCP)

Method for turning a pretrained autoregressive LLM into a diffusion language model by continuing pretraining with a bidirectional attention mask.

Neural Network

Why a stack of layers does what one cannot: two linear layers collapse into a single matrix, and a hidden layer of two units solves XOR.

Meta-Learning

Learning to learn: training a model across thousands of tiny tasks so it adapts from a handful of examples — and how that differs from fine-tuning.

Continue Learning

Explore our use-case guides and prompts to deepen your AI knowledge.

Browse AI Tools← Back to Glossary