Definition
Machine unlearning is the problem of making an already-trained model forget specific training examples — removing their influence on its parameters — without retraining the model from scratch. Someone asks a company to delete their data under a privacy law; a court orders a copyrighted book removed; a safety team wants a model to no longer know how to synthesise a nerve agent. In every case the data is already inside the model, smeared across its weights, and deleting the original file does nothing to the model that has shipped.
That last point is the one most people get wrong, so it is worth stating plainly. Training a model is a one-way compression: the model reads the data and adjusts millions or billions of parameters in response. Once that is done, erasing the source file is like burning a recipe after you have already baked the cake — the cake still tastes of every ingredient. Unlearning is the act of un-baking, and it is hard for exactly the reason baking is useful: the information has been mixed in everywhere.
The reason anyone bothers with a special technique, rather than the obvious "delete the row and retrain," is cost. Retraining is not free; for a large model it is the single most expensive thing the organisation does. So the entire field exists to answer one question: can we get the effect of retraining-without-that-data for far less than the cost of retraining-without-that-data?
How It Works
The baseline that everything is measured against
The only method that is unarguably correct is to throw the trained model away, remove the data in question, and train a fresh model on what remains. This is called retraining from scratch, and it is the reference every unlearning method is compared to: an unlearning algorithm is "good" to the extent its output is indistinguishable from this retrained model.
To see why it is avoided, cost it out. A rough but durable rule for transformer training
compute is C ≈ 6ND floating-point operations, where N is the number of parameters and
D the number of training tokens. A 70-billion-parameter model trained on 15 trillion
tokens costs
C = 6 × (70 × 10^9) × (15 × 10^12) ≈ 6.3 × 10^24 FLOPs
That is the entire run — weeks on a large accelerator cluster and, as of publication,
millions of dollars. Now suppose one user exercises their right to have their photos
removed. Naive compliance means spending that whole 6.3 × 10^24-FLOP budget again to
delete, perhaps, a few dozen images. Ten such requests over a model's life means ten full
retrains. This does not scale, and that economic wall is the entire motivation for
unlearning.
SISA: making exact unlearning cheap by design
The most influential exact-unlearning method is SISA training — short for Sharded, Isolated, Sliced, and Aggregated — from Bourtoule et al. (2021). The idea is to change how you train in the first place, so that deletion is cheap later.
Instead of training one model on all the data, you shard: split the data into, say, 20
disjoint pieces and train 20 models, each on its own shard and nothing else. At inference
you aggregate their outputs, for example by majority vote. Because each example touched
exactly one shard's model, deleting an example only requires retraining that one shard —
roughly 1/20 of the work — while the other 19 models are untouched. Slicing refines
this further: within a shard, data is added in ordered slices and checkpoints are saved, so
you can restart from the checkpoint just before the deleted point rather than from the
shard's beginning.
The catch is that the naive 1/20 intuition overstates the real gain, and the paper's
measured numbers make that concrete. Over retraining from scratch, SISA improves time-to-
unlearn by 4.63× on the Purchase dataset and 2.45× on SVHN, at a cost of under 2
percentage points of accuracy. On a genuinely hard task the benefit shrinks to 1.36× on
ImageNet. Why so much less than 20×? Because splitting the data 20 ways means each sub-model
sees only 1/20 of it, which hurts accuracy — so on hard tasks you cannot afford many shards,
which is exactly where deletion would have been cheapest. Sharding trades away some of the
statistical power that made training on all the data worthwhile. That tension — cheap
deletion versus model quality — is the defining trade-off of exact unlearning.
Approximate unlearning: editing the weights directly
The alternative gives up the guarantee to buy speed. Approximate unlearning takes the existing model and edits its weights to reduce the forgotten data's influence, with no retraining of any part. Common recipes include running gradient ascent on the forget set (pushing the loss up on exactly the examples you want gone), fine-tuning on the retained data to overwrite them, or — for safety use — redirecting the internal representations that encode a topic, as in the RMU method. These are orders of magnitude cheaper than retraining.
What they lack is any promise that the influence is truly gone. There is no theorem, for a deep network, that the edited model equals the retrained one. It might look like it has forgotten while the knowledge sits latent, one fine-tuning step or one clever prompt away from resurfacing. Which is why the whole field leans so heavily on evaluation.
How forgetting is measured
You cannot open a network and point to "the deleted person," so unlearning is judged statistically. The standard tool is the membership-inference attack: a probe that tries to decide whether a given example was in the model's training set, reading signals like how confidently the model predicts it. The logic is elegant — if unlearning worked, a forgotten example should look to the attacker exactly like one the model never saw, so the attack should fail. The NeurIPS 2023 Machine Unlearning Challenge formalised this: it compared the distribution of unlearned models against the distribution of models retrained from scratch, using statistical tests and membership-inference-style tools such as LiRA. Crucially, this is evidence of indistinguishability, not proof of erasure — a distinction the Challenges section returns to.
Types
The field divides cleanly in two, and the split is real — practitioners use these exact words to describe what guarantee they are offering.
-
Exact unlearning produces a model provably equivalent to one retrained without the deleted data. SISA is the canonical example: because each example's influence is isolated to one shard, retraining that shard yields exactly the model you would have gotten without the example. The guarantee is strong; the price is a training scheme you must adopt before any deletion request arrives, plus the accuracy cost of sharding.
-
Approximate unlearning edits an already-trained model to reduce the deleted data's influence, accepting that "reduced" is not "removed." Gradient ascent, retain-set fine-tuning, and representation-level methods like RMU all live here. It works on models that were trained normally, costs a fraction of retraining, and is the only realistic option for a frontier model that was never sharded — but it ships without a guarantee, so it lives or dies on how well it survives adversarial evaluation.
Real-World Applications
The right to erasure. The GDPR's Article 17 gives EU residents a right to have their personal data erased, and California's CPRA grants a similar right to delete. When that personal data has been used to train a model, deleting it from a database is arguably not enough — the model still carries its imprint. This legal pressure, more than any research curiosity, is why unlearning moved from a niche topic to a board-level one. Note that the law's exact demands on trained models remain contested, which is itself part of why the technical target is a moving one.
Copyright takedowns. The wave of lawsuits from authors, news organisations, and artists against model developers has created a second, closely related demand: not "delete this person" but "remove this book, this article, this artist's style" from a model that has demonstrably memorised it. Retraining a foundation model to comply with each takedown is economically impossible, which puts approximate unlearning squarely in the frame as the only plausible remedy.
Safety and dangerous knowledge. Unlearning is now a live safety tool for stripping hazardous capabilities from open-weight models. The WMDP benchmark (Li et al., 2024) makes this measurable: 3,668 multiple-choice questions probing proxy knowledge in biosecurity, cybersecurity, and chemical security, paired with the RMU unlearning method that suppresses a model's performance on those questions while trying to preserve its general ability. The goal is a model that stays useful for chemistry homework but has forgotten how to help synthesise a weapon.
Standardised evaluation. The NeurIPS 2023 Machine Unlearning Challenge — organised by Google with a broad group of academic and industry researchers, and run on Kaggle — is the clearest real deployment of unlearning evaluation. Its scenario was an age-prediction model trained on face images from which a subset had to be forgotten, and its scoring code and metric were open-sourced in May 2024, giving the field its first common yardstick.
Worth stating honestly: there is not yet a widely deployed, production unlearning system inside a frontier LLM that anyone can point to as solved. The applications above are real demands and real research anchors, not a claim that the problem is handled.
Challenges
Diffuse memorisation in LLMs. Everything about unlearning is harder for large language models because there is no "row" to delete. A fact a model knows is not stored in one weight; it is distributed across billions of parameters and reinforced by many overlapping documents that all imply the same thing. Removing one source often leaves the fact fully intact because a dozen others taught it too, and the sharding trick that makes SISA work is impractical at frontier scale, where cutting the data 20 ways would cripple the model.
Verified forgetting is largely unsolved. This is the honest limit of the whole field. For a deep network there is no method to prove that specific knowledge is gone — only to gather statistical evidence that an unlearned model resembles a retrained one. The NeurIPS challenge metric measures a distance between distributions; a passing score means "we could not tell the difference with these attacks," which is not the same as "the information is not there." A stronger attack tomorrow can retroactively invalidate today's confident unlearning.
Forgotten knowledge relearns. Approximate unlearning is often reversible. An adversarial line of work has shown that knowledge suppressed by methods like RMU can frequently be recovered — with a handful of fine-tuning steps on innocuous data, or by prompting that routes around the suppressed representations. If a "forgotten" capability can be restored cheaply, the forgetting was cosmetic, which is a serious problem when the capability is a hazardous one.
Utility collapse. Push unlearning too hard and the model gets worse at everything. Because the deleted information is entangled with information you want to keep, aggressive edits bleed into unrelated abilities — this is the entangled-capabilities problem, and why every serious method reports a retained-task score alongside its forgetting score. It is the same accuracy- versus-deletion tension SISA showed, now inside a single set of weights.
Deletion accumulates. Even SISA's per-request cheapness erodes over a model's life. Each deletion permanently shrinks a shard's data and nudges its accuracy down, so the hundredth request costs more, in quality, than the first. A scheme that is efficient for occasional deletions is not automatically efficient for a steady stream of them.
Future Trends
The most consequential shift is toward standardised, adversarial evaluation. The open-sourcing of the NeurIPS challenge metric in 2024 and subsequent multi-axis benchmarks reflect a field that has learned not to trust a single forgetting score — the direction of travel is to assume an attacker and measure what survives them, because the relearning results showed how easily a weak evaluation is fooled.
Representation-level methods aimed at safety, of which RMU is the current archetype, are the most active research front, precisely because they target the open-weight, hazardous-knowledge case that copyright- and privacy-driven methods do not address. Expect the next generation to be judged mainly on robustness to recovery attacks rather than on clean-benchmark forgetting.
Finally, the legal and technical definitions are converging under pressure. Regulators are being asked what "erasure" means for a trained model, and that answer will shape which guarantee — the strong one from exact methods or the cheap one from approximate methods — counts as compliance. Provable, or certified, approximate unlearning exists today for simple convex models; extending anything like a certificate to deep networks is the open problem whose solution would change the field most.
Note: this term was last reviewed in July 2026. The cited figures — the SISA speedups, the WMDP question count, and the challenge and metric dates — are stated as of that review and are the parts of this page most likely to age.