Definition
Synthetic data is data produced by a model or a simulation rather than collected from the world. A generative model learns an approximation of some real distribution — images, text, patient records — and then emits fresh samples from that approximation, which are used to train, test or fine-tune another model.
The distinction that matters most is against data augmentation, and it is easy to blur. Augmentation transforms an example you already have: rotate a photo, paraphrase a sentence, add noise to an audio clip. There is a real datum underneath every augmented one, and its label comes along for free — a rotated cat is still a cat. Synthetic data has no real datum underneath it. It is a sample from another model, so it inherits that model's blind spots, and its "label" is an assertion the generator makes, not an observation anyone verified. That single difference — transform of a real thing versus fresh draw from a model — is what makes synthetic data both far more flexible and far more dangerous. The specific danger has a name, model collapse, and it is the reason this term is worth understanding before you reach for it.
How It Works
Synthetic data is generated three broad ways, and they do not carry the same risk.
The first is sampling from a generative model trained on the same kind of data you want more of — a GAN or diffusion model for images, a large language model for text. The second is distillation: you take a stronger "teacher" model, have it produce inputs and answers, and fine-tune a smaller "student" on that transcript — the student learns to imitate the teacher for a fraction of the cost. (This is closely related to knowledge distillation, and it is how a great deal of open-model instruction tuning and supervised fine-tuning is actually done.) The third is simulation: a physics engine or game world renders labelled examples — a self-driving stack sees a rendered pedestrian, and the simulator already knows the exact bounding box because it drew it.
The first two are model-generated and share a failure mode; simulation is grounded in a hand-built world rather than in a learned model of your data, so it does not feed on itself the same way.
The mechanism that breaks: statistical approximation error
The 2024 Nature paper that formalised the risk (Shumailov et al., "AI models collapse when trained on recursively generated data") isolates the primary cause and it is not exotic. It is finite sampling. Any finite sample can, by chance, miss a low-probability event — and the model trained on that sample can no longer produce what its training data did not contain. Do this repeatedly — train a model, sample from it, train the next model on those samples — and each round shaves a little more off the tails of the distribution. The paper's phrase is that the process "first starts with tails disappearing"; carried far enough it converges to what they prove is a delta function, a single point with zero variance. They call the two stages early model collapse (the tails go) and late model collapse (the distribution narrows to near-nothing).
The arithmetic is worth doing once, because it shows the loss is not gradual. Suppose a genuine but
uncommon pattern — a rare disease presentation, an unusual code idiom, a minority dialect — appears in
about 1 in 1,000 real examples, so its probability is p = 0.001. A model trained on that data
reproduces it at roughly the same rate. Now generate a synthetic corpus of n = 1,000 examples to
train the next model. The expected number of examples containing the pattern is n × p = 1 — but
the probability that the corpus contains none of it is:
(1 − p)ⁿ = (0.999)^1000 ≈ 1/e ≈ 0.37
So a single "train, generate, retrain" round has about a 37% chance of dropping that pattern
entirely. And because a model cannot regenerate what it never saw, the loss is one-way: once the
pattern is gone from a generation's data it is gone for good. Chain the rounds and the survival
probability is roughly 0.63 per generation, so after ten it is 0.63¹⁰ ≈ 0.01 — about a 1% chance
the pattern is still there. This is exactly the Nature paper's discrete argument, where a state with
probability at or below 1/M (M being the sample size) is expected to vanish: the tail does not fade,
it falls off a cliff, one rare feature at a time.
Empirically the authors demonstrated it on language. They fine-tuned Meta's OPT-125M model on the wikitext2 corpus, generated a same-size dataset from it, fine-tuned the next generation on that, and repeated for nine generations. With no real data retained, quality degraded monotonically across the generations; keeping a slice of the original data each round left only minor degradation. Crucially, one distillation pass is not collapse — collapse is what happens when a model's own output becomes the training set for its successor, again and again, with the real distribution drained out of the loop.
Real-World Applications
Synthetic data is not a research curiosity; it is in production wherever real data is scarce, unsafe to collect, or illegal to share.
- Distilling a stronger model. Stanford's Alpaca (2023) fine-tuned a base LLaMA model on about 52,000 instruction-response examples generated by an OpenAI model, at a data cost of a few hundred dollars. A large share of today's open instruction- and chat-tuned models are trained this way — the student cannot exceed the teacher, but it can get most of the way there cheaply. This is the "filling in behaviour by copying a stronger model" use, and it works precisely because it is a single pass, not a recursive loop.
- Curated synthetic pretraining. Microsoft's Phi models ("Textbooks Are All You Need", 2023 onward) were trained substantially on synthetically generated textbook-style text and exercises, and a small Phi model matched far larger ones on reasoning and code. It was a headline demonstration that curated synthetic data can beat raw web scrape — when a stronger model and heavy filtering stand between the generator and the training set.
- Rare and dangerous edge cases. Autonomous-driving programmes (Waymo, Cruise) and robotics teams train and test perception on simulation — NVIDIA's Omniverse Replicator and Isaac Sim render labelled scenes on demand. A child darting between parked cars is unsafe and rare to capture on the road, but trivial to render a million times with pixel-perfect labels, because the simulator knows the ground truth it drew.
- Privacy-preserving records. In finance and healthcare the real rows cannot leave the building. Tools such as MOSTLY AI, Gretel and the open-source Synthetic Data Vault fit a generative model to a sensitive table and emit statistically similar fake rows that can be shared for analytics and model development — with the caveat, below, that "synthetic" is not automatically "private."
Key Concepts
The real-data anchor. The single most important dial is how much genuine data stays in the loop. The Nature study's mild-degradation setting kept 10% of the original data at every generation; a growing line of follow-up work finds that accumulating real and synthetic data, rather than replacing real with synthetic, arrests the drift. Synthetic data is a supplement, and it behaves very differently as a replacement.
Coverage versus fidelity. A synthetic example can look completely realistic on its own while the set systematically under-represents the tails. This is what makes collapse insidious: the failure is at the distribution level, not the example level, so eyeballing a handful of convincing samples tells you nothing about whether the rare cases are still there.
Label fidelity. A synthetic label is a claim, not a measurement. The robust pattern — the same one the data augmentation page recommends for generative augmentation — keeps a real example or an independent verifier in the loop rather than trusting a class name and a generator.
Challenges
Model collapse is the headline risk, and it is specifically the recursive regime. Training a model on its own (or other models') output, generation after generation, with the real distribution drained out, drives the tails to zero — the mechanism worked through above. This is a real and, as of the 2024 finding, well-characterised threat as the open web fills with model-generated text and images and clean human data becomes harder to isolate. A single distillation pass is not this; recursive self-training is.
Narrowing shows up on the tail, not the average. Even one generation quietly trims minority modes. Standard validation is dominated by the common cases, so an aggregate metric can look healthy while rare-case performance rots — you notice it on the unusual query, the underrepresented group, the long tail, which is exactly where a validation set is thinnest. It reads less like a loud failure and more like a slow, invisible overfitting to the head of the distribution.
Errors compound instead of averaging out. Any systematic mistake or bias in the generator becomes ground truth for the next model, which reproduces and amplifies it. This is adjacent to data poisoning: there the corruption is adversarial and deliberate, here it is self-inflicted, but in both a bad signal in the training data propagates forward and is hard to undo.
Synthetic is not automatically private. A generative model can memorise and regurgitate the real records it was fit on, and membership-inference and reconstruction attacks can recover individuals from "anonymous" synthetic tables. Privacy has to be enforced with a mechanism — differential privacy during training, for instance — not assumed from the fact that the output is generated. Treating synthetic data as safe to share by default is a privacy failure waiting to happen.
Provenance is getting harder. The Nature authors' closing point is economic: as model output saturates the web, the value of data provably collected from genuine human interaction goes up, and the ability to curate a clean real corpus — or even to tell human from machine text — goes down. That makes the real-data anchor above not just a training choice but a supply problem that gets worse over time.
Note: this term was last reviewed in July 2026. The named systems, tools and dated results cited above (the OPT-125M experiment, the Phi and Alpaca examples, the specific privacy tools) are the parts of this page most likely to age; the finite-sampling mechanism and its arithmetic do not.