Generative AI

Definition

Generative AI is the class of AI systems that produce new data — a paragraph, an image, a protein, a weather forecast — by learning the probability distribution the training data was drawn from, and then drawing fresh samples from it. That single sentence is the whole difference from the AI that was already everywhere before ChatGPT: your spam filter, your card-fraud alerts and your recommendation feed are discriminative systems, and they learn something much cheaper.

A spam filter learns P(y|x) — the probability of a label y given an email x. In practice it needs one number per email and a boundary to compare it against. It never has to know what emails look like in general, only which side of the line this one falls on. A generative model has the harder job: it must learn about P(x) itself, the distribution over all possible emails including the ones nobody has ever written, because you cannot sample from a distribution you have not represented. Andrew Ng and Michael Jordan set the two approaches against each other in "On Discriminative vs. Generative Classifiers" (NIPS 2001), and the trade-off they found still holds: the generative model learns from fewer examples but pays for modelling far more than the task requires.

That extra burden is exactly what buys you generation. It is also what breaks. Because the model's only operation is "produce something this distribution would plausibly contain", it has no way to produce nothing. A classifier can return 51% and be honestly unsure; a sampler always returns a sample. That asymmetry is the structural reason hallucinations are a property of the method rather than a bug awaiting a patch — and the first thing to understand before you put generated output anywhere it matters.

The modalities are the least interesting part of the story, because the mechanism is the same in all of them: text generation, image generation, video generation, speech, code and molecules are the same idea pointed at different data. This page is about the idea.

How It Works

The output space is the problem

Start by counting what the model has to choose from. A 512×512 RGB image is 512 × 512 × 3 = 786,432 numbers, each an integer from 0 to 255. The number of distinct images that could come out of that slot is 256^786,432, which is 10^1,893,917 — because 786,432 × 8 × log₁₀(2) ≈ 1,893,917. The observable universe is commonly estimated to hold about 10⁸⁰ atoms. The image space is larger than that by more than 1.8 million orders of magnitude.

Text is no gentler. A 1,000-token reply drawn from GPT-4's cl100k_base vocabulary of 100,277 tokens has 100,277^1,000 ≈ 10^5,001 possible values — a factor of 10^4,921 more options than there are atoms.

Now the important observation: almost every point in those spaces is noise. Draw a 512×512 image uniformly at random and you get static, every time, for as long as you care to keep drawing. Coherent images occupy a vanishingly thin sliver of the space. So the entire job of a generative model is to learn the shape of that sliver well enough to land inside it — and it is why a model that has learned it looks like magic, while the same sampling procedure over an unlearned distribution produces garbage.

Why you cannot just build the table

Take MNIST, the 28×28 8-bit grayscale handwritten-digit dataset, and compare the two jobs on the same data.

The discriminative job — decide which of ten digits this is — is a linear softmax classifier with 784 × 10 + 10 = 7,850 parameters. That is small enough to fit in a spreadsheet, and it gets you to 88% accuracy (12.0% test error, per the linear-classifier row of LeCun's original MNIST benchmark table).

The generative job — represent P(x) over 28×28 images — done naively, means a lookup table with one entry for every possible image: 256⁷⁸⁴ ≈ 10^1,888 cells. This is not "expensive". There is no storage medium, no compression scheme and no amount of money that gets you a table with 10^1,888 entries. 7,850 numbers versus 10^1,888 numbers is the gap generative modelling had to cross, and it is why the field spent decades on classification while generation stayed a curiosity.

There are two ways across, and every generative model in use takes one of them.

Route one: factorise the joint distribution

The chain rule of probability says that any joint distribution can be written as a product of conditionals:

P(x1, x2, ..., xn) = P(x1) · P(x2 | x1) · P(x3 | x1, x2) · ... · P(xn | x1 ... xn-1)
                   = ∏ P(xi | x1 ... xi-1)

This is an identity, not an approximation — it is exactly true for every distribution, always. And it is the single most consequential fact in modern AI, because of what it does to the arithmetic. Instead of one impossible distribution over 10^5,001 sequences, you now have 1,000 tractable distributions over 100,277 values each. Over a full response the model emits 1,000 × 100,277 ≈ 100 million numbers — a rounding error next to 10^5,001 table cells. A single neural network computes each conditional in turn, conditioned on everything it has emitted so far.

That is precisely why an LLM writes left to right, one token at a time, and why it cannot revise what it has already said: each step is one factor in the product. The transformer is a good estimator of P(xi | x1 ... xi-1), and that is all it is.

Claude Shannon did the same factorisation in 1948 with a table of word-pair frequencies, sampling this out of his second-order word approximation in A Mathematical Theory of Communication:

THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER THAT THE CHARACTER OF THIS POINT IS THEREFORE ANOTHER METHOD FOR THE LETTERS THAT THE TIME OF WHO EVER TOLD THE PROBLEM FOR AN UNEXPECTED.

Everything since has been a better estimator of the same conditionals. The recipe did not change.

Route two: go through a bottleneck, or run noise backwards

Pixels have no natural reading order, so factorising an image left-to-right works badly. The alternative is to exploit the fact that although a 512×512 image has 786,432 dimensions, natural images occupy a far lower-dimensional surface inside that space — the manifold hypothesis. An autoencoder learns a compression to a few hundred latent dimensions and a decoder back out, so sampling happens in the small space and gets expanded into the big one. A diffusion model instead learns to reverse a process that gradually adds Gaussian noise: start from pure static, apply the learned denoising step some tens of times, and the trajectory lands on the data manifold. Stable Diffusion combines both, running diffusion inside an autoencoder's latent space rather than on pixels.

Then you have to pick

Learning the distribution and choosing a point from it are separate steps. Given the model's probabilities for the next token, greedy decoding takes the highest, and temperature rescales the distribution before sampling — flatter for variety, sharper for determinism. Note that the model is generative because of what it learned, not because you sampled: greedy decoding at temperature 0 is still reading off a learned P(x), just always at its peak.

Conditioning is what makes it useful

A raw P(x) model produces a plausible something. What you want is a plausible something about your question, which means sampling from P(x | prompt) instead. In an autoregressive model this is nearly free — the prompt is simply the first few xi in the chain, so every subsequent conditional already depends on it. That is the whole reason prompt engineering works: the prompt is not an instruction the model obeys, it is the left-hand side of every conditional probability it computes next.

Types

The genuine taxonomy of generative models is not by modality but by how the model represents the distribution — and the choice determines what you can and cannot ask of it.

• Autoregressive — factorise via the chain rule and predict one element at a time. Gives you exact likelihoods and arbitrary-length output, at the cost of n sequential forward passes for n tokens. Every LLM is here.
• Diffusion — learn to invert a fixed noising process, denoising from static over tens of steps. Parallel across positions rather than sequential, which is why it dominates image generation and is now being tried on text as diffusion language models. Likelihoods are only bounded, not exact.
• Variational autoencoders — force data through a low-dimensional probabilistic bottleneck (Kingma & Welling, 2013). Samples tend to be blurry because the objective averages over reconstructions, but the latent space is smooth and interpolable, which is why VAEs survive as the compression layer inside latent diffusion.
• GANs — train a generator against a discriminator that tries to tell samples from real data (Goodfellow et al., 2014). Sharp outputs, no likelihood at all, and a training procedure prone to mode collapse where the generator covers only part of the distribution.
• Normalizing flows — build the distribution from invertible transforms, so density is computable exactly in both directions. Rarely used for images at scale because invertibility forbids dimensionality reduction, but valued where exact likelihoods matter.

Real-World Applications

Ensemble weather forecasting is the cleanest illustration that generative means sampling, not creating art. DeepMind's GenCast, published in Nature in December 2024, is a diffusion model that produces an ensemble of 50 or more distinct 15-day global forecasts, each a sample from the distribution of possible weather. It beat the ECMWF's ENS — the leading operational system — on 97.2% of 1,320 evaluation targets, and on 99.8% of them at lead times beyond 36 hours, producing a full forecast in about 8 minutes on a single TPU v5 against hours on a supercomputer. A discriminative model returns one number; you cannot price hurricane risk from one number, you need the spread, and the spread is what sampling gives you.

Drug discovery puts the same machinery over chemical space. Insilico Medicine used generative models both to nominate the target (TNIK) and to design the molecule for rentosertib, an idiopathic pulmonary fibrosis candidate. Its Phase IIa results were published in Nature Medicine in June 2025: 71 patients across 22 sites, with the 60 mg once-daily arm showing a mean forced-vital-capacity change of +98.4 mL over 12 weeks against −20.3 mL on placebo. Small and early, but it is a clinical readout on a molecule that came out of a sampler.

Creative tooling is where the volume is. Adobe reported that Firefly had generated over 24 billion assets since its March 2023 launch, as of mid-June 2025 — up from 22 billion announced on 24 April 2025, so about 2 billion assets in seven weeks. Treat those figures as a snapshot; the durable point is that generation at consumer scale is now measured in tens of billions of artefacts, which is why content provenance became an infrastructure problem rather than a research one.

Code assistance is the highest-value text application, and it works for a mechanical reason: source code has far lower entropy than prose, so the next-token conditionals are sharper and the sampled continuation is right more often.

Challenges

• There is no reject option. Sampling from P(x | prompt) always returns something. A model cannot represent "this prompt has no plausible continuation in what I learned", because every prompt has some region of the distribution nearest to it. This is why retrieval, tool use and abstention have to be bolted on from outside — none of them fall out of the generative objective.
• Likelihood and sample quality come apart. Theis, van den Oord and Bethge showed in "A note on the evaluation of generative models" (2016) that a model can score excellent log-likelihood while producing terrible samples, and vice versa — in high dimensions the two are almost independent. So there is no single number that tells you a generative model is good, which is why evaluation leans on human raters and task-specific benchmarks.
• Error compounds along the chain. The factorisation is exact only if each conditional is exact. In practice the model conditions on its own previous samples, so one improbable token early on shifts the rest of the sequence into a region it saw rarely in training, and the next conditionals get worse. This is why a long generation can start well and drift.
• Detection is adversarial by construction. The better a generative model matches the data distribution, the less any detector can distinguish its samples from real data — this is literally the objective a GAN's discriminator loses at when training succeeds. Reliable detection of deepfakes after the fact is fighting the model's training target, which is why signing content at the point of creation is the more tractable approach.
• Generation costs n times what classification costs. A spam verdict is one forward pass. A 1,000-token answer is 1,000 sequential forward passes, each one dependent on the last, so it cannot be parallelised away — only worked around with speculative decoding and better inference optimization. The three-orders-of-magnitude gap in serving cost between discriminative and generative AI is a direct consequence of the chain rule, not an engineering failure.

Future Trends

• Breaking the left-to-right constraint. Diffusion language models abandon the autoregressive factorisation and denoise all positions in parallel across a fixed number of steps, trading sequential token count for parallel step count. If this holds up at scale it changes the cost curve above, since the number of steps does not grow with output length.
• Generative models used as classifiers. With P(x|y) in hand, Bayes gives you P(y|x) ∝ P(x|y)P(y), so a diffusion model can be turned into a classifier by asking which class label makes the observed image most likely. Ng and Jordan's 2001 result — generative approaches converge faster from fewer examples — makes this attractive exactly where labelled data is scarce, and it collapses the distinction this page opened with.
• Provenance at generation time rather than detection after it. Because detection is adversarially hopeless in the limit, effort is moving to cryptographic signing of content as it is produced, via C2PA content credentials. This is a bet that the metadata survives distribution — which so far it frequently does not.

Code Example

Two things worth seeing in code: how large the space is, and how little machinery it takes to be genuinely generative.

from math import log10

# 1. The size of the output space.
pixels = 512 * 512 * 3                # 786,432 numbers, each 0-255
print(f"512x512 RGB images: 10^{pixels * log10(256):,.0f}")   # 10^1,893,917

tokens, vocab = 1000, 100_277         # a 1,000-token reply, GPT-4 cl100k_base
print(f"1,000-token replies: 10^{tokens * log10(vocab):,.0f}")  # 10^5,001

# Atoms in the observable universe: about 10^80.

import random
from collections import defaultdict

# 2. A complete generative model: learn P(next | current), then sample the chain.
corpus = "the cat sat on the mat. the cat ate the rat. the rat sat on the mat."

counts = defaultdict(lambda: defaultdict(int))
for a, b in zip(corpus, corpus[1:]):
    counts[a][b] += 1                 # this IS the training run

def sample_next(c):
    """Draw from P(next | c) in proportion to the counts."""
    options = counts[c]
    r = random.random() * sum(options.values())
    for ch, n in options.items():
        r -= n
        if r <= 0:
            return ch
    return " "

out = "t"
for _ in range(60):
    out += sample_next(out[-1])       # condition on what you just emitted
print(out)
# one run: "the cathe the athe the athe at the cat on on sat. on t."

The corpus has 12 distinct characters, so the entire model is at most 12 × 12 = 144 numbers — and it already emits strings that appear nowhere in its training data, because the chain visits combinations of transitions the corpus never had in that order. It is bad because a one-character context is a terrible estimator of P(next | everything before), not because the idea is wrong.

Replace the count table with a transformer, the single-character context with tens of thousands of tokens, and the 144 numbers with hundreds of billions of parameters, and you have an LLM. Nothing else in the recipe changes.