Generative Adversarial Network (GAN)

Definition

A generative adversarial network trains two neural networks against each other. The generator turns a vector of random numbers into something that should pass for real data — a face, a second of speech, a high-resolution version of a blurry photo. The discriminator is shown a mixture of genuine examples and the generator's output and has to say which is which. Whatever makes one of them look good makes the other look bad.

The part that is easy to miss, and that explains everything else on this page: the generator never sees a real example. There is no target output anywhere in its loss. It is never told "your face should have looked like this face." It is told "the judge scored your face at 0.3, and here is the direction to move your weights to make that number larger" — and the judge is itself a network that will have changed by the next step. The training signal is another model's opinion, and it is a moving one.

You can read off what success looks like as a number. If the generator's distribution ever matched the data exactly, the discriminator would have nothing left to detect and its best possible answer for every input would be 1/2, a coin flip (Goodfellow et al., 2014). Substitute 1/2 into the binary cross-entropy it is minimising and the discriminator's loss settles at 2 × ln 2 = 1.386, while the generator's settles at ln 2 = 0.693. Those are constants. They are the same two constants whether the generator is producing photographs or grey mush — which is why "the loss went down" is not evidence that a GAN is working, and why the usual way of telling whether training is going well simply does not exist here.

GANs arrived in 2014 and were the default way to synthesise images for most of the following decade. Diffusion models displaced them, and the reason is short: a diffusion model's training objective is an ordinary regression on added noise, with no second player, so it has a fixed loss landscape, scales to large models and datasets predictably, and has no incentive to cover only part of the data. That comparison belongs to the diffusion side of the story. What matters here is that the adversarial idea did not disappear with the architecture. It survives as a loss term inside systems that are not GANs at all, and it is still the technique of choice whenever a sample has to be produced in a single forward pass.

How It Works

One number, two opposite goals

The discriminator D takes a data point and outputs a probability that it is real. The generator G takes a noise vector z — a short list of random numbers drawn from a fixed simple distribution, the only source of variety it has — and outputs a data point. Training optimises a single quantity in two directions at once:

min_G  max_D   E_x[ log D(x) ]  +  E_z[ log(1 - D(G(z))) ]

D wants that expression large: score real data near 1, score generated data near 0. G wants it small, and it can only touch the second term. This is a two-player zero-sum game, and it is worth being precise about what that costs, because it is the source of every difficulty below. Ordinary supervised training walks downhill on a landscape that stays where it is. Here there is no landscape that stays put: each network's loss surface is redrawn by the other network's last update. The solution being sought is not a minimum but a saddle point — a place where neither player can improve unilaterally. Gradient descent was designed to find minima. Nobody has a comparable guarantee for saddle points of non-convex games.

The alternating loop

In practice you do not solve either side to completion. The original algorithm alternates k discriminator updates with one generator update, and the paper notes that its own experiments used k = 1 — the cheapest setting. One iteration looks like this:

1. Draw a batch of real examples and a batch of noise vectors; run the noise through G to get fakes.
2. Update D to push its score up on the real batch and down on the fake batch. G is frozen.
3. Draw fresh noise, generate again, and update G to push D's score on those fakes up. D is frozen, but the gradient travels through it: the discriminator is the differentiable path by which the generator finds out what "more realistic" means in weight space.

Step 3 is the whole trick. The discriminator is not just a scorer, it is the loss function — a learned one, which is why it can supply a useful gradient for something as hard to write down as "looks like a photograph".

Why the first version of the generator loss does not work

Written literally, G minimises log(1 - D(G(z))). Differentiate that with respect to the discriminator's pre-sigmoid output and you get exactly -D(G(z)). Early in training the discriminator rejects the generator's output confidently, D(G(z)) is near 0, and so is the gradient: the generator learns nothing precisely when it has the most to learn. The 2014 paper spotted this and prescribed the fix in one sentence — "Rather than training G to minimize log(1 − D(G(z))) we can train G to maximize log D(G(z))." Differentiate that and the magnitude is 1 - D(G(z)), which approaches 1 exactly where the original approached 0. Same fixed point, usable gradients. This non-saturating loss is what essentially every GAN implementation actually runs. Put a number on the difference: in the collapsed run at the bottom of this page the discriminator ends up scoring fakes at about 0.0003, where the saturating form would deliver a gradient of magnitude 0.0003 and the non-saturating form one of 0.9997 — a factor of over 3,000, from a one-line change.

What the game is really measuring

For any fixed generator, the best possible discriminator is D*(x) = p_data(x) / (p_data(x) + p_g(x)) — the posterior probability that a point came from the data. Substituting it back into the objective collapses to something interpretable:

C(G) = -log 4 + 2 · JSD(p_data ‖ p_g)

where JSD is the Jensen–Shannon divergence. So a GAN with a well-trained discriminator is minimising a symmetric statistical distance between the generated and real distributions, and its optimum is −log 4 ≈ −1.386, reached only when the two distributions are identical. That is a satisfying result and also a trap: the guarantee holds for an optimal discriminator over arbitrary functions, and a discriminator you actually trained for 200 steps is neither optimal nor arbitrary. The gap between the theory and the loop is where the failures live.

Real-World Applications

Speech synthesis, on hardware that could never run the alternative. Almost every modern text-to-speech system generates a mel-spectrogram and then needs a vocoder to turn it into a waveform, and that vocoder is usually adversarially trained. The arithmetic is the argument. WaveNet, the autoregressive predecessor, generates audio one sample at a time: in the HiFi-GAN comparison it produced 0.07 kHz on an NVIDIA V100, meaning 70 samples per second of GPU time for audio that needs 22,050 of them — roughly 315 seconds of datacenter GPU per second of speech. HiFi-GAN's smallest variant produced 296.38 kHz on a MacBook Pro laptop CPU, or 13.4 seconds of audio per second of compute, at a mean opinion score of 4.05 ± 0.08 against WaveNet's 4.02 ± 0.08 (Kong et al., 2020). Comparable quality, roughly 4,200 times the throughput (296.38 ÷ 0.07), on a laptop instead of a server. That gap is not an implementation detail — it is the difference between a feature that can run on a phone and one that cannot.

Super-resolution, where the pixel metric and the human disagree. SRGAN upscaled images 4× and reported something instructive on the Set14 benchmark: the adversarially trained model scored 26.02 dB PSNR against the non-adversarial baseline's 28.49 dB, while its mean opinion score rose from 2.98 to 3.72 (Ledig et al., 2017). The adversarial version is measurably worse by the pixel-error metric and clearly better to people. The reason is the same one that makes autoencoders blurry: when several outputs are plausible, minimising squared error rewards predicting their average, and the average of several plausible textures is mush. A discriminator does not accept mush, because mush is easy to catch. This is why GAN-based upscalers — SRGAN's open-source descendants ESRGAN (2018) and Real-ESRGAN (2021) among them — remain the ones embedded in photo and video tooling where the result has to appear while the user waits.

Inside the models that replaced them. The autoencoder that compresses images into the latent space of a latent diffusion model is itself trained adversarially — the Stable Diffusion paper's appendix states plainly that all its autoencoder models are trained "in an adversarial manner ... such that a patch-based discriminator is optimized to differentiate original images from reconstructions" (Rombach et al., 2022). Neural audio codecs use the same construction. The adversarial term is what stops the decoder from producing the smoothed output that a reconstruction loss alone would give it. So a great many pipelines described as "not GANs" contain a discriminator doing exactly the job described on this page.

Photorealistic faces, and the harm that followed. The 1024 × 1024 synthetic faces that made GANs famous around 2018–2019 came from this family; StyleGAN was trained on FFHQ, a dataset of 70,000 images at 1024² resolution (Karras et al., 2018). The same capability is the origin of the modern deepfake problem, and it illustrates the adversarial dynamic outside of training too: a deepfake detector is a discriminator, so building better detectors is, structurally, the same loop that made the generators good in the first place.

Challenges

Every difficulty below traces back to one fact: the objective is defined relative to an opponent that moves. These are not rough edges that better engineering has been slowly filing down. They are what a two-player game does.

Mode collapse. The generator discovers a narrow region of output space that the current discriminator scores well, and produces only that — one convincing face instead of the variety in the training set. Note that this is not a mistake in the generator's terms: every one of those samples does fool the present judge, which is exactly what the objective asked for. The 2014 paper already had a name for it, warning that G must not be trained too far ahead of D "in order to avoid 'the Helvetica scenario' in which G collapses too many values of z to the same value of x." The code below shows it happening: the same script, with only the width of the real data's clusters changed, covers 8 of 8 modes in one run and 2 of 8 in the other.

Vanishing gradients when the discriminator wins. This is the failure with the cleanest theory behind it. Real data lies on a thin, curved surface inside a much higher-dimensional pixel space, and so does the generator's output, and two such surfaces generically do not overlap. Arjovsky and Bottou proved that when the two supports fail to align, the Jensen–Shannon divergence is pinned at its maximum value of log 2 ≈ 0.693 regardless of how close the distributions actually are — and that as the discriminator approaches optimality, the gradient reaching the generator goes to zero (Arjovsky and Bottou, 2017). They measured it: with the generator frozen and a discriminator trained from scratch against it, gradient norms decayed over 4,000 discriminator iterations by five orders of magnitude in the best of the cases they plot. A discriminator that is too good is as useless as one that is too bad, and there is no way to know from the loss which side of that line you are on.

Non-convergence and oscillation. Even with no collapse and healthy gradients, the pair can circle a solution indefinitely. Each update is a best response to a snapshot of an opponent that will have moved by the time it lands, and simultaneous gradient steps in a game can rotate around an equilibrium rather than approach it. In practice this shows up as sample quality that visibly improves, then degrades, then improves again, with a loss curve that reports nothing unusual throughout.

There is no loss curve that tells you training is going well. Everywhere else in deep learning the validation loss is the instrument: it tells you when to stop, which run is better, whether a hyperparameter helped. A GAN takes that instrument away, and it is worth being concrete about why. At the healthy equilibrium the discriminator loss is 1.386 and the generator loss is 0.693 — but those are also roughly the numbers you see from a discriminator that has simply failed to learn anything. Meanwhile the Wasserstein GAN paper plotted the standard objective's estimate of the JS distance during ordinary GAN training and reported that the quantity "clearly correlates poorly" with sample quality and "often remains very close to log 2 ≈ 0.69, which is the highest value taken by the JS distance" while the samples were visibly getting better (Arjovsky et al., 2017). So: you cannot early-stop on the loss, cannot rank two runs by it, and cannot tune a learning rate against it. Teams fall back on looking at samples and on proxy metrics such as FID, both of which are noisier and slower than the number that any other model would just hand you.

No likelihood, either. A GAN is an implicit model: it defines a distribution only through the process of sampling from it, and never represents a density. You cannot ask it how probable a particular image is, which rules out held-out likelihood as an evaluation and rules out the whole family of uses — anomaly scoring, compression, model comparison — that depend on a density. The 2014 paper had to evaluate with a Parzen-window estimate, reporting 225 ± 2 on MNIST against a best prior result of 214 ± 1.1; that method is no longer trusted, and nothing has replaced it that is not also a proxy.

Balance is a hyperparameter with no principled setting. How many discriminator steps per generator step, how large each learning rate, how much capacity each network gets: these interact, they are dataset-dependent, and the signal you would normally use to tune them is the one that does not work here. The code below changes exactly one property of the data — nothing about the architecture, the learning rate or the number of steps — and moves the run from healthy to collapsed.

Future Trends

The interesting question is no longer whether GANs will reclaim image synthesis. It is which parts of the adversarial idea outlive the architecture, and there the answer is already visible in shipping systems.

The discriminator survives as a loss term. Adversarial training turns out to be most useful in small doses: a patch discriminator added to a reconstruction loss, contributing the sharpness that squared error cannot, while the reconstruction term supplies the stability that a pure adversarial objective lacks. That is the configuration inside latent diffusion decoders and neural audio codecs, and it is a genuinely different engineering proposition from training a GAN end to end, because the auxiliary losses keep the game from running away. Expect adversarial terms to keep spreading in this role while whole-model GANs stay rare.

One-step generation is a durable niche. A GAN generator is a single forward pass. Iterative samplers are not, and no amount of scheduler cleverness makes a many-step sampler free — for reference, the original denoising diffusion model used T = 1,000 network evaluations per sample (Ho et al., 2020). Anywhere the latency budget is a few milliseconds and the hardware is a phone or a browser, the arithmetic still favours one pass, which is why vocoders, upscalers and real-time video effects have not moved. The active line of work is adversarial distillation: training a fast one-step or few-step generator against a discriminator to imitate a slow iterative teacher. That work uses this page's machinery, keeps its instability problems, and is aimed squarely at the latency budget.

The instability is not going to be solved, only managed. A decade of proposed fixes — different divergences, gradient penalties, spectral normalisation, two-timescale updates — has made adversarial training usable rather than reliable. That should be read as information about the problem rather than about the effort spent on it: a saddle point of a non-convex two-player game does not admit the guarantees that minimisation does, so the honest expectation is better recipes, not a solved training procedure.

Code Example

This trains a complete GAN in NumPy — no framework, both networks visible — on a distribution whose modes can be counted. The real data is eight Gaussian blobs arranged on a circle, so "did the generator cover the distribution?" has an exact answer between 0 and 8.

The script runs the identical training loop twice, changing only spread: how wide the eight real blobs are. Nothing else differs — same architecture, same learning rate, same 8,000 steps, same seed.

import numpy as np

# Eight tight Gaussian blobs on a circle of radius 2 — the real distribution
# has exactly eight modes, so "did the generator cover them?" is countable.
ANG = np.arange(8) * np.pi / 4
MODES = np.stack([np.cos(ANG), np.sin(ANG)], 1) * 2.0
sig = lambda u: 1 / (1 + np.exp(-u))

def fwd(W, x):                          # one hidden tanh layer, with biases
    h = np.tanh(x @ W[0] + W[1])
    return h, h @ W[2] + W[3]

def back(W, x, h, dl):                  # weight grads, and grad w.r.t. the input
    dh = (dl @ W[2].T) * (1 - h ** 2)
    return [x.T @ dh, dh.sum(0), h.T @ dl, dl.sum(0)], dh @ W[0].T

def train(spread, iters=8000, n=256, lr=0.05, hid=64, seed=0):
    rng = np.random.default_rng(seed)
    real = lambda m: MODES[rng.integers(0, 8, m)] + spread * rng.normal(size=(m, 2))
    net = lambda ni, no: [rng.normal(size=(ni, hid)) * 0.5, np.zeros(hid),
                          rng.normal(size=(hid, no)) * 0.5, np.zeros(no)]
    G, D = net(8, 2), net(2, 1)
    vG, vD = [np.zeros_like(w) for w in G], [np.zeros_like(w) for w in D]

    def step(W, v, g):                  # SGD with momentum
        for i in range(len(W)):
            v[i] = 0.9 * v[i] - lr * g[i]
            W[i] += v[i]

    for it in range(1, iters + 1):
        # 1. Discriminator step: push D up on real data, down on the fakes.
        x, z = real(n), rng.normal(size=(n, 8))
        _, fake = fwd(G, z)
        hr, sr = fwd(D, x)
        hf, sf = fwd(D, fake)
        ga, _ = back(D, x, hr, (sig(sr) - 1) / n)
        gb, _ = back(D, fake, hf, sig(sf) / n)
        step(D, vD, [a + b for a, b in zip(ga, gb)])

        # 2. Generator step: same machinery, opposite goal — push D(fake) UP.
        #    The gradient is routed through D and on into G's weights.
        z = rng.normal(size=(n, 8))
        hg, fake = fwd(G, z)
        hf, sf = fwd(D, fake)
        _, dx = back(D, fake, hf, (sig(sf) - 1) / n)   # non-saturating -log D(G(z))
        step(G, vG, back(G, z, hg, dx)[0])

        if it % 2000 == 0:
            d_loss = -np.mean(np.log(sig(sr)) + np.log(1 - sig(sf)))
            g_loss = -np.mean(np.log(sig(sf)))
            s = fwd(G, rng.normal(size=(8192, 8)))[1]
            hit = np.argmin(((s[:, None] - MODES[None]) ** 2).sum(-1), 1)
            covered = (np.bincount(hit, minlength=8) / len(s) > 0.02).sum()
            print(f"  spread {spread}  step {it:5d}   D loss {d_loss:.3f}"
                  f"   G loss {g_loss:.3f}   modes covered {covered}/8")

for spread in (0.25, 0.08):
    train(spread)
    print()

Output:

  spread 0.25  step  2000   D loss 1.382   G loss 0.714   modes covered 8/8
  spread 0.25  step  4000   D loss 1.375   G loss 0.695   modes covered 8/8
  spread 0.25  step  6000   D loss 1.412   G loss 0.679   modes covered 8/8
  spread 0.25  step  8000   D loss 1.384   G loss 0.723   modes covered 8/8

  spread 0.08  step  2000   D loss 0.025   G loss 6.771   modes covered 2/8
  spread 0.08  step  4000   D loss 0.022   G loss 8.150   modes covered 2/8
  spread 0.08  step  6000   D loss 0.008   G loss 7.053   modes covered 2/8
  spread 0.08  step  8000   D loss 0.007   G loss 8.122   modes covered 2/8

The first run is the textbook picture. The discriminator loss sits at 1.38 and the generator loss at 0.70, against theoretical equilibrium values of 2·ln 2 = 1.386 and ln 2 = 0.693, and the generator covers all eight modes. Watch the two columns move against each other across the checkpoints. At step 6,000 the discriminator is having a bad moment — its loss is 1.412, above the 1.386 it would get by guessing — and the generator's is correspondingly below equilibrium at 0.679, meaning its fakes are being scored slightly above 0.5. By step 8,000 the pair has swapped: 1.384 and 0.723. Neither number is converging to anything. They are trading.

The second run collapsed, and the only change was making the eight real blobs three times tighter. Tighter clusters mean the real data occupies a thinner region, which the discriminator can separate from the generator's output almost perfectly — its loss falls to 0.007, and a generator loss of 8.12 means it is scoring D(fake) = e^-8.12 ≈ 0.0003. That is the vanishing-gradient regime from the Challenges section, arrived at without touching a single hyperparameter. The generator, unable to make progress broadly, has retreated to 2 of the 8 modes and stays there.

Two things about that second output are worth sitting with. First, the discriminator loss of 0.007 looks excellent by every instinct trained on ordinary supervised learning — a loss near zero is normally the goal, and here it is the alarm. Second, nothing in either number tells you which failure you have: a generator loss of 8 is consistent with mode collapse, with a discriminator that has memorised the training batch, and with a generator whose weights have diverged. You have to go and look at the samples, which is what the mode count in the third column is standing in for and what practitioners do by eye.