Image Generation

Definition

Image generation is the production of a picture by a model that starts from a grid of pure random noise and takes the noise away in steps, with a text prompt steering every subtraction. Nothing is drawn, composited or retrieved. The model performs exactly one operation — given this noisy array, predict what part of it is noise — a few dozen times in a row, and the image is what is left over.

That single loop is behind essentially every system people mean by the phrase: Stable Diffusion, Midjourney, DALL·E, Firefly, Imagen, FLUX. They differ in the size of the network, the text encoder, the training data and the license, and not much in the mechanism. If you came here to pick one rather than to understand one, the model catalog is the better destination — Stable Diffusion 3.5 (October 2024) is the open-weights model most people still run on their own hardware, and carries its versions, licence terms and hardware requirements. It is no longer the quality frontier, though: by 2026 that has moved to newer catalog entries like Seedream 5.0, Qwen-Image 2.0 and HunyuanImage 3.0 and to closed products such as Midjourney, while FLUX.1 and the Stable Diffusion tooling page cover the open-tooling side. This page is the mechanism, and the mechanism is what makes the two knobs in every one of those interfaces make sense.

Those knobs are why this is worth ten minutes. A generation that comes out washed-out, over-saturated, weirdly uniform across four samples, or subtly smudged in the fine detail is usually not the model failing. It is a step count too low, a guidance scale too high, or a latent decoder running out of resolution — three distinct causes with three different fixes, and they are indistinguishable from the outside unless you know what the loop is doing. Generative AI covers the prior question of why sampling from a learned distribution is a different kind of thing from classification; this page takes that as given and asks how a denoiser finds the vanishingly thin part of image space that looks like a photograph.

How It Works

The forward process: wreck the image on a fixed schedule

Training begins with a procedure that involves no learning at all. Take a real image x0, pick a step number t, and mix in Gaussian noise:

x_t = sqrt(alpha_bar_t) * x0  +  sqrt(1 - alpha_bar_t) * epsilon

alpha_bar_t is the fraction of the original signal's power that survives to step t, so sqrt(alpha_bar_t) is the fraction of the signal itself. It is fixed in advance by a noise schedule and never learned. Ho, Jain and Abbeel's original Denoising Diffusion Probabilistic Models (2020) set T = 1000 and let the per-step variance beta rise linearly from 0.0001 to 0.02, which is still the default in OpenAI's reference implementation today.

Run that schedule and print it, and the shape of the whole method appears:

Half the signal is gone by step 367, and 327 of the 1,000 steps have under 10% of the image left in them. That last third is nearly indistinguishable from static, which is the observation Nichol and Dhariwal turned into the cosine schedule — alpha_bar_t = cos²((t/T + 0.008)/1.008 · π/2) — where the signal does not halve until step 664 and the destruction is spread evenly across the chain instead of being crammed into the first third. It is also why samplers can skip most of the tail without visible harm: there is nothing there to reconstruct.

The important property is that this is a closed form. Any x_t, for any t, is one line of arithmetic away from x0. There is no need to simulate 800 steps to get a training example at step 800, which is what makes the training loop cheap: sample an image, sample a t, sample noise, mix, done.

The reverse process: the only thing the network is ever asked

The network — a U-Net in Stable Diffusion 1.x, a transformer in the newer MMDiT models — receives x_t, the step number t, and an encoding of the prompt, and predicts epsilon: the noise that was added. The loss is the squared error between its guess and the noise you actually mixed in. That is the entire training objective. There is no adversary, no reconstruction term, no perceptual loss, no notion of "good picture" anywhere in it.

Sampling inverts the same equation. Given the predicted noise, algebra gives you the model's implied guess at the clean image, and you re-noise that guess to the next lower step rather than jumping straight to zero:

x0_hat  = (x_t - sqrt(1 - alpha_bar_t) * eps_hat) / sqrt(alpha_bar_t)
x_{t-1} = sqrt(alpha_bar_{t-1}) * x0_hat + sqrt(1 - alpha_bar_{t-1}) * eps_hat

Taking a small step rather than the whole way is the point. The denoiser is only reliable at the noise level it was shown; its estimate of x0 from near-total static is a blurry average of every image it could plausibly be. Each step improves the estimate slightly, which lets the next step be a better one. The ## Code Example below runs exactly this loop on a one-dimensional toy where the correct answer is known in closed form, so you can watch it converge.

Where the prompt gets in

The prompt is tokenized and pushed through a frozen text encoder — CLIP ViT-L/14 in Stable Diffusion 1.x, larger encoders including T5 in later models — producing a sequence of embeddings. Those embeddings enter the denoiser through cross-attention layers: at every spatial position, the network attends over the prompt tokens and adjusts its noise prediction accordingly. This is the same transformer machinery that language models use, pointed at a grid instead of a sentence.

This is also the honest explanation of why prompt engineering works and why it is finicky. The prompt is not an instruction that gets obeyed; it is a set of vectors that bias 20 to 50 successive noise predictions. Word order, weighting and even token count change those vectors, and the effect compounds over the loop.

Guidance: two predictions per step, and how hard you lean on the difference

Conditioning alone turns out to be too gentle — early conditional diffusion models produced images that matched the prompt only loosely. Classifier-free guidance (Ho and Salimans, 2022) fixed this by training one network to do both jobs: during training the prompt is replaced by a null token some fraction of the time, so the same weights learn the conditional and unconditional denoiser. At sampling time you run both and extrapolate:

eps_guided = (1 + w) * eps(x_t, prompt) - w * eps(x_t, null)

At w = 0 you get plain conditional sampling. Above that you are amplifying the difference the prompt makes and pushing past it. Their ImageNet 64×64 results show the trade exactly: at w = 0 the model scored FID 1.80 and Inception Score 53.71; at w = 0.1, FID improved to 1.55 and IS to 66.11; by w = 4.0 IS had climbed to 260.2 while FID had collapsed to 26.22. Every individual sample looked more like its class, and the set of samples had stopped covering the class.

Two consequences you pay for directly. First, guidance doubles inference cost — two network evaluations per step, so the diffusers default of 50 steps at guidance_scale = 7.5 is 100 forward passes per image. Second, the over-saturated, high-contrast, everything-looks-the-same output people associate with "AI images" is largely a guidance artifact rather than a training-data one. The toy model below reproduces it in one dimension: asking for a class that truly is N(1.00, 0.50) returns samples with mean 0.999 and standard deviation 0.470 at w = 0, and mean 2.145 with standard deviation 0.225 at w = 7.5. The mode has been pushed to more than twice the true class mean and the spread has halved.

Why none of this happens on pixels

Run the loop on a 512×512 RGB image and every step processes 786,432 numbers. Stable Diffusion does not. An autoencoder compresses the image first, with a downsampling factor of 8 in each spatial dimension and 4 latent channels — the shipped vae/config.json for Stable Diffusion v1 lists latent_channels: 4 and four downsampling blocks, and the U-Net's sample_size is 64. So:

• 786,432 numbers become 16,384 — a factor of 48.
• 262,144 spatial positions become 4,096 — a factor of 64. Convolutional work scales linearly with that, so every convolution in the denoiser does 64× less arithmetic.
• Attention is worse than linear. Self-attention over 262,144 positions is 6.87 × 10¹⁰ pairs; over 4,096 positions it is 1.68 × 10⁷. That is a factor of 4,096.

The 48× and the 64× differ because the latent trades spatial resolution for channel depth — 64× fewer positions, but 4 channels instead of 3. And the saving is paid per step, so at 100 network evaluations per image it is the difference between a datacenter job and a laptop one. Rombach et al.'s Latent Diffusion paper (CVPR 2022) is where this was established; the practical proof is that Stable Diffusion shipped as a download rather than an API.

The decoder runs once, at the end. That asymmetry — expensive loop on the small representation, single expensive decode at full size — is the standard architecture for video generation too, where the same trick is applied to time as well as space.

Steps are the price, and the price fell fast

The original DDPM formulation ran the reverse chain the full 1,000 steps. Song, Meng and Ermon's DDIM (2020) reinterpreted the process as a deterministic non-Markovian one that can be evaluated on any subsequence of steps — using the same trained weights, with no retraining. Their CIFAR-10 FID by step count:

The second row is the control, and it is the interesting one: shortening the stochastic sampler to 10 steps produces FID 367 — noise. The same weights under the deterministic sampler give 13.36. Going from 1,000 steps to 50 costs 0.63 FID; going from 50 to 20 costs 2.17. That curve is why the industry standard settled near 20 to 50, and why the paper's claim of "10× to 50× faster in wall-clock time" held up.

Distillation pushed it further still: Stable Diffusion 3.5 Large Turbo generates in 4 steps, distilled from the 8.1-billion-parameter Large model. The general technique is knowledge distillation applied to the sampling trajectory — teach a student to jump in one step where the teacher took ten.

Types

There are not four kinds of image generator. There is one denoising loop, and four things you can vary about how you start it and what you condition on — which is why a single checkpoint does all of these and why the names describe interfaces rather than models.

• Text-to-image starts the latent from pure Gaussian noise and conditions only on the prompt. Every other mode is a modification of this one.
• Image-to-image encodes an existing picture to a latent and starts the loop partway through the schedule — at t = 500 rather than t = 999, say. The "strength" or "denoising" slider in every UI is literally the starting t: at 0.3 you keep 70% of the schedule's worth of the original structure, at 0.9 you keep almost none.
• Inpainting and outpainting run the normal loop but overwrite the known region of the latent with the correctly-noised original at every step, so the model only ever gets to decide the masked part while seeing the rest at the right noise level. Outpainting is inpainting on a canvas that was extended with empty space first.
• Structural conditioning — ControlNet, depth maps, pose skeletons, edge maps — adds a second conditioning signal alongside the text embeddings, injected into the denoiser at every step. This is how you get "this composition, that content", which text alone cannot express.

The genuinely architectural taxonomy — diffusion versus autoregressive versus GAN versus normalizing flow — is a fact about generative models in general rather than about images, and is covered under Generative AI. Images are simply the modality where diffusion won most decisively.

Real-World Applications

Generative Fill in Photoshop is the largest deployment of the inpainting variant above. Adobe put it into the Photoshop beta on 23 May 2023 and into the general release that September, and its behaviour is the mechanism showing through the UI: you select a region, the rest of the image is held fixed at each step's noise level, and the model fills the hole with something that is locally consistent with the surroundings because that is the only thing the loss ever trained it to do. The classic failure — an object that blends seamlessly at the seam but makes no sense at the scale of the whole picture — is the same fact seen from the other side.

Running the whole pipeline on a phone is the clearest evidence that latent compression was the decisive change rather than an optimisation. Qualcomm AI Research demonstrated Stable Diffusion 1.5 generating a 512×512 image in under 15 seconds at 20 inference steps on a Snapdragon 8 Gen 2 handset at Mobile World Congress on 23 February 2023, after quantizing the model from FP32 to INT8. A billion-parameter generative model, 20 sequential network evaluations, on a battery. Neither the quantization nor the step reduction would have been enough on a 786,432-number tensor.

Open weights changed who gets to build in a way that has no equivalent in text. Because Stable Diffusion's weights were downloadable, an ecosystem of fine-tunes, LoRAs and ControlNets grew around a model its authors did not control — which is why Stable Diffusion 3.5 (October 2024) remains the default base model people run locally even though, by 2026, the top of the public image arenas is held by newer models and closed APIs rather than by it. Open weights buy you tooling and control, not a place at the quality frontier — and its model page spends as much space on the Community License revenue threshold as on the architecture. If you are choosing between systems, that page and its neighbours — Seedream 5.0, Qwen-Image 2.0, HunyuanImage 3.0 (all 2025–2026) — are where the current comparisons live. They get updated; this page deliberately does not.

The harms are downstream of the same loop, and they have their own pages rather than a paragraph here. Synthetic media that impersonates a real, identifiable person is a deepfake, a legally distinct category from synthetic media in general. And because a detector's job is exactly the objective the generator was trained to defeat, the tractable response has moved to signing images at the moment of creation — the C2PA content provenance standard — rather than inspecting them afterwards.

Key Concepts

The knobs in every image-generation interface map one-to-one onto the loop above, which is the practical payoff of understanding it.

• Seed — the random numbers the latent is initialised with. It is the only stochastic input in a deterministic sampler, so the same seed, prompt, steps and guidance reproduce a generation exactly. Changing one word of the prompt with the seed held fixed is the cheapest way to learn what that word does.
• Steps and guidance scale are the two covered above, and the pair to reach for first: cost is linear in steps and doubled by guidance above 1, quality saturates near 50 steps on a non-distilled model, and guidance past roughly 10 is where the over-saturated look starts.
• Sampler / scheduler — the rule for stepping from t to t-1. DDIM, Euler, DPM-Solver++ and the rest are different numerical integrators of the same learned trajectory, which is why swapping them changes the image without any retraining.
• Negative prompt — the text put in place of the null token in the guidance formula. This is not a filter and not a rule; it is the thing the model is being pushed away from, with force w.
• Latent decode — the last step, and a resolution ceiling. Detail the encoder cannot represent at 48× compression cannot be produced by the decoder, no matter how many steps you run.

Challenges

Text inside images fails for a structural reason. The denoiser operates on a continuous latent grid, and nothing in the objective enforces that a glyph is one of a finite alphabet. A stroke that is 90% of the way to being an "R" is a locally plausible arrangement of latent values in a way that a 90%-correct word is not. Newer models improved this mostly by throwing far more text-bearing training data and much larger text encoders at it, which narrows the gap without changing the underlying reason it exists.

Counting and binding fail for the same reason. "Three red cups and one blue bowl" requires discrete quantity and correct attribute-to-object assignment, and cross-attention distributes prompt influence over space without any mechanism guaranteeing either. This is why the standard workaround is compositional: generate, then use inpainting or structural conditioning to fix the parts that came out wrong.

Guidance artifacts are misread as model quality. As the numbers above show, a high guidance scale demonstrably shrinks output diversity — in the one-dimensional toy, the standard deviation fell from 0.470 to 0.225 between w = 0 and w = 7.5, and the mean overshot the true class mean by more than double. Four samples that all look like the same photograph is usually the guidance scale, not a memorised training image.

Too few steps costs variety before it costs realism, which makes it hard to notice. At 5 steps the toy model still returns a mean of 1.000 — dead on — with a standard deviation of 0.225 against a true 0.500. Individual outputs look fine; the distribution has quietly collapsed toward the mode. Judging a step count from one sample will mislead you every time.

The decoder sets a floor on fine detail. Compression is lossy by construction, so at 48× the encoder discards information that no amount of denoising can recover. Skin texture, small distant faces, dense foliage and fine typography are where this shows first, and no prompt fixes it — it is a property of the autoencoder, not the diffusion model riding on it.

Training data provenance remains legally unsettled. The large open datasets were assembled by scraping, and the resulting disputes over whether training constitutes infringement are being litigated in several jurisdictions with no consistent answer yet. The practical consequence for anyone shipping generated images commercially is that indemnification terms and license text are load-bearing, which is why they occupy so much of a model page.

Future Trends

• Few-step generation becomes the default, not the fast option. The distillation results are already at 4 steps for an 8-billion-parameter model, and the direction of travel is toward one or two. When a generation costs two network evaluations instead of 100, image generation stops being a request-and-wait interaction and becomes something that updates as you type.
• Rectified flow is replacing the noise schedule. Stability's SD3 line moved from the DDPM-style curved trajectory to rectified flow, which trains the model to follow a straight line between noise and data. A straight path is far easier to integrate in few steps, so this and step distillation are the same trend approached from opposite ends.
• The U-Net is finishing its retirement. Newer flagships are transformers over latent patches rather than convolutional U-Nets, which is the same architectural convergence that already happened in language and is now happening in video — one architecture, three modalities, differing mainly in what the patches are.
• Provenance moves into the generator. Signing at creation time is the only approach that does not get worse as models improve, so expect content credentials to become a property of the model endpoint rather than an optional post-process. Whether the metadata survives being screenshotted remains the unsolved half.

Code Example

Two blocks, both runnable, both printing the numbers used above. The first is the noise schedule; nothing here is learned.

import numpy as np

T = 1000
betas = np.linspace(1e-4, 0.02, T)                     # DDPM's linear schedule
lin = np.cumprod(1.0 - betas)                          # alpha-bar: (signal fraction)^2

t = np.arange(T) / T                                   # cosine schedule, s = 0.008
f = np.cos((t + 0.008) / 1.008 * np.pi / 2) ** 2
cos = f / f[0]

print("  t   linear signal  cosine signal   linear SNR")
for i in (0, 100, 200, 300, 500, 700, 999):
    print(f"{i:4d}      {np.sqrt(lin[i]):.4f}         {np.sqrt(cos[i]):.4f}     {lin[i]/(1-lin[i]):9.3f}")

print(f"\nlinear: signal halves at t = {int(np.argmin(abs(np.sqrt(lin) - 0.5)))}"
      f", cosine at t = {int(np.argmin(abs(np.sqrt(cos) - 0.5)))}")
print(f"linear: {int((np.sqrt(lin) < 0.1).sum())} of 1000 steps have under 10% signal left")

  t   linear signal  cosine signal   linear SNR
   0      0.9999         1.0000      9999.000
 100      0.9461         0.9859         8.537
 200      0.8102         0.9480         1.910
 300      0.6277         0.8871         0.650
 500      0.2789         0.7027         0.084
 700      0.0829         0.4507         0.007
 999      0.0064         0.0016         0.000

linear: signal halves at t = 367, cosine at t = 664
linear: 327 of 1000 steps have under 10% signal left

The second is the sampler itself, in one dimension. The "data" is two Gaussians at −1 and +1, so the perfect denoiser is available in closed form and no network needs training — which isolates the loop from the model. The reverse process, the step count and classifier-free guidance are all exactly as a real image model implements them.

import numpy as np

T, SD = 1000, 0.5
alpha_bar = np.cumprod(1.0 - np.linspace(1e-4, 0.02, T))
# Data: two classes, "left" ~ N(-1, 0.5^2) and "right" ~ N(+1, 0.5^2).

def eps_class(x, ab, mu):
    """Exact noise prediction if the data were only N(mu, SD^2)."""
    return np.sqrt(1 - ab) * (x - np.sqrt(ab) * mu) / (ab * SD**2 + 1 - ab)

def eps_uncond(x, ab):
    """Exact noise prediction for the unconditional 50/50 mixture."""
    d = np.exp(-0.5 * (x - np.sqrt(ab) * np.array([[-1.0], [1.0]])) ** 2
               / (ab * SD**2 + 1 - ab))
    w = d / d.sum(0)                                    # posterior over the two classes
    return (w * np.stack([eps_class(x, ab, -1.0), eps_class(x, ab, 1.0)])).sum(0)

def sample(n_steps, w, n=20000):
    rng = np.random.default_rng(0)
    ts = np.linspace(T - 1, 0, n_steps).astype(int)
    x = rng.standard_normal(n)                          # start from pure noise
    for i, t in enumerate(ts):
        ab = alpha_bar[t]
        eps = (1 + w) * eps_class(x, ab, 1.0) - w * eps_uncond(x, ab)   # ask for "right"
        x0 = (x - np.sqrt(1 - ab) * eps) / np.sqrt(ab)  # the denoiser's guess at the answer
        ab_prev = alpha_bar[ts[i + 1]] if i + 1 < n_steps else 1.0
        x = np.sqrt(ab_prev) * x0 + np.sqrt(1 - ab_prev) * eps          # re-noise, less
    return x

print("the 'right' class is N(1.000, 0.500)\n")
for steps in (1000, 50, 20, 5):
    s = sample(steps, w=0.0)
    print(f"guidance 0.0, {steps:4d} steps -> mean {s.mean():.3f}  sd {s.std():.3f}")
print()
for w in (0.0, 1.0, 3.0, 7.5):
    s = sample(50, w)
    print(f"guidance {w:3.1f},   50 steps -> mean {s.mean():.3f}  sd {s.std():.3f}")

the 'right' class is N(1.000, 0.500)

guidance 0.0, 1000 steps -> mean 0.999  sd 0.497
guidance 0.0,   50 steps -> mean 0.999  sd 0.470
guidance 0.0,   20 steps -> mean 0.999  sd 0.429
guidance 0.0,    5 steps -> mean 1.000  sd 0.225

guidance 0.0,   50 steps -> mean 0.999  sd 0.470
guidance 1.0,   50 steps -> mean 1.317  sd 0.333
guidance 3.0,   50 steps -> mean 1.664  sd 0.267
guidance 7.5,   50 steps -> mean 2.145  sd 0.225

Read the two halves separately. Cutting steps leaves the mean untouched and quietly crushes the spread — 0.497 at 1,000 steps, 0.225 at 5, against a true 0.500. Raising guidance does something different and worse: it moves the mean, from 0.999 to 2.145, so the samples are no longer drawn from the class you asked for but from an exaggerated caricature of it, with the variety gone as well. Everything an image model does at 512×512 with 8 billion parameters, both failure modes included, is this loop with a better denoiser.