Classifier-Free Guidance (CFG)

Definition

Classifier-free guidance (CFG) is the trick that makes a diffusion model actually pay attention to your prompt. A diffusion model generates an image by starting from pure noise and removing a little of it over many steps; at each step the network predicts the noise it thinks is present. Left to itself, a model trained on captioned images will happily wander toward some plausible image that only loosely matches the words you gave it. CFG forces the issue: at every step it runs the model twice — once told the prompt, once told nothing — and then pushes the prediction in the direction that the prompt added, amplified by a dial called the guidance scale.

The thing worth carrying away before any maths is the trade-off that dial controls. Turn it up and the picture matches the words more literally, but past a point the colours blow out, contrast turns harsh and the model starts inventing artefacts. Turn it down and images look natural and varied but drift off-prompt. Every text-to-image tool exposes this number (Stable Diffusion calls it "CFG scale", and its pipeline default is 7.5) precisely because there is no single correct setting — it is a knob you trade quality against fidelity on.

How It Works

At one denoising step the model sees a noisy latent and outputs its estimate of the noise to remove. CFG asks for two such estimates of the same latent:

• the conditional prediction, with the prompt fed in — call it eps_cond;
• the unconditional prediction, with the prompt replaced by an empty "null" token — call it eps_uncond.

The difference eps_cond − eps_uncond is, roughly, "the part of the model's answer that the prompt is responsible for." CFG keeps the unconditional prediction as a baseline and then walks further in that prompt-driven direction:

eps_hat = eps_uncond + s * (eps_cond - eps_uncond)

Here s is the guidance scale. It is an extrapolation, not an average — for s above 1 the result lands beyond the conditional prediction, which is the whole point: it exaggerates whatever the prompt contributed.

The scale, in numbers

The behaviour of s falls out of the formula directly. At s = 1 the two terms after eps_uncond collapse to eps_cond − eps_uncond, so eps_hat = eps_cond exactly: you get the model's raw conditional output and no guidance at all. At s = 0 you would get the unconditional output — a generic image ignoring the prompt entirely. Everything interesting happens above 1.

Take a single coordinate of the noise tensor where the model is unsure without the prompt but confident with it: eps_uncond = 0.20 and eps_cond = 0.50, a prompt contribution of 0.30.

• At s = 1: 0.20 + 1 × 0.30 = 0.50 — unchanged from the conditional prediction.
• At s = 7.5: 0.20 + 7.5 × 0.30 = 0.20 + 2.25 = 2.45.

The prompt's 0.30 nudge has become a 2.25 shove — the signal is amplified more than sevenfold. That amplification, repeated across every coordinate and every denoising step, is what pins the output to the prompt; it is also, when s is too large, exactly what over-drives pixel values past the range that renders as natural colour and produces the tell-tale oversaturation.

Where the formula comes from

Ho and Salimans introduced this in Classifier-Free Diffusion Guidance (2022). Their paper writes the combination with a weight w:

eps_tilde = (1 + w) * eps_cond - w * eps_uncond      (Eq. 6)

This is the same equation. Expanding the extrapolation form above gives eps_hat = (1 − s)·eps_uncond + s·eps_cond, so the paper's w and the guidance scale s used by implementations are related by s = w + 1. Stable Diffusion's default of s = 7.5 is therefore w = 6.5 in the paper's notation. (Verify the s = 7.5 case against Eq. 6: 7.5 × 0.50 − 6.5 × 0.20 = 3.75 − 1.30 = 2.45, matching the extrapolation calculation above.)

The name is the second half of the story. The technique CFG replaced, classifier guidance (Dhariwal and Nichol, 2021), steered generation using the gradient of a separate image classifier trained to recognise classes in noisy images — an extra model to train, and one you could effectively run an adversarial attack against. Ho and Salimans dropped the classifier entirely: in their words, classifier-free guidance "instead mixes the score estimates of a conditional diffusion model and a jointly trained unconditional diffusion model." There is no classifier gradient in Eq. 6, which is what "classifier-free" means.

That "jointly trained unconditional model" is not a second network. The same model learns to do both jobs during training by randomly dropping the condition: with some probability p_uncond a training example has its caption replaced by the null token, so the network sees enough prompt-free examples to produce a sensible eps_uncond at inference time. The paper reports that p_uncond of 0.1 or 0.2 work about equally well, while 0.5 consistently performs worse — you need some unconditional training, but not half of it.

Real-World Applications

CFG is not an optional refinement; it is switched on by default in essentially every deployed text-to-image diffusion system, and their prompt-following would collapse without it.

• Stable Diffusion exposes it as the "CFG scale" slider in every front-end (Automatic1111, ComfyUI, the diffusers library), defaulting to 7.5. When a user complains that "the model ignored my prompt", raising this number is the first fix; when they complain that outputs look "burnt" or "deep-fried", lowering it is.
• Negative prompts, the feature that lets you type things to avoid ("blurry, extra fingers"), are a direct repurposing of the CFG formula: the unconditional term eps_uncond is swapped for a prediction conditioned on the negative text, so the model extrapolates away from it. This only functions because guidance is already running — with s = 1 a negative prompt has no effect.
• Commercial generators such as Midjourney and DALL·E apply the same conditional/unconditional mixing internally; the specific scale is usually tuned and hidden rather than exposed, but the mechanism is identical.
• Beyond images, the same guidance appears wherever conditional diffusion does: audio and music diffusion models, and the diffusion-based generators behind some voice-cloning and speech-to-speech systems, use a conditional-versus-unconditional extrapolation to make the output track the reference or transcript.

Key Concepts

• Guidance scale is a fidelity-versus-diversity dial. Low s gives varied, natural, loosely-matching images; high s gives literal, uniform, eventually-degraded ones. The useful band for most text-to-image models is roughly 5 to 12.
• It is extrapolation, not interpolation. s > 1 deliberately overshoots the conditional prediction. This is why the effect is strong enough to matter and also why it can go too far — nothing in the formula clamps the result to a sensible range.
• The cost is a doubled model evaluation. Every step computes both eps_cond and eps_uncond, so a guided sample does about twice the network compute of an unguided one. Implementations batch the two inputs into a single forward pass, but the model still evaluates two latents per step.
• p_uncond is a training-time hyperparameter, s is an inference-time one. You choose how much unconditional training the model got (0.1-0.2) once, when training; you choose the guidance scale freely at generation time, per image.

Challenges

• Oversaturation and artefacts at high scale. Because CFG amplifies rather than averages, pushing s high drives predicted values past the range that maps to natural colour. The result is blown-out contrast, posterised colours and hallucinated texture — the "deep-fried" look. This is not a bug to be fixed but the direct consequence of extrapolating too far, and it caps how literally you can force prompt adherence.
• The two-pass tax. Doubling the per-step compute is a real cost at scale. It is the specific target of guidance distillation, which trains a student model to reproduce the guided output in a single pass, and of the low-step "turbo" and consistency models that also fold guidance in — an active line of work precisely because the second pass is pure overhead once training is done.
• No universal setting. The best s shifts with the model, the sampler, the number of steps and even the individual prompt, so it stays a parameter users tune rather than a constant. A value that is crisp for one checkpoint is scorched for another.
• It can reduce diversity to the point of mode collapse. Strong guidance narrows the output distribution toward the highest-probability rendering of the prompt, so a batch generated at high s can come back looking almost identical — fidelity bought with variety.

Code Example

The entire mechanism is the extrapolation formula applied to the model's two predictions. This runs it on one 4-dimensional slice of a noise tensor at s = 1 (no guidance) and s = 7.5 (the Stable Diffusion default), and confirms that s = 1 reproduces the conditional prediction exactly.

# Classifier-free guidance on one denoising step.
# The model outputs two noise predictions for the same noisy latent:
#   eps_uncond  -- prompt dropped (null token)
#   eps_cond    -- conditioned on the prompt
# CFG extrapolates away from the unconditional prediction:
#   eps_hat = eps_uncond + s * (eps_cond - eps_uncond)
# s is the "guidance scale". It relates to Ho & Salimans' weight w by s = w + 1.

def cfg(eps_uncond, eps_cond, s):
    return [u + s * (c - u) for u, c in zip(eps_uncond, eps_cond)]

# A single 4-dimensional slice of a noise-prediction tensor (illustrative values).
eps_uncond = [0.20, -0.10,  0.05,  0.40]
eps_cond   = [0.50, -0.05,  0.30,  0.35]

for s in (1.0, 7.5):
    out = cfg(eps_uncond, eps_cond, s)
    print(f"s = {s}: eps_hat = {[round(x, 3) for x in out]}")

# Show that s = 1 reproduces the conditional prediction exactly (no guidance):
print("s=1 equals eps_cond exactly:", cfg(eps_uncond, eps_cond, 1.0) == eps_cond)

Output:

s = 1.0: eps_hat = [0.5, -0.05, 0.3, 0.35]
s = 7.5: eps_hat = [2.45, 0.275, 1.925, 0.025]

Notice the last coordinate: eps_uncond = 0.40 was above eps_cond = 0.35, so the prompt's contribution there was negative (-0.05), and guidance drives it down to 0.025 rather than up. CFG amplifies the prompt's influence in whatever direction the prompt pushed — the fourth value falls while the first climbs from 0.5 to 2.45. That per-coordinate, per-step amplification is the whole of classifier-free guidance.