Temperature

Definition

Temperature is a single number that controls how random a language model's output is. A low temperature makes the model pick the most likely next words, so its answers come out focused and nearly the same every time; a high temperature spreads the odds across more of the vocabulary, so its answers come out more varied, more surprising, and riskier. Most APIs expose it as one value, commonly on a scale from 0 to about 2, applied every time the model chooses a token.

The name is borrowed from physics: a "hot" system explores many states, a "cold" one settles into the most stable. In a language model the knob does the same thing to the model's guesses — it does not add new knowledge or change what the model believes is most likely, it only decides how boldly the model gambles on the less likely options when it samples the next word during text generation.

How It Works

Before a large language model picks a token, it produces a vector of raw scores called logits — one score per token in its vocabulary. To turn those scores into probabilities the model runs them through the softmax function, and temperature is a single division inserted just before that step: every logit is divided by the temperature T, so softmax sees z/T instead of z.

That one division is the whole mechanism. Writing z_i for the logit of token i, the probability the model assigns is:

P(i) = exp(z_i / T) / Σ_j exp(z_j / T)

The temperature T never touches the model's weights or its ranking of tokens — the most likely token stays the most likely at every temperature. What changes is the gap between the probabilities. Dividing by a small T (say 0.5) stretches the logits apart, so the leading token pulls even further ahead and the distribution gets sharper. Dividing by a large T (say 2.0) squeezes the logits together, so the distribution gets flatter and the trailing tokens get a real chance of being sampled.

Three points on the dial are worth naming:

• As T → 0 the division blows the leading logit's lead up without bound, and sampling collapses onto the single top token every time. This is greedy (argmax) decoding — the most focused, most repeatable setting.
• At T = 1 the logits pass through untouched, so softmax returns the model's own raw distribution — the probabilities the model actually learned.
• At T > 1 the distribution flattens toward uniform, handing probability mass to tokens the model rated unlikely — the source of both creativity and incoherence.

You can see the shift on a tiny worked example. Take three candidate tokens with logits [2.0, 1.0, 0.5]. At T = 0.5 the softmax probabilities are about 84.4% / 11.4% / 4.2% — the top token dominates. At T = 1.0 they spread to 62.9% / 23.1% / 14.0%, the model's raw view. At T = 2.0 they flatten further to 48.1% / 29.2% / 22.7%, and the least likely token has gone from a 1-in-24 shot to better than 1-in-5. Same three logits, same ranking; only the boldness changed. The Code Example below computes exactly these numbers.

This temperature-in-softmax formulation is the same one Hinton, Vinyals and Dean introduced for knowledge distillation in their 2015 paper Distilling the Knowledge in a Neural Network, where they note that "using a higher value for T produces a softer probability distribution over classes."

Temperature is easy to confuse with top-p (nucleus) and top-k sampling, but they are different operations that are usually combined rather than swapped. Temperature reshapes the entire distribution, stretching or squeezing every probability. Top-p and top-k truncate it: top-k keeps only the k highest-probability tokens and top-p keeps the smallest set of tokens whose probabilities add up to p, throwing the rest away before sampling. A typical pipeline applies temperature first to reshape, then top-p or top-k to trim the tail, then samples from what survives.

Real-World Applications

Temperature is a standard control on most text-generation interfaces — with a growing exception for reasoning models, covered under Challenges below. Hosted APIs from the major model providers expose a temperature argument on their completion and chat endpoints, and open-source inference stacks do the same: the Hugging Face Transformers generate() method takes a temperature argument, and high-throughput serving engines such as vLLM accept it per request. Because it is a single scalar with an obvious effect, it is usually the first knob a developer reaches for when an assistant's answers feel either too robotic or too unhinged.

The choice of value tracks the job rather than any fixed rule. When there is one right answer — extracting a field from a document, generating code that has to compile, returning a specific fact — a low temperature near 0 minimises the chance the model wanders off the most likely path during inference. When the goal is a range of options — drafting marketing copy, brainstorming names, writing fiction — a temperature closer to 1 lets the model explore. Many production systems even vary it by call: low for the tool-calling and retrieval steps that must be exact, higher for the final natural-language reply.

Temperature also does real work away from token sampling. In knowledge distillation, a small "student" model is trained to copy a large "teacher." Training on the teacher's hard top-1 answer throws away almost everything the teacher knows; instead the teacher's logits are divided by a temperature above 1 to produce soft targets — a smeared-out distribution that reveals, for example, that the teacher thought an image of a "2" looked a little like a "7". Those soft targets carry far more information per example, which is precisely the effect Hinton, Vinyals and Dean formalised in 2015. The same z/T division appears again in reinforcement learning, where a temperature on a softmax policy tunes how much an agent explores versus exploits.

Challenges

The most common surprise is that temperature 0 is not perfectly deterministic. Setting T to zero makes the sampling deterministic — it always takes the top token — but the logits feeding that choice are computed in floating-point, and the order of those additions can change with GPU batch size, kernel version or hardware. When two tokens are nearly tied, a rounding difference in the seventh decimal place can flip which one is "highest," so the same prompt at temperature 0 can still produce different completions across machines or even across differently-sized batches on the same machine. If you need bit-for-bit reproducibility, temperature 0 alone will not give it.

At the other end, high temperature buys diversity with coherence. Because T > 1 hands probability to tokens the model rated unlikely, raising it far enough eventually samples genuinely wrong continuations — a factual claim the model would rate implausible at T = 1 becomes reachable at T = 1.5, which is one concrete route to hallucination. The failure is gradual, not a cliff: mild increases add welcome variety, and only past roughly T = 1 does output start drifting toward word-salad, though the exact point depends on the model and the prompt.

A newer surprise is that the most capable models increasingly do not offer the knob at all. As of 2026 a growing class of reasoning models either rejects a non-default temperature outright or ignores it, because while such a model works through a long internal reasoning trace the diversity of its output is shaped by that process rather than by a single softmax division. OpenAI's o-series reasoning models list temperature (and top_p) among the sampling parameters they do not support, so setting a custom value returns an error instead of a hotter answer; Anthropic's Claude blocks changes to temperature while its extended-thinking mode is engaged. The framing that matters is the category, not any one model: on a model that is actively reasoning, sampling diversity is governed differently, and code that sets temperature out of habit has to special-case these models rather than assume the argument is always accepted.

Finally, temperature interacts with top-p and top-k, and setting both aggressively compounds. The two controls pull in tension: temperature widens the field of plausible tokens while top-p and top-k narrow it. A high temperature paired with a permissive top-p (near 1.0) leaves the flattened tail intact and samples freely from it — the most volatile combination — whereas a high temperature under a tight top-p is partly reined back in because the truncation discards the long tail the temperature just inflated. Tuning one while forgetting the other is why a "small" temperature change sometimes has a much larger or smaller effect than expected.

Code Example

The division that defines temperature is short enough to write from scratch. This block computes the softmax probabilities for the three logits used above at three temperatures; the max-subtraction is the standard trick for keeping exp from overflowing.

import math

def softmax_with_temperature(logits, T):
    scaled = [z / T for z in logits]
    m = max(scaled)                      # subtract max for numerical stability
    exps = [math.exp(s - m) for s in scaled]
    total = sum(exps)
    return [e / total for e in exps]

logits = [2.0, 1.0, 0.5]

for T in (0.5, 1.0, 2.0):
    probs = softmax_with_temperature(logits, T)
    pct = ", ".join(f"{p:.1%}" for p in probs)
    print(f"T={T}: {pct}")

Running it prints:

T=0.5: 84.4%, 11.4%, 4.2%
T=1.0: 62.9%, 23.1%, 14.0%
T=2.0: 48.1%, 29.2%, 22.7%

The ranking of the three tokens is identical in every row — the top token is always the top token. Only the spread changes: cold temperature concentrates the mass on the leader, hot temperature levels it out. Sampling from the T = 2.0 row picks the third token more than five times as often as sampling from the T = 0.5 row, which is the entire practical effect of the knob in one comparison.