Text Generation

Text generation is a language model producing text one token at a time: at each step it predicts a distribution over its vocabulary, picks a token, repeats.

Published Updated

On this page

Definition

Text generation is how an AI language model produces new text: it works one token at a time, and at each step it predicts a probability distribution over its entire vocabulary for the next token, picks one, appends it to what it has written so far, and feeds the whole sequence back in to predict the token after that. This loop — predict a distribution, pick a token, append it, repeat — is called autoregressive generation, and it is the complete mechanism behind a chatbot appearing to "write."

The counter-intuitive part is what is not there. The model has no plan, no outline, and no draft it is revising. When ChatGPT answers a question, it is making one very fast next-token guess, then another, then another — often hundreds in a row — and the coherence you read back is an emergent property of each guess being conditioned on every token before it. A token is usually a word-piece rather than a whole word (see tokenization), so "unbelievable" might be generated as un, bel, iev, able across four separate steps.

How It Works

Every generation step runs one forward pass of a large language model and ends with a vector of raw scores called logits — one score per token in the vocabulary. Vocabulary size is a per-model design choice, but it lands in a consistent range: GPT-2 used exactly 50,257 tokens, and the tokenizers behind more recent models sit near or above 100,000. Either way, each step is a choice among tens of thousands of options, so the logits are turned into a proper probability distribution by the softmax function, P(i) = exp(z_i) / Σ_j exp(z_j), which squashes the raw scores into positive numbers that sum to 1.

Take a concrete step. After the prompt "The mouse ate the ___", suppose the model assigns just three candidates the logits cheese = 3.0, bait = 1.5, cat = 0.5 (the other ~50,000 tokens get much lower scores and round to near-zero). Running those through softmax gives:

  • cheese → 76.6%
  • bait → 17.1%
  • cat → 6.3%

Now the model has a distribution, and a second, separate choice appears: how to turn it into an actual token. This is the decoding strategy, and it is the knob that most changes the character of the output:

  • Greedy decoding always takes the single highest-probability token — here, cheese, every time. Repeatable, but it never explores, which causes the failure described below.
  • Sampling draws a token in proportion to the probabilities. bait gets picked about 17% of the time and cat about 6% — the same numbers that make output varied also let it occasionally surprise you.
  • Temperature reshapes the distribution before sampling. Low temperature sharpens it toward the top token (closer to greedy); high temperature flattens it, handing real probability to the unlikely tokens.
  • Top-k and top-p (nucleus) sampling truncate the distribution first. Top-k keeps only the k highest tokens; top-p keeps the smallest set of tokens whose probabilities sum to p — throwing away the long tail so the model can never accidentally pick a bizarre continuation.

A real pipeline typically applies temperature to reshape, then top-p or top-k to trim the tail, then samples from what survives — every one of those choices is made at every single step.

Why generation is the slow part

Because each token depends on the one before it, the loop is strictly sequential: generating N tokens costs N forward passes through the model, in order. A 500-token answer is 500 passes, and token 5 genuinely cannot be computed until token 4 has been chosen and appended. This is the key asymmetry of inference. Reading the prompt (the prefill phase) processes every prompt token in one parallel pass, but writing the reply (the decode phase) can only go one token at a time — which is why the model can "read" a 2,000-word document almost instantly yet visibly types its answer out.

Each of those passes would otherwise have to re-process the entire growing sequence from scratch, an O(N²) explosion. The KV cache is what prevents it: the model stores the attention keys and values it already computed for earlier tokens and reuses them, so step number 500 only computes attention for the one new token rather than all 500. The cache removes the redundant work but not the sequential dependency — the passes still happen one after another, which is the durable reason text generation is latency-bound in a way that reading a prompt is not.

Real-World Applications

Text generation is the engine under the products people now use daily. Conversational assistants — ChatGPT, Claude, and Gemini — are autoregressive generation wrapped in a chat interface: every reply is produced token by token, which is exactly why you watch the answer stream in rather than appear at once. Coding assistants such as GitHub Copilot apply the identical loop to source code, predicting the next token of a function from the surrounding file, and inline features like Gmail's Smart Compose finish a sentence the same way with a low-temperature setting so the suggestion stays safe and predictable.

The distinction from natural language processing is worth keeping straight: NLP is the whole field of getting computers to work with human language, of which generation is one task. Classification, translation, and extraction are NLP too, and many predate the generative era. Text generation is also not the model itself — a large language model is the trained network; text generation is one thing you do with it at inference time. The same model that generates a story can be used purely to score how likely an existing sentence is, with no generation at all.

Challenges

The failure that greedy and very-low-temperature decoding produce is repetition and degeneration: text that collapses into a loop ("I think that I think that I think…") or drifts into bland, generic filler. The cause is structural — always taking the most probable token drives the model toward high-frequency, low-information words, and once it enters a repetitive rut each repeated token makes the same continuation look even more likely. This is precisely the problem the 2020 paper The Curious Case of Neural Text Degeneration diagnosed and answered by introducing top-p (nucleus) sampling, which is why pure greedy decoding is rarely used for open-ended writing.

The failure at the other end of the mechanism is hallucination: the model generates fluent, grammatical, confident text that is simply false. This is not a bug layered on top of generation — it falls straight out of the objective. The model is trained and decoded to produce likely text, and a plausible-sounding wrong answer can have a higher next-token probability than an awkwardly-phrased right one. Nothing in the predict-pick-append loop checks a claim against the world; the model optimises for what reads well, not for what is true, so the reader — not the model — has to supply the fact-checking.

The third challenge is the cost baked into the sequential loop. Because a reply of length N takes N forward passes no matter how fast the hardware, generation latency scales with how much you ask the model to write, and long outputs are expensive in both time and compute. It is the reason streaming exists (so the user sees progress instead of waiting for the whole reply) and the reason a large body of research — from the KV cache to speculative decoding — targets making each of those unavoidable sequential steps cheaper.

Code Example

The whole distribution-and-decoding step is short enough to write from scratch. This block computes the softmax probabilities for the three logits used above, shows what greedy decoding picks, and then samples 100,000 times to confirm that sampling reproduces the distribution — a token rated 17.1% really does get chosen about 17% of the time.

import math, random

# Three candidate next tokens after "The mouse ate the ___"
# with the raw scores (logits) a model might assign.
tokens = ["cheese", "bait", "cat"]
logits = [3.0, 1.5, 0.5]

# softmax -> a probability for every candidate
m = max(logits)
exps = [math.exp(z - m) for z in logits]
Z = sum(exps)
probs = [e / Z for e in exps]

for t, z, p in zip(tokens, logits, probs):
    print(f"{t:8s} logit {z:>4} -> P = {p:6.1%}")

# greedy: always the single highest-probability token
greedy = tokens[probs.index(max(probs))]
print("greedy pick:", greedy)

# sampling: draw in proportion to the probabilities
random.seed(0)
counts = {t: 0 for t in tokens}
N = 100_000
for _ in range(N):
    r, cum = random.random(), 0.0
    for t, p in zip(tokens, probs):
        cum += p
        if r < cum:
            counts[t] += 1
            break
print("sampling over", N, "draws:", {t: f"{counts[t]/N:.1%}" for t in tokens})

Output:

cheese   logit  3.0 -> P =  76.6%
bait     logit  1.5 -> P =  17.1%
cat      logit  0.5 -> P =   6.3%
greedy pick: cheese
sampling over 100000 draws: {'cheese': '76.6%', 'bait': '17.1%', 'cat': '6.3%'}

Greedy would answer "cheese" every single time; sampling answers "bait" or "cat" roughly one time in four. That one difference — argmax versus a weighted draw — is most of what separates a rigid, repeatable model from a creative, occasionally-wrong one.

Frequently Asked Questions

It generates one token at a time. At each step the model predicts a probability for every token in its vocabulary, picks one, appends it to the text so far, and feeds the whole sequence back in to predict the next token. Repeating that loop is called autoregressive generation.
Greedy decoding always takes the single highest-probability token, so it is repeatable but prone to bland, looping text. Sampling draws a token in proportion to the probabilities, so a token the model rated at 17% gets picked roughly 17% of the time — which is where variety, and risk, come from.
A prompt is read in one parallel forward pass, but generating N tokens takes N sequential forward passes — token 5 cannot be computed until token 4 exists. That strict left-to-right dependency, not the size of the prompt, is what makes generation the slow part.
Generation optimises for likely text, not true text. The model samples whatever continuation looks statistically plausible given its training, so a fluent, well-formed, and completely wrong sentence can be exactly what the next-token distribution favours. This is called hallucination.
They truncate the vocabulary before sampling. Top-k keeps only the k highest-probability tokens; top-p (nucleus) keeps the smallest set of tokens whose probabilities add up to p, discarding the long tail. Both cut off unlikely tokens so the model never accidentally picks a bizarre one.

Related Terms

Large Language Model (LLM)

A large language model predicts the next token of text. How that produces answers, where its knowledge is stored, and why an LLM is not a chatbot.

Temperature

Temperature is the number that controls how random a language model's output is — it divides the logits before softmax, from near-deterministic to diverse.

Token

The unit an AI model reads, writes and bills by — roughly 1.3 tokens per English word. Input, output and cached tokens are priced up to 10x apart.

Inference

Running a trained model to get an answer. Training is paid once; inference is paid on every request — and it is where most AI money goes.

AI Hallucinations

Models hallucinate because training and benchmarks reward guessing over admitting ignorance: a guess sometimes scores, "I don't know" scores zero every time.

KV Cache

The memory an LLM keeps so it never recomputes the past — why a long context costs money, and how many users one GPU can actually serve at once.

Continue Learning

Explore our use-case guides and prompts to deepen your AI knowledge.

Browse AI Tools← Back to Glossary