Inference

Definition

Inference is what happens when a trained model is actually used: new input goes in, the frozen weights are read, and a prediction comes out. Nothing is learned. Training is a one-time capital cost that eventually ends; inference is the bill that arrives on every single request, forever — which is why AWS's Well-Architected guidance states that "up to 90% of the infrastructure spend for developing and running ML applications is often on inference" (AWS Machine Learning Lens).

That asymmetry is the whole reason the concept has its own vocabulary. A model that is expensive to train but cheap to run is a good business. A model that is cheap to train but expensive to run is a product that gets more unprofitable the more people like it.

How It Works

Training vs. inference

The two phases run the same network and share almost nothing else.

The memory row is the one that surprises people. Training has to keep, for every single parameter, the gradient flowing back through it and whatever state the optimizer carries — the Adam optimizer holds two extra numbers per parameter — so the machine must fit several copies of the model at once. Inference needs one. That is why a model you cannot train on your hardware may still run on it perfectly well.

Inference or prediction? In machine learning the two words are used interchangeably: running the model is making a prediction, and nobody will misunderstand you. In statistics they are opposites — prediction is estimating an unseen outcome, inference is drawing conclusions about the underlying process that generated the data. The AI sense borrows the statistical word for the non-statistical meaning, which is why the term looks confusing arriving from either direction.

The forward pass

One run of a model is a forward pass: the input is turned into numbers, those numbers are multiplied through the network's layers, and the last layer produces an output — a class label, a probability, a next-token distribution. The weights are read and never written. The same input, the same weights and a temperature of zero should therefore give the same output every time. (Should. See Challenges.)

Around that sit two thin shells that are easy to underestimate: preprocessing, which must reproduce exactly what training did — the same tokenizer, the same image normalisation, the same feature scaling — and post-processing, which turns raw scores back into something usable. A model served with a preprocessing step that drifted from its training-time version will not error. It will simply be quietly worse.

One forward pass, one token

For a classifier, one input is one forward pass and the story ends. For a large language model it does not, and this is the single most common misunderstanding about the term.

A language model produces one token per forward pass. Your prompt is read in one parallel sweep — the prefill — and after that each word of the answer costs its own complete trip through the entire network, with everything already generated fed back in as input. Ask for a page of text and you get perhaps 700 tokens; that is roughly 700 forward passes, and the last one has to process 700 more tokens of context than the first. The KV cache exists precisely so those passes don't redo each other's work.

What that costs, on numbers

Here is why generation feels slow. Every one of those passes has to read the model's weights out of GPU memory. A 70B-class model — Llama 3.1 70B carries 70.6 billion of them — at one byte per parameter is 70.6 GB to read per token. An NVIDIA H100 SXM moves 3.35 TB/s (NVIDIA), so:

70.6 GB ÷ 3,350 GB/s = 21.1 ms per token
1 ÷ 0.0211 s        ≈ 47 tokens per second   <- ceiling for one user

Seven hundred tokens at that rate is about 15 seconds of pure memory traffic, before any arithmetic, networking or queueing. The memory wall page derives why the GPU's arithmetic units are almost entirely idle while this happens, and inference optimization shows how serving many users at once is the only thing that makes the economics work. The three facts to carry away from here: inference is repeated, it is memory-bound, and it is priced per token because that is genuinely the unit of work.

Types

The four labels people use — batch, real-time, edge, cloud — are not four alternatives. They are two independent choices, and every combination exists.

When the work runs. Real-time (or online) inference answers a request while someone waits, so latency is the constraint and the machine is often underused. Batch inference collects work and runs it when convenient, so throughput is the constraint and the hardware stays full. The second is dramatically cheaper for the same model — a batch API that gives the provider a 24-hour window instead of a two-second one typically charges half price, and nothing about the model changed.

Where the work runs. Cloud inference runs on a server you rent, with big accelerators and easy updates. Edge inference runs on the device — phone, car, camera — so the data never leaves, there is no network round trip, and it works offline, at the price of a much smaller model.

Real-World Applications

Apple ships edge inference to a billion devices. Apple Intelligence runs an on-device foundation model of roughly 3 billion parameters. The generation Apple documented in 2025 was squeezed onto a phone by compressing it to 2 bits per weight with quantization-aware training, and by sharing the KV cache between blocks of the network to cut cache memory by 37.5% (Apple Machine Learning Research). Every one of those choices is a concession to the constraint that this forward pass must finish on a battery.

Batch APIs sell the timing choice as a product. Anthropic's Message Batches API charges 50% of the standard price for requests it may take up to 24 hours to answer, and OpenAI's Batch API does the same. You are not buying a different model. You are selling the provider your latency, and it splits the savings with you.

Key Concepts

• Latency vs. throughput — how fast one answer arrives versus how many answers per second the fleet produces. Tuning for one costs the other; inference optimization is the negotiation between them. Where the latency budget is fixed by the product — a payment authorisation has tens of milliseconds — it selects the model before accuracy is even considered.
• Cost per token, or per prediction — the number that decides whether a feature ships. It is set by hardware, model size and how many users share a forward pass, and almost never by your application code.
• Model versioning and deployment — which weights answered a given request, and being able to say so six months later when someone disputes the answer.
• Monitoring — inference is the only place a model meets real data, so it is the only place you can find out the data changed.

Challenges

The same prompt does not reliably give the same answer — even at temperature 0. The cause is not sampling. Standard GPU kernels for matrix multiplication, RMSNorm and attention change their floating-point summation order depending on the batch size, so a request's numerical result depends on how many other people happened to be batched with it. Thinking Machines Lab documented this and shipped batch-invariant kernels that fix it. Determinism is not free: on Qwen-3-8B over 1,000 sequences their deterministic vLLM took 55 seconds against the default's 26, falling to 42 once the attention kernel was optimised. If you are building evaluations, caching, or anything that assumes reproducibility, this is the assumption that breaks.

The bill grows with success. Every other cost in a machine learning project is bounded: data collection ends, training runs finish. Inference does the opposite — a feature that becomes popular becomes more expensive in exact proportion, and there is no version of "it worked, so we're done paying".

Cold starts are measured in tens of gigabytes. Getting a large model ready to answer means moving its weights into accelerator memory. That is why scale-to-zero serving is easy for a small classifier and painful for a 70B model, and why idle GPUs are still charged for: keeping the weights resident is the service.

The weights are frozen; the world is not. A fraud model trained on last year's fraud degrades silently, returning confident answers about a distribution that no longer exists. Inference is where this shows up and where it must be detected, because the model itself has no way to notice.

Future Trends

• Inference became the scaling axis. Reasoning models spend far more compute per query by design — see test-time compute. The frontier moved from "train a bigger model" to "let it think longer at inference", which puts the cost squarely on the serving side.
• Dedicated inference silicon everywhere. NPUs in phones and laptops, and inference-specific accelerators in data centres, exist because running a model and training one are different workloads with different bottlenecks.
• Serving stacks split the phases apart. Because prompt processing is compute-bound and generation is memory-bound, large deployments increasingly run them on separate pools of hardware — the practical end point of everything on this page, covered in inference optimization.

Code Example

In a framework, the difference between training and inference is two explicit switches. Here it is in PyTorch:

# Training: gradients are tracked and the weights change
model.train()
loss = criterion(model(x), y)
loss.backward()        # the backward pass — inference never does this
optimizer.step()       # the weights are rewritten here

# Inference: the same weights, read-only
model.eval()           # dropout off, batch-norm switches to its running statistics
with torch.no_grad():  # stop recording the graph that gradients would need
    prediction = model(x)

The two lines do different jobs and both matter. model.eval() changes what the network computes: layers such as dropout and batch normalisation behave one way while learning and another way in deployment, and forgetting the call leaves dropout randomly zeroing activations — so the model still answers, just worse, and differently every time. torch.no_grad() changes only what is stored: the output is identical either way, but without it every forward pass builds an autograd graph for a backward pass that will never come, wasting memory for nothing.