Inference Optimization

Definition

The same model, on the same GPUs, running the same code, can cost 47x more to serve depending on one number that no user ever sees. That number is the batch size — how many people's requests the server pushes through each forward pass — and inference optimization is, first and mostly, the craft of keeping it high without making the product feel slow.

Everything else follows from a single awkward fact: throughput and latency are the same dial turned in opposite directions. Raise the batch and the server produces far more tokens per second per GPU, so the cost of a token collapses; every individual user also waits longer between words. No configuration maximises both. An operator picks a point on that curve, and that choice is what you feel when a chatbot streams smoothly, and what you pay when the invoice arrives.

How It Works

Two clocks, two bottlenecks

Serving a request has two phases, and the metrics that matter map one-to-one onto them (Databricks names them the way the industry now does):

• Time to first token (TTFT) — how long before anything appears. It covers queueing plus prefill: the model reading your whole prompt in one parallel pass. That pass is a dense matrix multiplication and it is compute-bound.
• Time per output token (TPOT) — how fast the words then arrive. Each one is a decode step, which reads all of the model's weights and all of the cached context to produce a single token. It is memory-bound.

Two phases, opposite limits — the memory wall page derives why. What matters here is the consequence: an optimisation that helps one clock usually hurts the other, and a serving stack is a negotiated settlement between them.

Static batching stalls at the speed of its slowest sequence

The obvious way to batch is to collect eight requests, run them together, and return the results. It fails, and the arithmetic shows how badly.

Say seven of the eight requests want 100 tokens and one wants 1,000 — a perfectly ordinary spread, since nobody knows in advance how long an answer will be. With static batching the batch slot is held until the last sequence finishes. After step 100 seven slots are dead, and the GPU spends the next 900 steps doing full-width forward passes for one real user. Useful work as a fraction of what the batch occupied: (7 x 100 + 1,000) / (8 x 1,000) = 21%. Four fifths of the machine is computing padding.

Continuous batching: schedule the iteration, not the request

The fix is to stop treating a request as the unit of scheduling. Continuous batching — also called in-flight batching, and introduced as iteration-level scheduling by Orca (OSDI 2022) — reschedules after every forward pass. A sequence that emitted its end-of-text token is evicted immediately, a request waiting in the queue takes its slot on the very next step, and the batch becomes a rolling population rather than a fixed cohort. Nothing waits for a straggler, because there is no cohort for a straggler to hold up. Orca reported up to 36.9x the throughput of FasterTransformer at the same latency — not from faster maths, but from never running a padded slot.

Admitting a newcomer, though, means prefilling its prompt, and an 8,000-token prefill wedged into a decode step makes every user in the batch wait for it. That stutter is why Sarathi-Serve (OSDI 2024) splits a prefill into chunks and mixes them into decode batches, keeping each iteration roughly the same size — worth 2.6x the serving capacity of vLLM on Mistral-7B, and 3.5x under a strict 100 ms latency target. The name of that paper is "Taming Throughput-Latency Tradeoff", which is the honest description of the whole field.

The worked example: one dial, 47x the price

Take a standard deployment: Llama 3.1 70B in FP8 on one eight-GPU H100 SXM node. Three inputs, all sourced:

• Weights: 70.6 GB (70.6B parameters at one byte each — see quantization).
• KV cache: 320 KiB per token (KV cache derives it), so a 4,096-token conversation carries 1.34 GB.
• Hardware: eight H100 SXM, each 80 GB of HBM at 3.35 TB/s (NVIDIA) — 640 GB and 26.8 TB/s in aggregate. Rented at Lambda's on-demand rate of $3.99 per GPU-hour (Lambda), the node costs $31.92/hour.

The two prices below — the rental rate and the API rate quoted against it — are list prices at the time of writing, and they will move. Read the table for its ratios, which are set by the hardware and the model rather than by anyone's price list: the cost per token falls roughly with the batch size until the caches overtake the weights, and it stops falling after that. Redo the arithmetic with today's rates and the shape is unchanged.

A decode step reads the weights once and every batched sequence's cache once, so with batch size B:

bytes read per step = 70.6 GB + B x 1.34 GB
time per step       = bytes / 26,800 GB/s        <- this is TPOT
node throughput     = B / time per step

Read the two bold numbers first. Serving one user at a time on hardware that costs $32 an hour puts the price of a million tokens at $23.80 — while the same node, same model, batch of 384, serves them for 51 cents. That is the entire gap between a hobby deployment and a business. You can read a provider's batch size straight off the table: Together AI lists a 70B-class model at $1.04 per million tokens (pricing) — below what the batch-of-32 row costs to produce, which means nobody selling at that price is running batches of 32.

And look what the user paid for it. TPOT went from 2.7 ms to 21.9 ms — from 373 tokens per second to 46, an 8x slowdown in the speed of the streaming text — to buy a 47x cut in cost.

Where the free lunch ends

The table's last two rows are the interesting ones: throughput rises only 24% from batch 128 to batch 384, while TPOT gets 2.4x worse. Batching stops paying, and the formula says exactly when.

Batching is free while it amortises a fixed cost: the 70.6 GB of weights get read once and serve the whole batch. But each sequence brings its own cache, and that is a variable cost — bytes that must be re-read on every step for every user. The two are equal when B x 1.34 GB = 70.6 GB, at B ≈ 53. Below that, doubling the batch nearly doubles throughput. Above it, the cache reads dominate, throughput asymptotes, and all you are still buying is latency for your users. (At batch 384 the node is also nearly out of memory — 586 GB of a 640 GB budget — so it could not go much further even if it wanted to.)

The model holds only while decode stays memory-bound, and it does: at batch 384 the arithmetic runs at roughly 310 TFLOP/s per GPU against the ~1,979 dense FP8 TFLOPS an H100 offers. It ignores attention kernels, sampling and the interconnect, so these are ceilings — a real stack lands well under them, and it is the shape that transfers, not the absolute numbers.

Prefill and decode want different machines

Because prefill is compute-bound and decode is memory-bound, running both on one GPU means every prompt you admit steals time from every answer you are streaming. Disaggregation takes the obvious step: two pools of hardware, one sized for compute and one for bandwidth, with the KV cache shipped across the network between them. DistServe (OSDI 2024) showed this serves 7.4x more requests, or meets a 12.6x tighter latency target, than co-located serving. The two phases were never the same workload, and the last thing you should do is average them.

Real-World Applications

DeepSeek published its meter reading, and it is this page in production. Its V3/R1 inference system overview describes a disaggregated fleet — prefill units spanning 4 nodes, decode units spanning 18 — averaging 226.75 H800 nodes over a day, at roughly 73.7k input tokens/s per node in prefill against 14.8k output tokens/s per node in decode. That ratio is the compute-bound / memory-bound split with a number on it: a node emits output about 5x slower than it ingests input. The economics that fall out: at an assumed $2 per GPU-hour, $87,072 of compute per day against $562,027 of tokens at list price — a theoretical 545% margin — while each user still saw a comfortable 20–22 tokens per second. That is the curve, with someone standing on a well-chosen point of it.

A batch API is this tradeoff sold as a product. Anthropic's Message Batches API charges 50% of the standard price for requests it may take up to 24 hours to answer, and OpenAI's Batch API does the same. Nothing about the model changed. You handed the scheduler a 24-hour latency budget instead of a two-second one, it packed you into whatever batch had room whenever the fleet was quiet, and it split the savings with you.

Running a model on your own GPU is the worst row of the table. Batch size 1: nobody to amortise the weight read with, and an effective cost per token an order of magnitude above what an API charges. Local inference is worth doing for privacy, offline use or control — essentially never to save money, and the reason is arithmetic, not the quality of your software.

Key Concepts

Batching is what this page owns. The other levers are real, each has its own page, and each is one sentence here on purpose:

• Quantization shrinks the fixed term in the formula above — fewer bytes per weight means a faster decode step at every batch size, and the memory it frees becomes more batch.
• KV cache management attacks the variable term: fewer KV heads, fewer bits, and paged allocation all mean more sequences fit in the same memory, which is the only way to reach the big batches at all.
• Speculative decoding buys single-user latency by spending idle compute — which is exactly the resource a full batch no longer has, so it helps least where throughput is already best.
• Mixture-of-Experts cuts the bytes read per token by activating a fraction of the parameters, shrinking the fixed term far more aggressively than quantization can.
• Prefix caching skips prefill entirely for a repeated prompt, which is a TTFT optimisation and does nothing at all for TPOT.

Challenges

Optimise only for throughput and you ship a product that feels broken. The failure is rarely in the average — it is in the tail. A scheduler chasing GPU utilisation will admit a large batch and a long prefill at the moment your user is mid-sentence, and their stream stalls for hundreds of milliseconds. The dashboard shows tokens per second climbing; the user sees the text stop, and leaves. This is why serious systems optimise goodput — requests served within a stated latency target — rather than raw tokens per second, and why a mean TPOT is nearly useless next to its 99th percentile.

Optimise only for latency and you cannot price your product. The batch-of-8 row is a lovely user experience and a $3.36-per-million-token cost base in a market charging $1.04. No engineering closes that gap, because it is not an engineering gap; it is the batch size. Teams find this out after launch, working out why the gross margin is negative and discovering that the answer is a scheduler setting.

The benchmark you ran does not resemble your traffic. Most published serving benchmarks use a fixed batch and fixed sequence lengths. Real traffic arrives in bursts, with output lengths that vary by orders of magnitude — and a system tuned on the synthetic load can be worse under the real one, because it was optimised for a batch that never has to evict anyone.

Long context quietly deletes your batch. The variable term scales with context length as well as batch size, so the node that held 384 sequences of 4,096 tokens holds 13 at 128k — and throughput falls roughly 30x with it. Nothing in your configuration changed; the users just started pasting in documents. The context window is a cost parameter, not only a capability.

Future Trends

The shift already underway is from throughput to goodput: schedulers that take a latency target as an input and maximise the number of requests meeting it, rather than tokens produced per second. That reframing is what makes disaggregation, chunked prefill and SLO-aware admission control one idea rather than three tricks — each is a way of spending a latency budget the operator actually declared.

The second is that the workload is changing shape underneath the optimisation. Reasoning models and agents emit enormous numbers of tokens no human ever reads — chains of thought, tool calls, retries — which are pure decode, and therefore pure memory-bandwidth cost. As test-time compute becomes the way models get smarter, decode stops being the tail of a request and becomes the entire bill. A workload that is mostly output tokens with nobody waiting on them is also, conveniently, one where the operator can turn the batch dial all the way up. The tradeoff does not disappear. It gets easier to resolve, because the impatient human at the other end of it has left the loop.