---
source: 'https://howaiworks.ai/glossary/inference-optimization'
section: glossary
title: Inference Optimization
description: >-
  How a served LLM is made cheap: continuous batching, the prefill/decode split,
  and the throughput-versus-latency tradeoff behind every API price.
tags:
  - Inference Optimization
  - Inference
  - AI Infrastructure
  - LLM
  - Performance Optimization
  - GPU
  - Memory Bandwidth
  - Efficiency
category: Infrastructure & Hardware
datePublished: '2026-07-13'
lastUpdated: '2026-07-13'
---

# Inference Optimization

> How a served LLM is made cheap: continuous batching, the prefill/decode split, and the throughput-versus-latency tradeoff behind every API price.

## Definition

**The same model, on the same GPUs, running the same code, can cost 47x more to serve depending on
one number that no user ever sees.** That number is the batch size — how many people's requests the
server pushes through each forward pass — and inference optimization is, first and mostly, the craft
of keeping it high without making the product feel slow.

Everything else follows from a single awkward fact: **throughput and latency are the same dial
turned in opposite directions.** Raise the batch and the server produces far more tokens per second
per GPU, so the cost of a token collapses; every individual user also waits longer between words. No
configuration maximises both. An operator picks a point on that curve, and that choice is what you
feel when a chatbot streams smoothly, and what you pay when the invoice arrives.

## How It Works

### Two clocks, two bottlenecks

Serving a request has two phases, and the metrics that matter map one-to-one onto them
([Databricks](https://www.databricks.com/blog/llm-inference-performance-engineering-best-practices)
names them the way the industry now does):

- **Time to first token (TTFT)** — how long before anything appears. It covers queueing plus
  *prefill*: the model reading your whole prompt in one parallel pass. That pass is a dense matrix
  multiplication and it is **compute-bound**.
- **Time per output token (TPOT)** — how fast the words then arrive. Each one is a *decode* step,
  which reads all of the model's weights and all of the cached context to produce a single token.
  It is **memory-bound**.

Two phases, opposite limits — the [memory wall](https://howaiworks.ai/glossary/memory-wall) page derives why. What
matters here is the consequence: **an optimisation that helps one clock usually hurts the other**,
and a serving stack is a negotiated settlement between them.

### Static batching stalls at the speed of its slowest sequence

The obvious way to batch is to collect eight requests, run them together, and return the results.
It fails, and the arithmetic shows how badly.

Say seven of the eight requests want 100 tokens and one wants 1,000 — a perfectly ordinary spread,
since nobody knows in advance how long an answer will be. With static batching the batch slot is
held until the *last* sequence finishes. After step 100 seven slots are dead, and the GPU spends the
next 900 steps doing full-width forward passes for one real user. Useful work as a fraction of what
the batch occupied: `(7 x 100 + 1,000) / (8 x 1,000) = 21%`. **Four fifths of the machine is
computing padding.**

### Continuous batching: schedule the iteration, not the request

The fix is to stop treating a request as the unit of scheduling. **Continuous batching** — also
called in-flight batching, and introduced as *iteration-level scheduling* by
[Orca](https://www.usenix.org/conference/osdi22/presentation/yu) (OSDI 2022) — reschedules after
*every forward pass*. A sequence that emitted its end-of-text token is evicted immediately, a
request waiting in the queue takes its slot on the very next step, and the batch becomes a rolling
population rather than a fixed cohort. Nothing waits for a straggler, because there is no cohort for
a straggler to hold up. Orca reported up to **36.9x the throughput of FasterTransformer at the same
latency** — not from faster maths, but from never running a padded slot.

Admitting a newcomer, though, means prefilling its prompt, and an 8,000-token prefill wedged into a
decode step makes every user in the batch wait for it. That stutter is why
[Sarathi-Serve](https://arxiv.org/abs/2403.02310) (OSDI 2024) splits a prefill into chunks and
mixes them into decode batches, keeping each iteration roughly the same size — worth **2.6x the
serving capacity** of vLLM on Mistral-7B, and **3.5x** under a strict 100 ms latency target. The
name of that paper is "Taming Throughput-Latency Tradeoff", which is the honest description of the
whole field.

### The worked example: one dial, 47x the price

Take a standard deployment: **Llama 3.1 70B in FP8 on one eight-GPU H100 SXM node.** Three inputs,
all sourced:

- Weights: **70.6 GB** (70.6B parameters at one byte each — see
  [quantization](https://howaiworks.ai/glossary/quantization)).
- KV cache: **320 KiB per token** ([KV cache](https://howaiworks.ai/glossary/kv-cache) derives it), so a 4,096-token
  conversation carries **1.34 GB**.
- Hardware: eight H100 SXM, each **80 GB** of HBM at **3.35 TB/s**
  ([NVIDIA](https://www.nvidia.com/en-us/data-center/h100/)) — **640 GB** and **26.8 TB/s** in
  aggregate. Rented at Lambda's on-demand rate of **$3.99 per GPU-hour**
  ([Lambda](https://lambda.ai/pricing)), the node costs **$31.92/hour**.

The two prices below — the rental rate and the API rate quoted against it — are list prices at the
time of writing, and they will move. Read the table for its **ratios**, which are set by the
hardware and the model rather than by anyone's price list: the cost per token falls roughly with
the batch size until the caches overtake the weights, and it stops falling after that. Redo the
arithmetic with today's rates and the shape is unchanged.

A decode step reads the weights once and every batched sequence's cache once, so with batch size
`B`:

```
bytes read per step = 70.6 GB + B x 1.34 GB
time per step       = bytes / 26,800 GB/s        <- this is TPOT
node throughput     = B / time per step
```

| Batch | Bytes read per step | TPOT | Speed one user sees | Node throughput | Cost per 1M tokens |
|---|---|---|---|---|---|
| 1 | 71.9 GB | 2.7 ms | 373 tok/s | 373 tok/s | **$23.80** |
| 8 | 81.3 GB | 3.0 ms | 330 tok/s | 2,636 tok/s | $3.36 |
| 32 | 113.5 GB | 4.2 ms | 236 tok/s | 7,553 tok/s | $1.17 |
| 128 | 242.4 GB | 9.0 ms | 111 tok/s | 14,152 tok/s | $0.63 |
| 384 | 586.0 GB | 21.9 ms | 46 tok/s | 17,562 tok/s | **$0.51** |

Read the two bold numbers first. Serving one user at a time on hardware that costs $32 an hour puts
the price of a million tokens at **$23.80** — while the same node, same model, batch of 384, serves
them for **51 cents**. That is the entire gap between a hobby deployment and a business. You can
read a provider's batch size straight off the table: Together AI lists a 70B-class model at
**$1.04 per million tokens** ([pricing](https://www.together.ai/pricing)) — *below* what the
batch-of-32 row costs to produce, which means nobody selling at that price is running batches of 32.

And look what the user paid for it. TPOT went from 2.7 ms to 21.9 ms — from 373 tokens per second to
**46**, an **8x** slowdown in the speed of the streaming text — to buy a **47x** cut in cost.

### Where the free lunch ends

The table's last two rows are the interesting ones: throughput rises only **24%** from batch 128 to
batch 384, while TPOT gets **2.4x worse**. Batching stops paying, and the formula says exactly when.

Batching is free while it *amortises a fixed cost*: the 70.6 GB of weights get read once and serve
the whole batch. But each sequence brings its own cache, and that is a *variable* cost — bytes that
must be re-read on every step for every user. The two are equal when `B x 1.34 GB = 70.6 GB`, at
**B ≈ 53**. Below that, doubling the batch nearly doubles throughput. Above it, the cache reads
dominate, throughput asymptotes, and all you are still buying is latency for your users. (At batch
384 the node is also nearly out of memory — 586 GB of a 640 GB budget — so it could not go much
further even if it wanted to.)

The model holds only while decode stays memory-bound, and it does: at batch 384 the arithmetic runs
at roughly 310 TFLOP/s per GPU against the ~1,979 dense FP8 TFLOPS an H100 offers. It ignores
attention kernels, sampling and the interconnect, so these are *ceilings* — a real stack lands well
under them, and it is the shape that transfers, not the absolute numbers.

### Prefill and decode want different machines

Because prefill is compute-bound and decode is memory-bound, running both on one GPU means every
prompt you admit steals time from every answer you are streaming. **Disaggregation** takes the
obvious step: two pools of hardware, one sized for compute and one for bandwidth, with the KV cache
shipped across the network between them.
[DistServe](https://www.usenix.org/conference/osdi24/presentation/zhong-yinmin) (OSDI 2024) showed
this serves **7.4x more requests, or meets a 12.6x tighter latency target**, than co-located
serving. The two phases were never the same workload, and the last thing you should do is average
them.

## Real-World Applications

**DeepSeek published its meter reading, and it is this page in production.** Its
[V3/R1 inference system overview](https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md)
describes a disaggregated fleet — prefill units spanning 4 nodes, decode units spanning 18 —
averaging **226.75 H800 nodes** over a day, at roughly **73.7k input tokens/s per node in prefill**
against **14.8k output tokens/s per node in decode**. That ratio *is* the compute-bound /
memory-bound split with a number on it: a node emits output about **5x** slower than it ingests
input. The economics that fall out: at an assumed **$2 per GPU-hour**, **$87,072 of compute per
day** against **$562,027** of tokens at list price — a theoretical **545% margin** — while each user
still saw a comfortable **20–22 tokens per second**. That is the curve, with someone standing on a
well-chosen point of it.

**A batch API is this tradeoff sold as a product.** Anthropic's
[Message Batches API](https://www.anthropic.com/news/message-batches-api) charges **50% of the
standard price** for requests it may take up to 24 hours to answer, and OpenAI's Batch API does the
same. Nothing about the model changed. You handed the scheduler a 24-hour latency budget instead of
a two-second one, it packed you into whatever batch had room whenever the fleet was quiet, and it
split the savings with you.

**Running a model on your own GPU is the worst row of the table.** Batch size 1: nobody to amortise
the weight read with, and an effective cost per token an order of magnitude above what an API
charges. Local [inference](https://howaiworks.ai/glossary/inference) is worth doing for privacy, offline use or control —
essentially never to save money, and the reason is arithmetic, not the quality of your software.

## Key Concepts

Batching is what this page owns. The other levers are real, each has its own page, and each is one
sentence here on purpose:

- **[Quantization](https://howaiworks.ai/glossary/quantization)** shrinks the *fixed* term in the formula above — fewer
  bytes per weight means a faster decode step at every batch size, and the memory it frees becomes
  more batch.
- **[KV cache](https://howaiworks.ai/glossary/kv-cache) management** attacks the *variable* term: fewer KV heads, fewer
  bits, and paged allocation all mean more sequences fit in the same memory, which is the only way
  to reach the big batches at all.
- **[Speculative decoding](https://howaiworks.ai/glossary/speculative-decoding)** buys single-user latency by spending
  idle compute — which is exactly the resource a *full* batch no longer has, so it helps least
  where throughput is already best.
- **[Mixture-of-Experts](https://howaiworks.ai/glossary/mixture-of-experts)** cuts the bytes read per token by activating
  a fraction of the parameters, shrinking the fixed term far more aggressively than quantization
  can.
- **Prefix caching** skips prefill entirely for a repeated prompt, which is a TTFT optimisation and
  does nothing at all for TPOT.

## Challenges

**Optimise only for throughput and you ship a product that feels broken.** The failure is rarely in
the average — it is in the tail. A scheduler chasing GPU utilisation will admit a large batch and a
long prefill at the moment your user is mid-sentence, and their stream stalls for hundreds of
milliseconds. The dashboard shows tokens per second climbing; the user sees the text stop, and
leaves. This is why serious systems optimise *goodput* — requests served **within a stated latency
target** — rather than raw tokens per second, and why a mean TPOT is nearly useless next to its 99th
percentile.

**Optimise only for latency and you cannot price your product.** The batch-of-8 row is a lovely user
experience and a $3.36-per-million-token cost base in a market charging $1.04. No engineering closes
that gap, because it is not an engineering gap; it is the batch size. Teams find this out after
launch, working out why the gross margin is negative and discovering that the answer is a scheduler
setting.

**The benchmark you ran does not resemble your traffic.** Most published serving benchmarks use a
fixed batch and fixed sequence lengths. Real traffic arrives in bursts, with output lengths that vary
by orders of magnitude — and a system tuned on the synthetic load can be *worse* under the real one,
because it was optimised for a batch that never has to evict anyone.

**Long context quietly deletes your batch.** The variable term scales with context length as well as
batch size, so the node that held 384 sequences of 4,096 tokens holds **13** at 128k — and
throughput falls roughly **30x** with it. Nothing in your configuration changed; the users just
started pasting in documents. The [context window](https://howaiworks.ai/glossary/context-window) is a cost parameter,
not only a capability.

## Future Trends

The shift already underway is from *throughput* to *goodput*: schedulers that take a latency target
as an input and maximise the number of requests meeting it, rather than tokens produced per second.
That reframing is what makes [disaggregation](https://howaiworks.ai/glossary/disaggregated-serving), chunked prefill and SLO-aware admission control one
idea rather than three tricks — each is a way of spending a latency budget the operator actually
declared.

The second is that the workload is changing shape underneath the optimisation. Reasoning models and
agents emit enormous numbers of tokens *no human ever reads* — chains of thought, tool calls,
retries — which are pure decode, and therefore pure memory-bandwidth cost. As
[test-time compute](https://howaiworks.ai/glossary/test-time-compute) becomes the way models get smarter, decode stops
being the tail of a request and becomes the entire bill. A workload that is mostly output tokens
with nobody waiting on them is also, conveniently, one where the operator can turn the batch dial
all the way up. The tradeoff does not disappear. It gets easier to resolve, because the impatient
human at the other end of it has left the loop.

## Frequently Asked Questions

### What is inference optimization?

It is everything a serving system does to get more tokens per second per dollar out of a model whose weights nobody is changing. The single biggest lever is batching — running many users' requests through the same forward pass — and the price of pulling that lever is that each individual user waits longer for each word.

### What is continuous batching?

Scheduling at the granularity of one forward pass instead of one request. After every step the server drops sequences that have finished and admits waiting ones in their place, so the batch is a rolling set rather than a fixed group. It is the difference between a GPU running at a fraction of capacity and one that stays full.

### What is the difference between TTFT and TPOT?

Time to first token is how long you wait before anything appears; it is set by prompt processing, which is compute-bound. Time per output token is how fast the words then arrive; it is set by generation, which is memory-bound. They have different bottlenecks, they respond to different fixes, and a system tuned for one can be bad at the other.

### Does a bigger batch always mean more throughput?

No. Batching is free only while it amortises a fixed cost — reading the model's weights. Once the batch's key-value caches are bigger than the weights, every extra request adds bytes that must be read on every step, and throughput flattens while per-user latency keeps getting worse.

### Why is a batch API 50% cheaper than a real-time one?

Because you are selling the provider your latency. Given a 24-hour window instead of a two-second one, the scheduler can pack requests into large batches and run them when its fleet is otherwise idle, which raises tokens per second per GPU. The discount is the throughput-latency tradeoff, priced.

### What is prefill/decode disaggregation?

Running the two phases of a request on separate pools of hardware, because they hit opposite bottlenecks. Prompt processing wants compute; token generation wants memory bandwidth. Keeping them on the same GPU means each phase interferes with the other's latency, so large serving stacks split them and ship the cache between them.

## Related

### Related terms

- [Inference](https://howaiworks.ai/glossary/inference)
- [KV Cache](https://howaiworks.ai/glossary/kv-cache)
- [Quantization](https://howaiworks.ai/glossary/quantization)
- [Memory Wall](https://howaiworks.ai/glossary/memory-wall)
- [Model Deployment](https://howaiworks.ai/glossary/model-deployment)
- [Performance](https://howaiworks.ai/glossary/performance)

---

Source: https://howaiworks.ai/glossary/inference-optimization — HowAIWorks.ai