Definition
Disaggregated serving runs the two phases of large language model inference — reading the prompt (prefill) and generating the answer one token at a time (decode) — on separate pools of GPUs instead of the same one, because the two phases hit opposite hardware limits. Prefill saturates a GPU's arithmetic units; decode leaves them mostly idle and instead waits on memory bandwidth. Put both on one GPU and each phase degrades the other's latency, so large serving stacks split them, size each pool for the limit it actually hits, and ship the KV cache across the network between them.
The reason this is not obvious is that both phases run the same model on the same GPU architecture, so it looks like one workload. It is two. A prompt of 8,000 tokens goes through the network once, in a single forward pass that does thousands of tokens' worth of matrix multiplication — that is prefill, and it keeps a modern GPU's tensor cores busy. Then the model emits the reply one token at a time, and each of those steps reads the entire multi-gigabyte weight matrix to produce a single token — that is decode, and it spends almost all its time waiting on the memory wall while the arithmetic units sit idle. Ignore this and you build a serving cluster that is simultaneously compute-starved and bandwidth-starved, because you sized every GPU for an average of two workloads that have no average.
How It Works
Every request served by an autoregressive transformer has the same two-act structure, and the acts have different names for their latency budgets.
Prefill processes the whole prompt at once. Because all the input tokens are known up front, the GPU runs them through the model in one big batched matrix multiplication — high arithmetic intensity, tensor cores near saturation. This is the compute-bound phase. The metric it owns is time to first token (TTFT): how long the user waits before anything appears. A few hundred prompt tokens is already enough to push a data-center GPU into the compute-bound regime — the DistServe paper (Zhong et al., OSDI 2024) notes that the prefill of a 512-token sequence is enough to make an A100 near compute-bound for a 13B model.
Decode generates the output one token at a time. Each step feeds the single most recent token back in and must read every weight in the model — tens of gigabytes — to compute the next one. The arithmetic per step is tiny; the memory traffic is enormous. This is the memory-bandwidth-bound phase, and it is why the high-bandwidth memory (HBM) on the chip, not its FLOP rating, sets the ceiling. The metric it owns is inter-token latency (ITL), sometimes called time per output token — the gap between successive tokens as the answer streams.
Why one GPU cannot serve both well
Consider a decode step for a batch of streaming users. Producing one token for the whole batch takes a fixed slice of time — call it 40 ms, which streams each user about 25 tokens per second, comfortable reading speed. The scheduler runs one such step, then the next, and everyone's text flows smoothly.
Now an 8,000-token prompt arrives. On a co-located server the scheduler has to run that prefill, and a single forward pass over 8,000 tokens is not free — suppose it takes about 1,000 ms on this GPU. While that one pass runs, the GPU is doing nothing else. Every streaming user who should have received a token every 40 ms receives zero tokens for a full second. Their inter-token latency spikes from 40 ms to over 1,000 ms — a 25-fold jump — and on screen the answer visibly freezes, then lurches. One long request at the head of the queue has blocked all the short ones behind it. This is head-of-line blocking, and it is the specific failure disaggregation exists to prevent.
The arithmetic underneath is durable: a decode step does the matmul work of one token; a prefill pass over N tokens does roughly N times that work. So the compute packed into one 8,000-token prefill is equivalent to about 8,000 decode steps. The GPU cannot do both at once. Whichever you prioritize, the other's latency target suffers — TTFT and ITL are in direct competition on shared hardware. Worse, the two phases want opposite things from that hardware: prefill wants raw FLOPs, decode wants bandwidth, and no single GPU is optimally provisioned for both.
Notice that the average hides all of this. Across the second, the GPU was busy the whole time, so mean throughput barely moved — while the 99th-percentile inter-token latency exploded. This is why serious stacks measure goodput (requests served within a stated latency target) rather than raw tokens per second.
The disaggregated answer
Disaggregation makes the split physical. Prefill runs on one pool of GPUs, decode on another. A request lands on a prefill instance, which reads the prompt and builds the KV cache; that cache is then transferred over the interconnect to a decode instance, which streams the answer. The two pools are sized independently — you can buy compute-dense chips for prefill and bandwidth-dense chips for decode, and scale each pool to its own queue. Crucially, a long prefill never touches a decode GPU, so it can no longer stall anyone's stream. DistServe reported that this served 7.4x more requests, or met a 12.6x tighter latency target, than state-of-the-art co-located serving (vLLM and DeepSpeed-MII), while staying within latency constraints for over 90% of requests (OSDI 2024). The two phases were never the same workload, and the last thing you should do is average them onto one machine.
Disaggregation is not free, and the cost is the transfer. The KV cache for a long prompt can be gigabytes, and it now has to cross the network from prefill GPU to decode GPU on every request. That only pays off when the interconnect is fast enough and the traffic high enough that isolating the phases raises goodput by more than the transfer costs. You also keep a full copy of the model weights in each pool.
The other answer: chunked prefill
Disaggregation is one of two ways to kill head-of-line blocking; the other keeps the phases co-located and fixes the scheduling instead. Chunked prefill, introduced by Sarathi-Serve (2024), splits a long prefill into small chunks of a few hundred tokens and interleaves each chunk with the ongoing decode steps in the same batch — the paper calls the result "stall-free" batching, because new prefills join a batch "without pausing ongoing decodes." Returning to the example above, instead of one 1,000 ms block, the 8,000-token prefill becomes a run of small chunks, and the longest a decode is ever delayed is a single chunk rather than the whole prompt. Sarathi-Serve reported this improving serving capacity by up to 2.6x for Mistral-7B on a single A100 versus vLLM, within its latency targets (2024).
The two approaches trade off cleanly:
- Chunked prefill needs no KV-cache transfer and no duplicated weights, because nothing moves — it is a scheduler change on existing hardware. But prefill and decode still share the same GPU's compute, and slicing prefill into small chunks lowers its arithmetic intensity, so you pay a little prefill efficiency to bound the stall.
- Disaggregation eliminates interference completely and lets you provision each phase's hardware separately, at the cost of the KV-cache transfer, a fast interconnect, and a second copy of the weights.
Real stacks increasingly do both — disaggregate at the pool level and chunk within the prefill pool — because they attack the same underlying fact from different angles.
Real-World Applications
DeepSeek runs a disaggregated fleet in production and published the numbers. Its V3/R1 inference-system overview describes prefill and decode as separate node groups sized independently — prefill units spanning a handful of nodes, decode units spanning many more — because a node ingests input far faster than it emits output. The full economics of that deployment, and the compute-bound/memory-bound ratio with a number on it, are worked through on the inference optimization page; the point here is that the split is not a research curiosity but the shape of a live serving cluster.
DistServe (OSDI 2024) is the research system that named and measured the pattern, and Splitwise (Microsoft Azure, ISCA 2024) demonstrated the same phase-splitting idea on production-representative traces, matching prefill-heavy and decode-heavy machines to their phases. Both are the primary evidence that separating the phases raises goodput at scale.
Serving frameworks now ship disaggregation as a feature. NVIDIA's Dynamo inference framework (2025) offers disaggregated prefill/decode serving out of the box, and the open-source vLLM ecosystem has added prefill/decode-disaggregation support. When you deploy a large model on managed AI infrastructure today, whether prefill and decode are co-located or disaggregated is increasingly a configuration choice rather than something you build yourself.
The decision someone actually makes differently: an operator sizing a cluster for a chat product with long system prompts and short replies will provision a small compute-dense prefill pool and a large bandwidth-dense decode pool, rather than buying one homogeneous fleet — a purchasing decision that only makes sense once you see prefill and decode as two workloads.
Key Concepts
- TTFT vs ITL — the two latency budgets in tension. Time to first token is prefill's metric; inter-token latency is decode's. Co-location forces a trade between them; disaggregation lets you hit both.
- KV cache transfer — the price of disaggregation. The cache built during prefill must move to the decode GPU across the interconnect, so the pattern lives or dies on network bandwidth between pools.
- Goodput — requests served within a latency target, as opposed to raw throughput. Disaggregation and chunked prefill are both goodput optimizations; neither reliably improves the mean.
- Head-of-line blocking — a long request at the front of a shared queue stalling the short ones behind it. On a co-located GPU, one prefill blocks every stream; it is the concrete harm both techniques remove.
- Compute-bound vs memory-bound — the root cause. Prefill saturates arithmetic; decode saturates memory bandwidth. Because these are different resources on the same chip, one workload can be starved while the other has slack.
Challenges
The KV-cache transfer can eat the win. Disaggregation replaces a compute-interference problem with a network problem: the cache for a long context is large, and moving it per request demands a high-bandwidth interconnect between the prefill and decode pools. On a slow link the transfer latency lands right back in TTFT, and you have paid for two GPU pools to get co-located performance. The technique assumes the kind of fast in-rack interconnect that only makes sense above a certain fleet size.
It duplicates the weights. Each pool holds a full copy of the model, so a small deployment pays twice for weight storage and warm capacity while serving the same traffic a single co-located fleet could handle. Below the crossover load, co-location — with chunked prefill to tame interference — is simply cheaper.
Pool sizing is a live balancing act. Prefill and decode demand shift with the workload: a burst of long documents overloads prefill while decode idles; a burst of long generations does the reverse. Get the ratio wrong and one pool is a bottleneck while the other's GPUs sit unused — the very waste disaggregation was supposed to remove. This makes autoscaling each pool independently a hard, ongoing scheduling problem rather than a one-time configuration.
It only helps at scale. Every benefit above is contingent on traffic. For a single user, a laptop model, or a lightly loaded endpoint, there is no interference to eliminate and no batch to fill — disaggregation adds transfer latency and duplicated cost for nothing. It is a technique for busy fleets, and applying it to a quiet one makes things worse.
Future Trends
The clearest direction is that disaggregation is generalizing from two phases into a pipeline of specialized pools. Once you accept that prefill and decode want different hardware, the same logic extends to giving the attention computation (which is bandwidth-hungry and dominated by the KV cache) and the feed-forward computation (which is compute-hungry) their own resources, and to placing techniques like grouped-query attention and FlashAttention where they help most. The unit of provisioning is shifting from "a GPU that does everything" toward "a pool sized for one bottleneck."
The second is that the interconnect is becoming the design center. As soon as the KV cache is something you move rather than something you keep local, the network fabric between GPUs — its bandwidth and latency — becomes as important to serving performance as the GPUs themselves, which is pushing the industry toward larger tightly-coupled scale-up domains where cross-pool transfer is cheap. Disaggregated serving turned a memory-locality question into a networking one, and the systems that win will be the ones with the fastest wire between their pools.