Term tag: Inference Optimization

Disaggregated Serving (Prefill/Decode)

Disaggregated serving runs LLM prefill and decode on separate GPU pools because the two phases hit opposite limits: compute versus memory bandwidth.

FlashAttention

The exact-attention algorithm that made long context affordable: it never writes the n x n score matrix to GPU memory, so it runs faster and changes no output.

Grouped-Query Attention (GQA)

Grouped-query attention lets query heads share key/value heads, shrinking the KV cache that dominates long-context inference at near multi-head quality.

Inference Optimization

How a served LLM is made cheap: continuous batching, the prefill/decode split, and the throughput-versus-latency tradeoff behind every API price.

KV Cache

The memory an LLM keeps so it never recomputes the past — why a long context costs money, and how many users one GPU can actually serve at once.

Model Routing

Model routing sends each request to the cheapest capable model — a small model handles the easy majority and hard queries escalate to a frontier model.