Term tag: Memory Bandwidth

AI Infrastructure

Why AI clusters are designed around memory bandwidth and interconnect rather than FLOPs, and what that changes when you size a training run or a serving fleet.

Disaggregated Serving (Prefill/Decode)

Disaggregated serving runs LLM prefill and decode on separate GPU pools because the two phases hit opposite limits: compute versus memory bandwidth.

FlashAttention

The exact-attention algorithm that made long context affordable: it never writes the n x n score matrix to GPU memory, so it runs faster and changes no output.

Grouped-Query Attention (GQA)

Grouped-query attention lets query heads share key/value heads, shrinking the KV cache that dominates long-context inference at near multi-head quality.

Inference Optimization

How a served LLM is made cheap: continuous batching, the prefill/decode split, and the throughput-versus-latency tradeoff behind every API price.

Interconnect (NVLink, InfiniBand)

The wiring that moves data between AI chips: on-package memory, NVLink inside a server, InfiniBand between them — and why each tier is far slower than the last.

KV Cache

The memory an LLM keeps so it never recomputes the past — why a long context costs money, and how many users one GPU can actually serve at once.

Neural Processing Unit (NPU)

An NPU is a fixed-function matrix engine inside a phone or laptop chip that runs AI models at a few watts — and why TOPS is the wrong number to…

Quantization

Running a model in fewer bits — FP16, FP8, INT4 — to cut memory and cost, what accuracy it actually costs, and why a memory-bound model also gets faster.