Definition
A modern GPU can multiply numbers far faster than it can read them, and that gap — not the speed of the arithmetic — is what limits most AI systems today. The memory wall is the widening gap between processor throughput and memory bandwidth. It has a sharp consequence for language models: when a chatbot is streaming words at you, the accelerator producing them is running its arithmetic units at well under 1% of their rated capacity. It is not computing. It is waiting for the model's weights to arrive from memory, over and over, once per word.
The term comes from a 1995 note by William Wulf and Sally McKee, Hitting the Memory Wall, which pointed out something merely arithmetical: if processor speed and memory speed both improve exponentially but at different rates, the gap between them also grows exponentially, and eventually every program is a memory program. Thirty years on, that is where large-model inference sits.
How It Works
Arithmetic intensity: how much maths you do per byte you move
Every computation has a ratio you can measure: floating-point operations performed, divided by bytes moved between memory and the chip. This is its arithmetic intensity, in FLOPs per byte. Multiplying two large matrices has high intensity — each element loaded is reused across many rows and columns, so a lot of maths happens per byte fetched. Adding two vectors has an intensity near zero: read two numbers, do one addition, write one number, repeat.
Intensity is a property of the job, not the hardware, and it decides which of the machine's two speed limits you hit.
The roofline: every chip has a break-even intensity
The roofline model (Williams, Waterman and Patterson, CACM 2009) makes this concrete. A chip has a peak arithmetic rate and a peak memory bandwidth, and dividing one by the other gives its break-even intensity — the FLOPs per byte a workload must supply to keep the arithmetic units fed.
Take NVIDIA's H100 SXM, whose published specification is 3.35 TB/s of memory bandwidth and 3,958 teraFLOPS of FP8 tensor-core throughput — "with sparsity", meaning the dense figure you can actually count on is half that, 1,979 TFLOPS. Its break-even intensity is therefore:
1,979 × 10¹² FLOP/s ÷ 3.35 × 10¹² bytes/s ≈ 590 FLOPs per byte
A workload that does more than ~590 FLOPs for every byte it reads is compute-bound: the tensor cores are the limit, and faster memory would buy you nothing. A workload below that line is memory-bound: it is limited by bandwidth, and a chip with twice the FLOPs would run it at exactly the same speed. Note which way that threshold has been moving. Over the past twenty years, peak server FLOPS grew 3.0× every two years while DRAM bandwidth grew only 1.6×, according to AI and Memory Wall (Gholami et al., IEEE Micro, 2024). Compound those for a decade and compute rises 3⁵ ≈ 243× against bandwidth's 1.6⁵ ≈ 10.5× — the break-even intensity climbs roughly 23× every ten years. The bar for being compute-bound keeps rising, so more and more real work falls below it.
Generating one token is 2 FLOPs per byte
Here is the fact this whole page exists for. An autoregressive model produces text one token at a time, and each token is a full forward pass through the network. That forward pass touches every weight in the model exactly once — so every weight has to be read out of memory. The arithmetic it does with them is roughly two FLOPs per parameter (one multiply and one add in the matrix multiplications; the FLOPs page owns that estimate).
Two FLOPs per parameter. One byte per parameter, if the model is stored in 8-bit precision. The arithmetic intensity of generating a token for a single user is therefore 2 FLOPs per byte — against a break-even of 590. The workload is not slightly memory-bound. It is nearly 300× below the line.
The worked example: a token-rate ceiling you can compute in one line
Llama 3.1 70B has 70.6 billion parameters (model card). Serve it in FP8, one byte per weight:
- Weights to read, per token:
70.6 × 10⁹ params × 1 byte = 70.6 GB - Time to read them at the H100's spec bandwidth:
70.6 GB ÷ 3,350 GB/s = 21.1 ms - Ceiling on generation speed:
1 ÷ 0.0211 s = 47 tokens per second
That is a hard ceiling for a single request, from two numbers: the size of the model and the bandwidth of the memory. Nothing about the GPU's arithmetic capability appears in it. A vendor cannot beat it with a better kernel, and a real system will land under it once you count the KV cache, attention, and communication.
Now look at what the tensor cores did during those 21.1 ms. The maths per token is
2 × 70.6 × 10⁹ = 141 GFLOP. At 47 tokens per second that is 141 GFLOP × 47 ≈ 6.7 TFLOP/s —
against 1,979 TFLOPS available. The H100 is running at 0.34% of its arithmetic capacity. Put
the other way: the chip spends 0.07 ms doing the multiplications and 21.1 ms waiting for the
numbers to multiply. Roughly 99.7% of the silicon you are paying for is idle while the model
"generates fast".
Prefill sits on the other side of the ridge
Processing the prompt is a different workload with the same weights. A 10,000-token prompt is processed in one pass, so each weight that gets read is used for 10,000 tokens' worth of arithmetic: the intensity is 2 × 10,000 = 20,000 FLOPs per byte, well past the break-even, and the GPU is genuinely compute-bound. This is why the first token takes a while and the rest arrive at a steady drip — the two halves of a request live on opposite sides of the roofline, and they want different optimisations and, increasingly, different machines.
Real-World Applications
Batching is the entire economics of an inference server. If the GPU reads the 70.6 GB of weights once and uses them for 64 requests at the same time, the bytes read stay the same while the arithmetic done with them goes up 64-fold: the intensity becomes 2 × 64 = 128 FLOPs per byte, and the server produces roughly 64× the tokens per second in aggregate at almost no cost in latency per user. That is why inference servers exist instead of everyone running a model locally, and why systems like vLLM work so hard to keep the batch full. Extending the arithmetic: you would need about 295 concurrent requests to reach the H100's break-even intensity of 590, which is why a serving fleet is never compute-bound — only less catastrophically memory-bound.
Quantization is a bandwidth optimisation before it is a capacity one. Halving the bytes per weight halves step 1 of the calculation above and therefore roughly halves the time to produce a token. Dropping our 70.6B model from FP8 to 4-bit takes the weights from 70.6 GB to 35.3 GB and the ceiling from 47 to about 95 tokens per second on the same GPU, with nothing about the chip changed. See quantization for what the formats do and what accuracy costs.
Mixture-of-Experts is a way to read fewer bytes per token. DeepSeek-V3 has 671B total parameters but activates only 37B for each token (technical report) — so a decode step reads about 37 GB instead of 671 GB, an 18× cut in the only quantity that sets the token rate. Enormous in memory, cheap in bandwidth: exactly the trade the memory wall rewards. See Mixture-of-Experts.
The KV cache is the other thing being read. Weights are not the only bytes crossing the bus each step: attention re-reads the cached keys and values for every previous token in the conversation. That cache grows with the context window and with the batch, and it competes for the same bandwidth — which is why long conversations slow down token-by-token even though the weights have not changed. The KV cache page does the size arithmetic.
HBM scarcity is this page in commercial form. An H100 carries 80 GB of memory not because 80 GB is a lot — a two-socket server can hold terabytes of ordinary DRAM — but because that memory is High Bandwidth Memory, delivering 3.35 TB/s. Bandwidth is what the accelerator is short of, HBM is how it gets bandwidth, and the world cannot make enough: SK hynix said in October 2025 that its DRAM, NAND and HBM capacity for the whole of the following year was already sold out (TechSpot). The queue for HBM is the memory wall with a price tag on it.
Key Concepts
- Break-even intensity: a job is memory-bound when it does fewer FLOPs per byte than the machine's ratio of peak arithmetic to peak bandwidth. Decoding one token at 2 FLOPs/byte falls under that line on every accelerator ever built, and no future chip rescues it by adding FLOPs.
- The bytes are the thing you optimise: every effective inference optimisation — batching, quantization, MoE routing, speculative decoding, KV-cache compression — either moves fewer bytes or extracts more arithmetic from the bytes it already moved. There is no third option.
- Peak FLOPS is a marketing number for decoding: it does not appear anywhere in the 47 tokens-per-second calculation. Memory bandwidth and model size do.
Challenges
What breaks if you get this wrong is your capacity plan. Size a fleet by dividing your expected token volume by an accelerator's advertised TFLOPS and you will over-predict its throughput by two or three orders of magnitude — the H100 above delivers 0.34% of its peak arithmetic during decode. The same mistake at purchase time buys the chip with the biggest FLOPs number instead of the one with the most bandwidth, and the two rankings are not the same list.
It sends optimisation effort to the wrong place. Hand-tuning a CUDA matmul kernel to shave 20% off the arithmetic of a decode step, when the arithmetic units are 99.7% idle, buys you nothing at all. The wins on a memory-bound path come from moving fewer bytes or from fusing operations so that intermediate values never leave the chip.
Latency and cost per token are the same dial. Because throughput comes from batching, a provider selling cheap tokens is running a large batch, and a low-latency endpoint is running a small one. There is no configuration that is both, and a single user running a model on their own GPU gets the worst case in the table: batch size 1, a token rate fixed by bandwidth, and a chip that is idle almost all the time. Every "why is my local model so slow when the GPU is barely at 5% utilisation" question has this answer.
Adding GPUs does not straightforwardly help one request. Splitting a model across chips with tensor parallelism does add their bandwidths together, but it also adds a network round-trip at every layer — and interconnect bandwidth has been growing at 1.4× every two years, slower than DRAM. The wall you climb over shows up again one level out.
Future Trends
The hardware response is to keep pushing bandwidth: HBM3e and HBM4, wider interfaces, more stacks, memory placed physically closer to the compute. It helps, but it is fighting a 3.0× versus 1.6× trend, and the historical record says the gap widens even as the absolute numbers improve.
The algorithmic response is more interesting, because it accepts the wall and works with it. Every technique currently winning in inference is an arithmetic-intensity play. Speculative decoding has a small model propose several tokens and the big model verify all of them in one forward pass — the same weight read now does k tokens' worth of maths, turning spare FLOPs into speed. MoE routes each token to a fraction of the parameters. Disaggregated serving puts the compute-bound prefill and the memory-bound decode on separate machines so each can be provisioned for the limit it actually hits.
The through-line is that model architecture is now being reshaped by a property of memory buses. When a lab ships a model that is huge in total parameters but small in active ones, or a serving stack that will do anything to grow the batch, you are watching a design negotiate with the memory wall.