---
source: 'https://howaiworks.ai/glossary/memory-wall'
section: glossary
title: Memory Wall
description: >-
  Why LLM inference waits on memory rather than maths — arithmetic intensity,
  the roofline, and the one fact that explains HBM scarcity and the price of a
  token.
tags:
  - Inference
  - GPU Computing
  - AI Infrastructure
  - AI Hardware
  - LLM
  - Performance Optimization
category: Infrastructure & Hardware
datePublished: '2026-07-13'
lastUpdated: '2026-07-24'
---

# Memory Wall

> Why LLM inference waits on memory rather than maths — arithmetic intensity, the roofline, and the one fact that explains HBM scarcity and the price of a token.

## Definition

**A modern GPU can multiply numbers far faster than it can read them, and that gap — not the speed
of the arithmetic — is what limits most AI systems today.** The memory wall is the widening gap
between processor throughput and memory bandwidth. It has a sharp consequence for language models:
when a chatbot is streaming words at you, the accelerator producing them is running its arithmetic
units at well under **1%** of their rated capacity. It is not computing. It is waiting for the
model's weights to arrive from memory, over and over, once per word.

The term comes from a 1995 note by William Wulf and Sally McKee, [Hitting the Memory
Wall](https://dl.acm.org/doi/10.1145/216585.216588), which pointed out something merely arithmetical:
if processor speed and memory speed both improve exponentially but at different rates, the gap
between them also grows exponentially, and eventually every program is a memory program. Thirty
years on, that is where large-model [inference](https://howaiworks.ai/glossary/inference) sits.

## How It Works

### Arithmetic intensity: how much maths you do per byte you move

Every computation has a ratio you can measure: floating-point operations performed, divided by bytes
moved between memory and the chip. This is its **arithmetic intensity**, in FLOPs per byte.
Multiplying two large matrices has high intensity — each element loaded is reused across many rows
and columns, so a lot of maths happens per byte fetched. Adding two vectors has an intensity near
zero: read two numbers, do one addition, write one number, repeat.

Intensity is a property of the *job*, not the hardware, and it decides which of the machine's two
speed limits you hit.

### The roofline: every chip has a break-even intensity

The [roofline model](https://dl.acm.org/doi/abs/10.1145/1498765.1498785) (Williams, Waterman and
Patterson, *CACM* 2009) makes this concrete. A chip has a peak arithmetic rate and a peak memory
bandwidth, and dividing one by the other gives its **break-even intensity** — the FLOPs per byte a
workload must supply to keep the arithmetic units fed.

The number that matters is the *ratio*, not any one chip's spec sheet, so pick a well-documented
chip and date it. Take NVIDIA's H100 SXM, whose [published specification](https://www.nvidia.com/en-us/data-center/h100/)
as of 2026 is **3.35 TB/s** of memory bandwidth and **3,958 teraFLOPS** of FP8 tensor-core
throughput — "with sparsity", meaning the dense figure you can actually count on is half that,
**1,979 TFLOPS**. Its break-even intensity is therefore:

`1,979 × 10¹² FLOP/s ÷ 3.35 × 10¹² bytes/s ≈ 590 FLOPs per byte`

That **~590 FLOPs per byte** is the durable statement; the two absolute numbers it divides are
the perishable inputs, and every accelerator has its own break-even in the same
low-hundreds-to-thousand range (the current 2026 landscape — H200, Blackwell B200, AMD Instinct
MI300X — appears in Real-World Applications below). A workload that does more than its chip's
break-even FLOPs for every byte it reads is **compute-bound**: the tensor
cores are the limit, and faster memory would buy you nothing. A workload below that line is
**memory-bound**: it is limited by bandwidth, and a chip with twice the FLOPs would run it at
exactly the same speed. Note which way that threshold has been moving. Over the past twenty years,
peak server FLOPS grew **3.0× every two years** while DRAM bandwidth grew only **1.6×**, according
to [AI and Memory Wall](https://arxiv.org/abs/2403.14123) (Gholami et al., *IEEE Micro*, 2024).
Compound those for a decade and compute rises 3⁵ ≈ **243×** against bandwidth's 1.6⁵ ≈ **10.5×** —
the break-even intensity climbs roughly **23× every ten years**. The bar for being compute-bound
keeps rising, so more and more real work falls below it.

### Generating one token is 2 FLOPs per byte

Here is the fact this whole page exists for. An autoregressive model produces text one token at a
time, and each token is a full forward pass through the network. That forward pass touches **every
weight in the model exactly once** — so every weight has to be read out of memory. The arithmetic it
does with them is roughly **two FLOPs per parameter** (one multiply and one add in the
[matrix multiplications](https://howaiworks.ai/glossary/matrix-multiplication); the [FLOPs](https://howaiworks.ai/glossary/flops) page owns
that estimate).

Two FLOPs per parameter. One byte per parameter, if the model is stored in 8-bit precision. The
arithmetic intensity of generating a token for a single user is therefore **2 FLOPs per byte** — a
number set by the algorithm, not the hardware, so it does not move when the chip does. Against the
H100's break-even of 590 the workload is not slightly memory-bound; it is nearly **300× below** the
line. On a higher-FLOP chip the gap is *wider*, not narrower: more arithmetic capacity over similar
bandwidth pushes the break-even up while decode stays pinned at 2, so no future accelerator rescues
a single-stream decode by being faster at maths.

### The worked example: a token-rate ceiling you can compute in one line

Llama 3.1 70B has **70.6 billion parameters** ([model
card](https://huggingface.co/meta-llama/Llama-3.1-70B)). Serve it in FP8, one byte per weight:

1. Weights to read, per token: `70.6 × 10⁹ params × 1 byte = 70.6 GB`
2. Time to read them at the H100's 2026 spec bandwidth: `70.6 GB ÷ 3,350 GB/s = 21.1 ms`
3. Ceiling on generation speed: `1 ÷ 0.0211 s = 47 tokens per second`

That is a **hard ceiling for a single request**, from two numbers: the size of the model and the
bandwidth of the memory. Nothing about the GPU's arithmetic capability appears in it. A vendor
cannot beat it with a better kernel, and a real system will land under it once you count the
[KV cache](https://howaiworks.ai/glossary/kv-cache), attention, and communication. Swap in a faster memory and the
ceiling moves with the bandwidth and nothing else: the same weights on an H200's 4.8 TB/s clear
about **68 tok/s**, and on a Blackwell B200's 8 TB/s about **113** — the arithmetic is one
division either way, and the FLOP rating of the chip never enters it.

Now look at what the tensor cores did during those 21.1 ms. The maths per token is
`2 × 70.6 × 10⁹ = 141 GFLOP`. At 47 tokens per second that is `141 GFLOP × 47 ≈ 6.7 TFLOP/s` —
against 1,979 TFLOPS available. The H100 is running at **0.34% of its arithmetic capacity**. Put
the other way: the chip spends **0.07 ms** doing the multiplications and **21.1 ms** waiting for the
numbers to multiply. Roughly 99.7% of the silicon you are paying for is idle while the model
"generates fast".

### Prefill sits on the other side of the ridge

Processing the *prompt* is a different workload with the same weights. A 10,000-token prompt is
processed in one pass, so each weight that gets read is used for 10,000 tokens' worth of arithmetic:
the intensity is 2 × 10,000 = 20,000 FLOPs per byte, well past the break-even, and the GPU is
genuinely compute-bound. This is why the first token takes a while and the rest arrive at a steady
drip — the two halves of a request live on opposite sides of the roofline, and they want different
optimisations and, increasingly, different machines.

## Real-World Applications

**Batching is the entire economics of an inference server.** If the GPU reads the 70.6 GB of weights
once and uses them for 64 requests at the same time, the bytes read stay the same while the
arithmetic done with them goes up 64-fold: the intensity becomes 2 × 64 = 128 FLOPs per byte, and
the server produces roughly 64× the tokens per second in aggregate at almost no cost in latency per
user. That is why inference servers exist instead of everyone running a model locally, and why
systems like vLLM work so hard to keep the batch full. Extending the arithmetic: you would need
about **295** concurrent requests to reach the H100's break-even intensity of 590, which is why a
serving fleet is never compute-bound — only less catastrophically memory-bound.

**Quantization is a bandwidth optimisation before it is a capacity one.** Halving the bytes per
weight halves step 1 of the calculation above and therefore roughly halves the time to produce a
token. Dropping our 70.6B model from FP8 to 4-bit takes the weights from 70.6 GB to 35.3 GB and the
ceiling from 47 to about **95 tokens per second** on the same GPU, with nothing about the chip
changed. See [quantization](https://howaiworks.ai/glossary/quantization) for what the formats do and what accuracy costs.

**Mixture-of-Experts is a way to read fewer bytes per token.** DeepSeek-V3 has **671B total
parameters but activates only 37B for each token** ([technical
report](https://arxiv.org/abs/2412.19437)) — so a decode step reads about 37 GB instead of 671 GB,
an **18× cut** in the only quantity that sets the token rate. Enormous in memory, cheap in
bandwidth: exactly the trade the memory wall rewards. See
[Mixture-of-Experts](https://howaiworks.ai/glossary/mixture-of-experts).

**The KV cache is the other thing being read.** Weights are not the only bytes crossing the bus each
step: attention re-reads the cached keys and values for every previous token in the conversation.
That cache grows with the [context window](https://howaiworks.ai/glossary/context-window) and with the batch, and it
competes for the same bandwidth — which is why long conversations slow down token-by-token even
though the weights have not changed. The [KV cache](https://howaiworks.ai/glossary/kv-cache) page does the size
arithmetic.

**HBM scarcity is this page in commercial form.** An accelerator carries tens of gigabytes of memory
not because that is a lot — a two-socket server can hold terabytes of ordinary DRAM — but because
that memory is High Bandwidth Memory, and bandwidth is the scarce thing. Read the current
data-center parts as a single sentence spoken in different accents, all figures as of 2026: NVIDIA's
H100 SXM pairs 80 GB of HBM3 with **3.35 TB/s**
([NVIDIA](https://www.nvidia.com/en-us/data-center/h100/)); its H200 keeps the same Hopper compute
but lifts memory to 141 GB of HBM3e at **4.8 TB/s** — a pure bandwidth upgrade, which tells you what
the vendor thinks the bottleneck is; the Blackwell **B200** reaches 192 GB of HBM3e at **8 TB/s**;
and AMD is not a spectator — its **Instinct MI300X** carries 192 GB of HBM3 at **5.3 TB/s**
([AMD](https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html)). Every one of them
spends its most expensive transistors on HBM for the same reason, and the world cannot make enough:
SK hynix said in October 2025 that its DRAM, NAND **and HBM capacity for the whole of the following
year was already sold out**
([TechSpot](https://www.techspot.com/news/110058-sk-hynix-completely-sells-out-semiconductor-supply-ai.html)).
The queue for HBM is the memory wall with a price tag on it.

## Key Concepts

- **Break-even intensity**: a job is memory-bound when it does fewer FLOPs per byte than the
  machine's ratio of peak arithmetic to peak bandwidth. Decoding one token at 2 FLOPs/byte falls
  under that line on every accelerator ever built, and no future chip rescues it by adding FLOPs.
- **The bytes are the thing you optimise**: every effective inference optimisation — batching,
  quantization, MoE routing, speculative decoding, KV-cache compression — either moves fewer bytes
  or extracts more arithmetic from the bytes it already moved. There is no third option.
- **Peak FLOPS is a marketing number for decoding**: it does not appear anywhere in the 47
  tokens-per-second calculation. Memory bandwidth and model size do.

## Challenges

**What breaks if you get this wrong is your capacity plan.** Size a fleet by dividing your expected
token volume by an accelerator's advertised TFLOPS and you will over-predict its throughput by two
or three orders of magnitude — the H100 above delivers 0.34% of its peak arithmetic during decode.
The same mistake at purchase time buys the chip with the biggest FLOPs number instead of the one
with the most bandwidth, and the two rankings are not the same list.

**It sends optimisation effort to the wrong place.** Hand-tuning a [CUDA](https://howaiworks.ai/glossary/cuda) matmul
kernel to shave 20% off the arithmetic of a decode step, when the arithmetic units are 99.7% idle,
buys you nothing at all. The wins on a memory-bound path come from moving fewer bytes or from fusing
operations so that intermediate values never leave the chip.

**Latency and cost per token are the same dial.** Because throughput comes from batching, a provider
selling cheap tokens is running a large batch, and a low-latency endpoint is running a small one.
There is no configuration that is both, and a single user running a model on their own GPU gets the
worst case in the table: batch size 1, a token rate fixed by bandwidth, and a chip that is idle
almost all the time. Every "why is my local model so slow when the GPU is barely at 5% utilisation"
question has this answer.

**Adding GPUs does not straightforwardly help one request.** Splitting a model across chips with
tensor [parallelism](https://howaiworks.ai/glossary/parallel-processing) does add their bandwidths together, but it also
adds a network round-trip at every layer — and interconnect bandwidth has been growing at 1.4× every
two years, *slower* than DRAM. The wall you climb over shows up again one level out.

## Future Trends

The hardware response is to keep pushing bandwidth: HBM3e and HBM4, wider interfaces, more stacks,
memory placed physically closer to the compute. It helps, but it is fighting a 3.0× versus 1.6×
trend, and the historical record says the gap widens even as the absolute numbers improve.

The algorithmic response is more interesting, because it accepts the wall and works with it. Every
technique currently winning in inference is an arithmetic-intensity play. **Speculative decoding**
has a small model propose several tokens and the big model verify all of them in one forward pass —
the same weight read now does *k* tokens' worth of maths, turning spare FLOPs into speed. **MoE**
routes each token to a fraction of the parameters. **Disaggregated serving** puts the compute-bound
prefill and the memory-bound decode on separate machines so each can be provisioned for the limit it
actually hits.

The through-line is that model architecture is now being reshaped by a property of memory buses. When
a lab ships a model that is huge in total parameters but small in active ones, or a serving stack
that will do anything to grow the batch, you are watching a design negotiate with the memory wall.

*Note: this page was last reviewed in July 2026. The durable content is the mechanism — arithmetic
intensity, the roofline, decode at ~2 FLOPs per byte, and the token-rate formula (bandwidth ÷ model
size). The perishable parts are the specific accelerator numbers (H100, H200, B200, MI300X capacities
and bandwidths) and the HBM-supply note; those are the lines most likely to age, and they are stated
as ratios or dated inputs so the argument survives when the chips change.*

## Frequently Asked Questions

### What is the memory wall in simple terms?

Processors can do arithmetic much faster than memory can supply the numbers to work on, and the gap has been widening for thirty years. A chip that is waiting for data is not computing, so for many workloads the speed of memory — not the speed of the maths — sets how fast the job runs.

### Why is generating one token from an LLM memory-bound?

To produce a single token the GPU has to read every weight in the model out of memory, and then does only about two arithmetic operations per byte it read. That is far below the several hundred FLOPs per byte a modern accelerator needs to keep its tensor cores busy — roughly 590 on an H100 as of 2026, and more on newer chips — so the chip spends nearly all its time waiting on memory.

### How do I calculate the maximum tokens per second for a model?

Divide the accelerator's memory bandwidth by the size of the model's weights in memory — the formula is the same for any chip. A 70-billion-parameter model in 8-bit precision is about 70 GB; on an H100 SXM with 3.35 TB/s of bandwidth (its 2026 spec) that is 3,350 / 70.6 = roughly 47 tokens per second for a single request, before any overhead. On a higher-bandwidth part the ceiling scales with the bandwidth. Real systems land below it, never above.

### Why does batching make LLM serving cheaper?

The weights get read from memory once and are then used for every request in the batch, so the same bytes do many times more arithmetic. Batching does not make one request faster — it raises the work done per byte moved, which is the only quantity that matters when a workload is memory-bound.

### Why is HBM in such short supply?

Because bandwidth, not capacity, is the scarce resource in AI inference, and HBM is how accelerators get bandwidth. Demand has outrun what the three suppliers can make: SK hynix said in October 2025 that its DRAM, NAND and HBM capacity for the following year was already sold out.

## Related

### Related terms

- [High Bandwidth Memory (HBM)](https://howaiworks.ai/glossary/high-bandwidth-memory)
- [Inference](https://howaiworks.ai/glossary/inference)
- [KV Cache](https://howaiworks.ai/glossary/kv-cache)
- [Quantization](https://howaiworks.ai/glossary/quantization)
- [GPU Computing](https://howaiworks.ai/glossary/gpu-computing)
- [Model Size](https://howaiworks.ai/glossary/model-size)

---

Source: https://howaiworks.ai/glossary/memory-wall — HowAIWorks.ai