Definition
The memory on an AI accelerator is not on the motherboard. It is inside the chip's package, a few millimetres from the processor, as a tower of DRAM dies drilled through with thousands of vertical wires. High Bandwidth Memory is that tower, and it exists because of a fact about physics rather than a fact about chips: past a certain point you cannot make a wire carry bits faster, so you have to use more wires.
The trade is stark enough to see in one comparison. A GDDR7 chip — the kind on a gaming graphics card — signals at up to 32 Gb/s per pin across a 32-bit interface, giving 128 GB/s per device (Rambus). An HBM3E stack runs each of its pins at roughly a quarter of that speed and delivers over 1.2 TB/s (Micron). Slower wires, ten times the bandwidth, because there are 1,024 of them instead of 32. That is the whole idea — and it is why HBM, not the processor die, is the component that gates how many AI accelerators the world can build.
How It Works
Bandwidth is width times rate, and only one of those is cheap
Every memory interface obeys the same equation: bandwidth equals the number of data pins multiplied by the bits per second each pin carries. There are therefore only two ways to get more of it, and the industry has spent thirty years pushing on the second one. DDR5, LPDDR5 and GDDR "all take the approach of ramping up the data rate, which causes signal integrity issues as the time for each bit to transfer across the bus shrinks" (Synopsys).
That sentence hides a hard physical limit, and it is worth making concrete. A signal on a standard
FR-4 circuit board travels at roughly 6 inches per nanosecond, about 15 cm/ns
(AllPCB).
At GDDR7's 32 Gb/s, one bit occupies 1 ÷ 32 × 10⁹ = 31 picoseconds, and in 31 ps a signal covers
15 cm/ns × 0.031 ns ≈ 4.7 mm of board. So a 10-centimetre trace from a memory chip to a GPU holds
about twenty bits in flight simultaneously — twenty bits attenuating, reflecting off every
impedance discontinuity, and interfering with each other. Every increment of speed makes that worse.
This is not a problem anyone is going to solve; it is what a transmission line is. HBM sidesteps it
by making the wire short instead of making the bit fast.
The stack, the vias and the interposer
Three pieces of construction do the work:
- The stack. Four, eight, twelve or sixteen DRAM dies are thinned and bonded on top of each other into a single cube. JEDEC's HBM4 standard, published in April 2025, covers 4-, 8-, 12- and 16-high configurations of 24 Gb or 32 Gb dies, giving up to 64 GB in one stack (JEDEC).
- Through-silicon vias. The dies are connected vertically by copper wires driven straight through the silicon rather than routed around its edge. A stack contains thousands of them, and they are what makes a 1,024-bit interface physically expressible in a component the size of a fingernail.
- The interposer. The finished stack does not go on the motherboard. It is bonded onto a slab of silicon — "effectively a miniature PCB that goes inside the package" (Synopsys) — sitting beside the processor die. The connection is now millimetres of silicon rather than centimetres of board, and the signals are short and unterminated.
Now the wide bus becomes affordable. JEDEC's HBM3 gives each stack 1,024 data bits, organised as
16 channels of 64 bits (32 pseudo-channels), at 6.4 Gb/s per pin
(Synopsys) — which
multiplies out to 1,024 × 6.4 × 10⁹ ÷ 8 = 819 GB/s from a single cube. HBM4 doubles the interface
to 2,048 bits and raises the pin rate only modestly, to 8 Gb/s, for up to 2 TB/s per stack
(JEDEC, above). Note what the industry did when it needed more bandwidth: it doubled the wires. It
always doubles the wires.
Worked example: where an H200's 4.8 TB/s actually comes from
NVIDIA's H200 SXM lists 141 GB of HBM3e and 4.8 TB/s of memory bandwidth (NVIDIA). Micron, which supplies it, sells the part as a 24 GB 8-high HBM3E cube and states it is "shipping with NVIDIA H200 Tensor Core GPUs" (Micron). So:
- Stacks on the package:
141 GB ÷ 24 GB ≈ 6(six cubes, 144 GB built, 141 GB exposed). - Bandwidth per stack:
4,800 GB/s ÷ 6 = 800 GB/s. - A stack is 1,024 bits = 128 bytes wide, so the per-pin rate is
800 ÷ 128 = 6.25 Gb/s.
Six thousand one hundred and forty-four data wires, each dawdling along at 6.25 Gb/s — below what JEDEC allows and roughly a fifth of a GDDR7 pin's speed — and that is where 4.8 terabytes per second comes from. To reach the same figure with GDDR7 at 128 GB/s per device you would need 37 chips and about 1,200 data pins routed across a circuit board at 32 Gb/s each. Nobody can build that board. The stack is not an optimisation; it is the only construction that gets there.
What the bandwidth buys, in tokens
Generating one token requires reading every weight in the model out of memory, so the token rate is
bounded by bandwidth ÷ weight bytes. Memory Wall derives that ceiling and
explains why decoding is bandwidth-bound rather than compute-bound; this page simply spends it.
Llama 3.1 70B in FP8 is about 70.6 GB of weights, so:
- On an H100 SXM, at NVIDIA's stated 3.35 TB/s (NVIDIA):
3,350 ÷ 70.6 ≈ 47 tokens/second. - On an H200, at 4.8 TB/s:
4,800 ÷ 70.6 ≈ 68 tokens/second.
Hold on to that pair, because it is the same processor. Both parts are rated at 3,958 FP8 TFLOPS with sparsity (NVIDIA, both pages). The arithmetic capability is identical. The H200 generates about 43% more tokens per second because someone bolted faster memory to it — and for a memory-bound workload, that is the only number that moved.
Real-World Applications
Choosing an accelerator. The two figures on the spec sheet answer different questions. Capacity decides what fits: the weights, plus a KV cache that grows with context and batch size and competes with them for the same gigabytes. Bandwidth decides how fast it runs. Buying for capacity and then being surprised by the token rate is the single most common hardware mistake in this field, and the arithmetic above is the whole explanation.
Why the supply is inelastic. HBM is short in a way that ordinary components are not, and the reasons are structural rather than commercial:
- It cannot be added later. HBM must be co-packaged with the logic die on a silicon interposer, so an accelerator's output is capped by advanced packaging lines as well as by memory fabs — a step that has been the binding constraint on AI GPU shipments, not wafer starts. You cannot ship the GPU and add memory in the field.
- It is not a commodity part. Each supplier's stack has to pass the accelerator vendor's qualification, and that has taken years, not months: Samsung's 12-layer HBM3E was approved by NVIDIA in September 2025, "about 18 months after Samsung completed development of the chip", after a string of failed attempts on thermal performance (KED Global). As of June 2026, exactly three firms — SK hynix, Samsung and Micron — are certified to supply HBM4 for NVIDIA's Vera Rubin platform (Reuters). A fourth entrant cannot simply start selling.
- Making more of it destroys ordinary memory. An HBM gigabyte consumes roughly three times the wafer capacity of a DDR5 gigabyte, because stacking loses yield and the vias eat die area (Tom's Hardware). Every wafer converted to HBM therefore removes about three times as much conventional DRAM from the market — which is why an AI build-out shows up as a rise in the price of laptop RAM.
Every inference optimisation is an HBM optimisation. Quantization, batching, Mixture-of-Experts routing and KV-cache compression all exist to move fewer bytes across this bus or to extract more arithmetic from the bytes already moved. Memory Wall owns the principle; Quantization owns the number formats. HBM is the physical object all of them are negotiating with.
Key Concepts
- Width beats speed inside a package, and only inside a package. The reason HBM can run a 1,024-bit bus is that its wires are millimetres long. The reason your motherboard cannot is that its wires are not. Any future memory technology that wants HBM-class bandwidth will have to be co-packaged too, and will inherit HBM's packaging bottleneck along with its bandwidth.
- Capacity and bandwidth are sold together and mean different things. They rise together across generations, which lets people confuse them for years without being corrected.
- The pins are deliberately slow. The H200 runs HBM3E below its rated pin speed. Six thousand wires switching under a 700-watt processor is a thermal budget, and heat is the constraint that keeps the rate down — Samsung's qualification failures were thermal, not logical.
Challenges
What breaks: buying an accelerator on its FLOPS number. A chip's headline TFLOPS figure does not appear anywhere in the token-rate calculation above. The H100 and H200 have identical peak FLOPS and differ by 43% in decode throughput; conversely, two chips can post similar TFLOPS and differ by a factor of two in tokens per second because one has a generation-older memory stack. If your workload is memory-bound — and single-stream LLM decoding always is — the FLOPS column on the comparison table is close to decorative. Sort by bandwidth per dollar instead, and check what the bandwidth costs in watts.
Thermal stacking. A tower of sixteen dies thinned to tens of microns sits millimetres from a processor dissipating hundreds of watts, and DRAM leaks and refreshes more as it heats. This is why HBM generations slip, why suppliers fail qualification, and why stacks cannot simply keep growing taller.
It is one component, and it scraps as one. The stack is bonded to the same interposer as the GPU, so a defect in a single DRAM die can write off a package containing a full accelerator die — a compounding yield problem, not an additive one. It also means there is no upgrade path: memory and compute are bought, deployed and retired as one unit, and a fleet's bandwidth is fixed on the day it is purchased.
Future Trends
The industry's answer to needing more bandwidth has been the same answer twice, and HBM4 makes it three times: double the wires. The 2,048-bit interface arrives with a per-pin rate barely above HBM3E's, which tells you where the engineers believe the headroom is not.
The more interesting change is what sits underneath the stack. HBM4's base die — the layer that talks to the processor — is moving to a logic process rather than a DRAM one, so it can hold real circuitry, and vendors are customising it per customer. That quietly dissolves the boundary between "the memory" and "the chip": if the base die can compute, some operations stop needing to cross the bus at all.
The constraint to watch, though, is not bandwidth per stack but stacks per package. Each one needs interposer area and packaging throughput, and both are finite. The wire count has been doubling; the number of places to put the wires has not.