---
source: 'https://howaiworks.ai/glossary/high-bandwidth-memory'
section: glossary
title: High Bandwidth Memory (HBM)
description: >-
  Stacked DRAM bonded beside the GPU die — the scarce component that gates AI
  accelerator supply, and why bandwidth rather than capacity is the constraint.
tags:
  - HBM
  - AI Hardware
  - GPU Computing
  - AI Infrastructure
  - Inference
  - NVIDIA
  - Performance Optimization
category: Infrastructure & Hardware
datePublished: '2026-07-13'
lastUpdated: '2026-07-24'
---

# High Bandwidth Memory (HBM)

> Stacked DRAM bonded beside the GPU die — the scarce component that gates AI accelerator supply, and why bandwidth rather than capacity is the constraint.

## Definition

**The memory on an AI accelerator is not on the motherboard. It is inside the chip's package, a few
millimetres from the processor, as a tower of DRAM dies drilled through with thousands of vertical
wires.** High Bandwidth Memory is that tower, and it exists because of a fact about physics rather
than a fact about chips: past a certain point you cannot make a wire carry bits faster, so you have
to use more wires.

The trade is stark enough to see in one comparison. A GDDR7 chip — the kind on a gaming graphics card
— signals at up to **32 Gb/s per pin** across a 32-bit interface, giving **128 GB/s per device**
([Rambus](https://www.rambus.com/blogs/all-you-need-to-know-about-gddr7/)). An HBM3E stack runs each
of its pins at roughly a *quarter* of that speed and delivers over **1.2 TB/s**
([Micron](https://www.micron.com/products/memory/hbm/hbm3e)). Slower wires, ten times the bandwidth,
because there are 1,024 of them instead of 32. That is the whole idea — and it is why HBM, not the
processor die, is the component that gates how many AI accelerators the world can build.

## How It Works

### Bandwidth is width times rate, and only one of those is cheap

Every memory interface obeys the same equation: bandwidth equals the number of data pins multiplied
by the bits per second each pin carries. There are therefore only two ways to get more of it, and
the industry has spent thirty years pushing on the second one. DDR5, LPDDR5 and GDDR "all take the
approach of ramping up the data rate, which causes signal integrity issues as the time for each bit
to transfer across the bus shrinks"
([Synopsys](https://www.synopsys.com/blogs/chip-design/designing-hbm3-high-bandwidth-memory-interfaces.html)).

That sentence hides a hard physical limit, and it is worth making concrete. A signal on a standard
FR-4 circuit board travels at roughly **6 inches per nanosecond**, about 15 cm/ns
([AllPCB](https://www.allpcb.com/blog/pcb-knowledge/high-speed-fr-4-pcb-design-techniques-for-signal-integrity.html)).
At GDDR7's 32 Gb/s, one bit occupies `1 ÷ 32 × 10⁹ = 31 picoseconds`, and in 31 ps a signal covers
`15 cm/ns × 0.031 ns ≈ 4.7 mm` of board. So a 10-centimetre trace from a memory chip to a GPU holds
about **twenty bits in flight simultaneously** — twenty bits attenuating, reflecting off every
impedance discontinuity, and interfering with each other. Every increment of speed makes that worse.
This is not a problem anyone is going to solve; it is what a transmission line is. HBM sidesteps it
by making the wire short instead of making the bit fast.

### The stack, the vias and the interposer

Three pieces of construction do the work:

- **The stack.** Four, eight, twelve or sixteen DRAM dies are thinned and bonded on top of each
  other into a single cube. JEDEC's HBM4 standard, published in April 2025, covers 4-, 8-, 12- and
  16-high configurations of 24 Gb or 32 Gb dies, giving up to **64 GB in one stack**
  ([JEDEC](https://www.jedec.org/news/pressreleases/jedec%C2%AE-and-industry-leaders-collaborate-release-jesd270-4-hbm4-standard-advancing)).
- **Through-silicon vias.** The dies are connected vertically by copper wires driven straight
  through the silicon rather than routed around its edge. A stack contains thousands of them, and
  they are what makes a 1,024-bit interface physically expressible in a component the size of a
  fingernail.
- **The interposer.** The finished stack does not go on the motherboard. It is bonded onto a slab of
  silicon — "effectively a miniature PCB that goes inside the package" (Synopsys) — sitting beside
  the processor die. The connection is now millimetres of silicon rather than centimetres of board,
  and the signals are short and unterminated.

Now the wide bus becomes affordable. JEDEC's HBM3 gives each stack **1,024 data bits**, organised as
16 channels of 64 bits (32 pseudo-channels), at 6.4 Gb/s per pin
([Synopsys](https://www.synopsys.com/glossary/what-is-high-bandwitdth-memory-3.html)) — which
multiplies out to `1,024 × 6.4 × 10⁹ ÷ 8 = 819 GB/s` from a single cube. HBM4 doubles the interface
to **2,048 bits** and sets a JEDEC baseline pin rate of 8 Gb/s, for up to **2 TB/s per stack**
(JEDEC, above) — a floor the first shipping parts already clear: Samsung's HBM4, in mass production
since February 2026, runs at **11.7 Gb/s** for up to **3.3 TB/s per stack**
([Samsung](https://news.samsung.com/global/samsung-ships-industry-first-commercial-hbm4-with-ultimate-performance-for-ai-computing)).
Note what the industry did when it needed more bandwidth: it doubled the wires. It always doubles the
wires.

### Worked example: where an H200's 4.8 TB/s actually comes from

NVIDIA's H200 SXM lists **141 GB of HBM3e** and **4.8 TB/s** of memory bandwidth
([NVIDIA](https://www.nvidia.com/en-us/data-center/h200/)). Micron, which supplies it, sells the part
as a **24 GB 8-high HBM3E** cube and states it is "shipping with NVIDIA H200 Tensor Core GPUs"
([Micron](https://www.micron.com/products/memory/hbm/hbm3e)). So:

1. Stacks on the package: `141 GB ÷ 24 GB ≈ 6` (six cubes, 144 GB built, 141 GB exposed).
2. Bandwidth per stack: `4,800 GB/s ÷ 6 = 800 GB/s`.
3. A stack is 1,024 bits = **128 bytes** wide, so the per-pin rate is `800 ÷ 128 = 6.25 Gb/s`.

Six thousand one hundred and forty-four data wires, each dawdling along at 6.25 Gb/s — *below* what
JEDEC allows and roughly a fifth of a GDDR7 pin's speed — and that is where 4.8 terabytes per second
comes from. To reach the same figure with GDDR7 at 128 GB/s per device you would need **37 chips and
about 1,200 data pins routed across a circuit board** at 32 Gb/s each. Nobody can build that board.
The stack is not an optimisation; it is the only construction that gets there.

### What the bandwidth buys, in tokens

Generating one token requires reading every weight in the model out of memory, so the token rate is
bounded by `bandwidth ÷ weight bytes`. [Memory Wall](https://howaiworks.ai/glossary/memory-wall) derives that ceiling and
explains why decoding is bandwidth-bound rather than compute-bound; this page simply spends it.
Llama 3.1 70B in FP8 is about **70.6 GB** of weights, so:

- On an H100 SXM, at NVIDIA's stated **3.35 TB/s** ([NVIDIA](https://www.nvidia.com/en-us/data-center/h100/)):
  `3,350 ÷ 70.6 ≈ 47 tokens/second`.
- On an H200, at 4.8 TB/s: `4,800 ÷ 70.6 ≈ 68 tokens/second`.

Hold on to that pair, because it is the same processor. Both parts are rated at **3,958 FP8 TFLOPS
with sparsity** (NVIDIA, both pages). The arithmetic capability is identical. The H200 generates
about **43% more tokens per second** because someone bolted faster memory to it — and for a
memory-bound workload, that is the only number that moved.

The H100 and H200 anchor this example because their specifications are public and stable, not because
they are the frontier — by mid-2026 they sit two generations back. NVIDIA's Blackwell B200 pushed
HBM3e to roughly **8 TB/s** per accelerator, and AMD's MI325X reaches **256 GB of HBM3e**
([Tom's Hardware](https://www.tomshardware.com/tech-industry/artificial-intelligence/amds-instinct-mi325x-smiles-for-the-camera-256-gb-of-hbm3e)),
while the first **HBM4** parts entered mass production in early 2026 (see Future Trends). Every one of
those figures will be superseded; the arithmetic that turns bandwidth into tokens will not. Read this
section for the mechanism, then drop in whatever the current part sheet says.

## Real-World Applications

**Choosing an accelerator.** The two figures on the spec sheet answer different questions. Capacity
decides *what fits*: the weights, plus a [KV cache](https://howaiworks.ai/glossary/kv-cache) that grows with context and
batch size and competes with them for the same gigabytes. Bandwidth decides *how fast it runs*.
Buying for capacity and then being surprised by the token rate is the single most common hardware
mistake in this field, and the arithmetic above is the whole explanation.

**Why the supply is inelastic.** HBM is short in a way that ordinary components are not, and the
reasons are structural rather than commercial:

- **It cannot be added later.** HBM must be co-packaged with the logic die on a silicon interposer,
  so an accelerator's output is capped by *advanced packaging* lines as well as by memory fabs — a
  step that has been the binding constraint on AI GPU shipments, not wafer starts. You cannot ship
  the GPU and add memory in the field.
- **It is not a commodity part.** Each supplier's stack has to pass the accelerator vendor's
  qualification, and that has taken years, not months: Samsung's 12-layer HBM3E was approved by
  NVIDIA in September 2025, "about 18 months after Samsung completed development of the chip", after
  a string of failed attempts on thermal performance
  ([KED Global](https://www.kedglobal.com/korean-chipmakers/newsView/ked202509190008)). As of June
  2026, exactly three firms — SK hynix, Samsung and Micron — are certified to supply HBM4 for
  NVIDIA's Vera Rubin platform
  ([Reuters](https://finance.yahoo.com/sectors/technology/articles/nvidia-certifies-samsung-sk-hynix-133001560.html)).
  A fourth entrant cannot simply start selling.
- **Making more of it destroys ordinary memory.** An HBM gigabyte consumes roughly **three times the
  wafer capacity** of a DDR5 gigabyte, because stacking loses yield and the vias eat die area
  ([Tom's Hardware](https://www.tomshardware.com/pc-components/ram/hbm-is-eating-your-ram)). Every
  wafer converted to HBM therefore removes about three times as much conventional DRAM from the
  market — which is why an AI build-out shows up as a rise in the price of laptop RAM.

**Every inference optimisation is an HBM optimisation.** Quantization, batching, Mixture-of-Experts
routing and KV-cache compression all exist to move fewer bytes across this bus or to extract more
arithmetic from the bytes already moved. [Memory Wall](https://howaiworks.ai/glossary/memory-wall) owns the principle;
[Quantization](https://howaiworks.ai/glossary/quantization) owns the number formats. HBM is the physical object all of
them are negotiating with.

## Key Concepts

- **Width beats speed inside a package, and only inside a package.** The reason HBM can run a
  1,024-bit bus is that its wires are millimetres long. The reason your motherboard cannot is that
  its wires are not. Any future memory technology that wants HBM-class bandwidth will have to be
  co-packaged too, and will inherit HBM's packaging bottleneck along with its bandwidth.
- **Capacity and bandwidth are sold together and mean different things.** They rise together across
  generations, which lets people confuse them for years without being corrected.
- **The pins are deliberately slow.** The H200 runs HBM3E below its rated pin speed. Six thousand
  wires switching under a 700-watt processor is a thermal budget, and heat is the constraint that
  keeps the rate down — Samsung's qualification failures were thermal, not logical.

## Challenges

**What breaks: buying an accelerator on its FLOPS number.** A chip's headline TFLOPS figure does not
appear anywhere in the token-rate calculation above. The H100 and H200 have identical peak FLOPS and
differ by 43% in decode throughput; conversely, two chips can post similar TFLOPS and differ by a
factor of two in tokens per second because one has a generation-older memory stack. If your workload
is memory-bound — and single-stream LLM decoding always is — the FLOPS column on the comparison table
is close to decorative. Sort by bandwidth per dollar instead, and check what the bandwidth costs in
watts.

**Thermal stacking.** A tower of sixteen dies thinned to tens of microns sits millimetres from a
processor dissipating hundreds of watts, and DRAM leaks and refreshes more as it heats. This is why
HBM generations slip, why suppliers fail qualification, and why stacks cannot simply keep growing
taller.

**It is one component, and it scraps as one.** The stack is bonded to the same interposer as the GPU,
so a defect in a single DRAM die can write off a package containing a full accelerator die — a
compounding yield problem, not an additive one. It also means there is no upgrade path: memory and
compute are bought, deployed and retired as one unit, and a fleet's bandwidth is fixed on the day it
is purchased.

## Future Trends

The industry's answer to needing more bandwidth has been the same answer twice, and HBM4 makes it
three times: **double the wires**. HBM4 is no longer a roadmap item. SK hynix completed development
and readied the world's first HBM4 mass production in September 2025
([SK hynix](https://news.skhynix.com/sk-hynix-completes-worlds-first-hbm4-development-and-readies-mass-production/)),
and Samsung began commercial shipment in February 2026
([Samsung](https://news.samsung.com/global/samsung-ships-industry-first-commercial-hbm4-with-ultimate-performance-for-ai-computing)),
bound for NVIDIA's Rubin and AMD's next Instinct parts. Its 2,048-bit interface arrived with a per-pin
rate the standard set only modestly above HBM3E's — telling you where the engineers believe the
headroom is not, even as the first shipping stacks push the pin well past that 8 Gb/s floor.

The more interesting change is what sits underneath the stack. HBM4's base die — the layer that talks
to the processor — has moved to a logic process rather than a DRAM one (Samsung builds its shipping
HBM4 base die on a 4 nm logic node), so it can hold real circuitry, and vendors are customising it per
customer. That quietly dissolves the boundary between "the memory" and "the chip": if the base die can
compute, some operations stop needing to cross the bus at all.

The constraint to watch, though, is not bandwidth per stack but stacks per package. Each one needs
interposer area and packaging throughput, and both are finite. The wire count has been doubling; the
number of places to put the wires has not.

## Frequently Asked Questions

### What is High Bandwidth Memory in simple terms?

It is DRAM built as a vertical tower of chips, drilled through with thousands of tiny wires and mounted a few millimetres from the processor inside the same package. Ordinary memory talks to the chip over a 64-bit bus on the motherboard; one HBM stack talks over a 1,024-bit bus that never leaves the package.

### Why is HBM faster than DDR5 or GDDR7 if its pins run slower?

Because bandwidth is pins multiplied by per-pin speed, and HBM buys bandwidth with pins. A GDDR7 device signals at up to 32 Gb/s per pin but is only 32 bits wide, giving 128 GB/s. An HBM3E stack runs each pin at roughly a quarter of that rate but is 1,024 bits wide, so it delivers over 1.2 TB/s — about ten times more from wires that are individually much slower.

### Why can't memory just be made faster instead of wider?

Signal integrity. At 32 Gb/s a single bit lasts about 31 picoseconds, and in that time a signal travels only around 5 mm of circuit board — so a 10 cm trace has roughly twenty bits in flight at once, all attenuating and reflecting. Inside the package the wire is short enough that the problem disappears, which is what lets HBM run 1,024 of them at once.

### Why is HBM in short supply?

It is not a part you can bolt on later: it must be bonded onto a silicon interposer next to the logic die, so accelerator output is capped by advanced packaging lines as well as by memory fabs. It is also not a commodity — each supplier must pass the accelerator vendor's qualification, which has taken years, and only three firms are qualified to supply it.

### Does more HBM capacity make a model run faster?

No. Capacity decides what fits; bandwidth decides how fast it runs. A larger memory lets you load a bigger model or hold a longer KV cache, but the token rate is set by how many bytes per second the accelerator can read, not by how many bytes it can hold.

### Is HBM only used in NVIDIA GPUs?

No. Every serious AI accelerator uses it — AMD's Instinct line, Google's TPUs, and most startup silicon — because they all face the same constraint. HBM is what an accelerator uses to get memory bandwidth, and there is currently no alternative that delivers it.

## Related

### Related terms

- [Memory Wall](https://howaiworks.ai/glossary/memory-wall)
- [Advanced Packaging](https://howaiworks.ai/glossary/advanced-packaging)
- [GPU Computing](https://howaiworks.ai/glossary/gpu-computing)
- [NVIDIA GPU for AI](https://howaiworks.ai/glossary/nvidia-gpu-ai)
- [Semiconductor Manufacturing](https://howaiworks.ai/glossary/semiconductor-manufacturing)
- [Inference](https://howaiworks.ai/glossary/inference)

---

Source: https://howaiworks.ai/glossary/high-bandwidth-memory — HowAIWorks.ai