AI Infrastructure
Why AI clusters are designed around memory bandwidth and interconnect rather than FLOPs, and what that changes when you size a training run or a serving fleet.
Definition
AI infrastructure is the hardware and software models are trained and served on — and it is not
designed around arithmetic. It is designed around moving bytes. Every accelerator has three
budgets, and whichever runs out first decides how fast your job goes.
Take NVIDIA’s H100 SXM as a worked example. By its
published specification it has 3.35 TB/s of
on-package memory bandwidth, 900 GB/s of NVLink to the GPUs in its own server, and — through a
typical 400 Gbps port — about 50 GB/s to a GPU in another rack: a ratio of roughly
67 : 18 : 1. A byte that stays in the accelerator’s own memory arrives sixty-seven times faster
than one crossing the data-center floor, and nearly every cluster design decision is an attempt to
keep bytes in the fastest tier they can be kept in.
Those absolute figures are a mid-2026 snapshot and will rot; the ordering of the tiers does not.
Newer accelerators push every number up without flattening the hierarchy — NVIDIA’s Blackwell
B200 carries 192 GB of HBM3e at roughly 8 TB/s
(NVIDIA), and AMD’s Instinct MI300X holds
192 GB at 5.3 TB/s, with the 2025 MI355X at 288 GB
(AMD) — but on-package
bandwidth still beats intra-server NVLink, which still beats the cross-rack fabric by a wide
multiple. It is the ratio, not any one chip’s spec sheet, that this page is about.
How It Works
Decoding a token is a memory problem, not a maths problem
Divide a chip’s peak arithmetic rate by its memory bandwidth and you get its break-even
intensity: the FLOPs it must perform per byte read to keep the tensor cores busy. For an H100 in
FP8 that is 1,979 × 10¹² FLOP/s ÷ 3.35 × 10¹² bytes/s ≈ 590 FLOPs per byte.
Generating one token for one user comes nowhere near it. The forward pass reads every weight once
and does about two floating-point operations per parameter, so its intensity is 2 FLOPs per byte
— nearly 300× under the line. Batching helps, because weights once read serve every request in the
batch: intensity becomes 2 × batch. At a batch of 64 that is 128 FLOPs per byte, about 22% of
peak arithmetic, and you would need roughly 295 concurrent requests merely to break even. The
memory wall page works this through in full.
Processing the prompt sits on the opposite side: a 10,000-token prompt goes through in one pass, so
each weight read does 10,000 tokens’ worth of arithmetic — around 20,000 FLOPs per byte, firmly
compute-bound. Prefill and decode are two workloads sharing one set of weights.
What that means when you size a serving fleet
The capacity of an inference machine is batch × bandwidth ÷ bytes of weights. Serve
Llama 3.1 70B in 8-bit precision — 70.6 GB —
across eight GPUs whose bandwidths add to 26.8 TB/s, and a decode step takes
70.6 ÷ 26,800 ≈ 2.6 ms: about 380 steps per second, or roughly 24,000 tokens per second at a
batch of 64. Peak FLOPS appears nowhere in that calculation — nor in what erodes it: the
KV cache, read from the same memory and growing with context and batch, and
the all-reduce inside every layer of a split model. Size a fleet from TFLOPS and you will
over-predict its throughput by orders of magnitude.
Training makes the network the ceiling
A training run costs about 6ND — six FLOPs per parameter per token. Seventy
billion parameters over 15 trillion tokens is 6 × 70×10⁹ × 15×10¹² = 6.3 × 10²⁴ FLOPs; 1,024 H100s
at 989 dense BF16 TFLOPS each peak at 1.01 × 10¹⁸ FLOP/s. No run achieves peak, and the fraction
it does achieve — Model FLOPs Utilization — decides everything: at 40% MFU that run takes 180
days, at 60% it takes 120. Twenty percentage points is two months of a thousand-GPU cluster,
about 1.5 million GPU-hours, on identical hardware.
MFU is mostly lost to the fabric. Distributed training is
synchronous: every step ends with a gradient all-reduce, and the standard ring implementation moves
about twice the gradient tensor in and out of each GPU. For a 70B model with BF16 gradients that is
2 × 141 GB = 282 GB per GPU per step — 5.6 seconds at 50 GB/s of cross-rack bandwidth, unless
it hides under the backward pass. Published figures land accordingly: 46.2% MFU for Google’s
540B PaLM run (PaLM), roughly 40% for Meta’s Llama 3 405B on
16,384 H100s (The Llama 3 Herd of Models). Half the silicon in a
state-of-the-art training cluster is waiting, and faster chips do not fix it.
Power decides whether the cluster can exist
A rack of AI accelerators draws on the order of 120 kW against roughly 8 kW for a conventional one,
and past about 30 kW air can no longer carry the heat away — hence capacity quoted in megawatts, and
grid interconnection rather than chip supply setting the schedule. See
AI data center and
AI energy consumption.
Real-World Applications
Rack-scale machines exist because of the byte hierarchy. NVIDIA’s GB200 NVL72 puts 72 GPUs into
one NVLink domain and sells the rack as a single unit — a product category built to move traffic
into the 900 GB/s tier instead of the 50 GB/s one. Google took a different route with
TPU pods, wiring 4,096 v4 chips in a 3D torus with optical
circuit switches that reconfigure the topology per job and route around dead racks
(TPU v4, ISCA 2023). Two vendors, one problem: interconnect.
Cloud instance choice is an interconnect decision. The current top-end training SKUs — AWS’s
P6e-GB200 UltraServers,
Azure’s ND GB200 v6
and Google Cloud’s A4X,
all built on NVIDIA’s GB200 NVL72 as of mid-2026 and succeeding the 2023-era ND H100 v5 and A3
instances — sell the same accelerators; what distinguishes them is non-blocking InfiniBand or EFA
between nodes. An engineer training a model pays for that fabric and packs every
node into one availability zone; serving the same model, they do the opposite, because a request
crossing an ocean has lost more latency than any kernel optimisation wins back. See
model deployment and
inference optimization.
Model architecture is now negotiated with the hardware. Mixture-of-Experts
routing and quantization both cut the bytes read per decode step, and both
are infrastructure decisions wearing the clothes of model design.
Key Concepts
- Model FLOPs Utilization: achieved arithmetic over peak — set by the fabric and the failure
rate, not the chip, and convertible straight into calendar time. - An idle accelerator costs what a busy one costs: a fleet averaging 40% utilisation pays 2.5×
the marginal cost per token. That is the real unit economics of serving.
Challenges
Capacity planning goes wrong in a specific, expensive way. Teams divide expected token volume by
an accelerator’s advertised TFLOPS and find the fleet delivers a fraction of the projection. The
correct denominator for decode is memory bandwidth, and the two rankings — most FLOPs, most
bandwidth — are not the same list.
Failure is routine at cluster scale. Individual accelerators are reliable; multiplying them by
sixteen thousand makes interruptions a design parameter. Checkpoint frequency, spare-node pools and
topology-aware restart become architecture rather than hygiene, and each costs utilisation — see
distributed training for failure rates from a real run.
Hardware arrives faster than the power to run it. Accelerators ship in months; substations and
grid interconnection do not, which is why siting now precedes purchasing. HBM is contested too — the
memory bought for weights is the memory the KV cache grows into, so raising the
batch to buy arithmetic intensity eventually runs out of room.
Future Trends
The strongest current direction is disaggregation: prefill and decode on separate pools, each
provisioned for the limit it actually hits. The same logic is growing scale-up domains — more
accelerators inside one high-bandwidth island before traffic falls to the slow tier — which is what
rack-scale products are for, and why co-packaged optics matters: electrical signalling is running
out of reach at the bandwidths the fabric now needs.
The pacing item on the memory side is high-bandwidth memory and
the advanced packaging that stacks it beside the die — a supply
constraint set by semiconductor manufacturing rather than
architecture. Bandwidth per chip has grown far more slowly than FLOPs per chip, so the gap this page
is about widens even as absolute numbers improve. The counterweight is algorithmic: sparser models,
KV-cache compression and speculative decoding all buy speed by
moving fewer bytes. And expect training campuses and inference footprints near users — including
edge AI on device — to keep diverging, because the two jobs want opposite
machines.
Common questions
What is the most important part of AI infrastructure?
High-end data-center [GPUs](/glossary/gpu-computing) — NVIDIA's Hopper (H100/H200) and Blackwell (B200/GB200) lines, with AMD's Instinct MI300X and MI355X as the main alternative — are the most critical components. However, networking (connecting the GPUs) and memory bandwidth are equally important for large-scale training.
Why is AI infrastructure so expensive?
It requires cutting-edge silicon, massive amounts of electricity for cooling and power, and specialized networking hardware like InfiniBand that can handle the high-speed communication between thousands of processors.
Is AI infrastructure limited by compute or by memory?
Both, but not at the same time. Processing a prompt is compute-bound — one pass reuses each weight for thousands of tokens' worth of arithmetic. Generating each subsequent token is memory-bound: the accelerator reads every weight out of memory to produce one word, doing about two arithmetic operations per byte it read, so the tensor cores sit nearly idle while text streams out.
What is MFU, and why do infrastructure teams care about it?
Model FLOPs Utilization is the arithmetic a training run actually performs divided by what the hardware could perform at peak. Real large runs land somewhere around 40-50%, and the gap is mostly time spent waiting on the network and on stragglers. Because the cluster costs the same whether it is computing or waiting, MFU converts almost directly into wall-clock time and money: on a 1,024-GPU cluster, 40% versus 60% is the difference between a six-month run and a four-month one.
How is inference infrastructure different from training infrastructure?
Training is one synchronous job that has to finish, so it wants thousands of accelerators wired tightly together in one building, close to cheap power. Inference is a steady stream of independent, latency-sensitive requests, so it wants smaller units of hardware placed near users. The same chip is used for both, but almost every decision around it — network, region, batch size, redundancy — comes out differently.