Definition
AI infrastructure is the hardware and software models are trained and served on — and it is not designed around arithmetic. It is designed around moving bytes. Every accelerator has three budgets, and whichever runs out first decides how fast your job goes.
Take NVIDIA's H100 SXM as a worked example. By its published specification it has 3.35 TB/s of on-package memory bandwidth, 900 GB/s of NVLink to the GPUs in its own server, and — through a typical 400 Gbps port — about 50 GB/s to a GPU in another rack: a ratio of roughly 67 : 18 : 1. A byte that stays in the accelerator's own memory arrives sixty-seven times faster than one crossing the data-center floor, and nearly every cluster design decision is an attempt to keep bytes in the fastest tier they can be kept in.
Those absolute figures are a mid-2026 snapshot and will rot; the ordering of the tiers does not. Newer accelerators push every number up without flattening the hierarchy — NVIDIA's Blackwell B200 carries 192 GB of HBM3e at roughly 8 TB/s (NVIDIA), and AMD's Instinct MI300X holds 192 GB at 5.3 TB/s, with the 2025 MI355X at 288 GB (AMD) — but on-package bandwidth still beats intra-server NVLink, which still beats the cross-rack fabric by a wide multiple. It is the ratio, not any one chip's spec sheet, that this page is about.
How It Works
Decoding a token is a memory problem, not a maths problem
Divide a chip's peak arithmetic rate by its memory bandwidth and you get its break-even
intensity: the FLOPs it must perform per byte read to keep the tensor cores busy. For an H100 in
FP8 that is 1,979 × 10¹² FLOP/s ÷ 3.35 × 10¹² bytes/s ≈ 590 FLOPs per byte.
Generating one token for one user comes nowhere near it. The forward pass reads every weight once
and does about two floating-point operations per parameter, so its intensity is 2 FLOPs per byte
— nearly 300× under the line. Batching helps, because weights once read serve every request in the
batch: intensity becomes 2 × batch. At a batch of 64 that is 128 FLOPs per byte, about 22% of
peak arithmetic, and you would need roughly 295 concurrent requests merely to break even. The
memory wall page works this through in full.
Processing the prompt sits on the opposite side: a 10,000-token prompt goes through in one pass, so each weight read does 10,000 tokens' worth of arithmetic — around 20,000 FLOPs per byte, firmly compute-bound. Prefill and decode are two workloads sharing one set of weights.
What that means when you size a serving fleet
The capacity of an inference machine is batch × bandwidth ÷ bytes of weights. Serve
Llama 3.1 70B in 8-bit precision — 70.6 GB —
across eight GPUs whose bandwidths add to 26.8 TB/s, and a decode step takes
70.6 ÷ 26,800 ≈ 2.6 ms: about 380 steps per second, or roughly 24,000 tokens per second at a
batch of 64. Peak FLOPS appears nowhere in that calculation — nor in what erodes it: the
KV cache, read from the same memory and growing with context and batch, and
the all-reduce inside every layer of a split model. Size a fleet from TFLOPS and you will
over-predict its throughput by orders of magnitude.
Training makes the network the ceiling
A training run costs about 6ND — six FLOPs per parameter per token. Seventy
billion parameters over 15 trillion tokens is 6 × 70×10⁹ × 15×10¹² = 6.3 × 10²⁴ FLOPs; 1,024 H100s
at 989 dense BF16 TFLOPS each peak at 1.01 × 10¹⁸ FLOP/s. No run achieves peak, and the fraction
it does achieve — Model FLOPs Utilization — decides everything: at 40% MFU that run takes 180
days, at 60% it takes 120. Twenty percentage points is two months of a thousand-GPU cluster,
about 1.5 million GPU-hours, on identical hardware.
MFU is mostly lost to the fabric. Distributed training is
synchronous: every step ends with a gradient all-reduce, and the standard ring implementation moves
about twice the gradient tensor in and out of each GPU. For a 70B model with BF16 gradients that is
2 × 141 GB = 282 GB per GPU per step — 5.6 seconds at 50 GB/s of cross-rack bandwidth, unless
it hides under the backward pass. Published figures land accordingly: 46.2% MFU for Google's
540B PaLM run (PaLM), roughly 40% for Meta's Llama 3 405B on
16,384 H100s (The Llama 3 Herd of Models). Half the silicon in a
state-of-the-art training cluster is waiting, and faster chips do not fix it.
Power decides whether the cluster can exist
A rack of AI accelerators draws on the order of 120 kW against roughly 8 kW for a conventional one, and past about 30 kW air can no longer carry the heat away — hence capacity quoted in megawatts, and grid interconnection rather than chip supply setting the schedule. See AI data center and AI energy consumption.
Real-World Applications
Rack-scale machines exist because of the byte hierarchy. NVIDIA's GB200 NVL72 puts 72 GPUs into one NVLink domain and sells the rack as a single unit — a product category built to move traffic into the 900 GB/s tier instead of the 50 GB/s one. Google took a different route with TPU pods, wiring 4,096 v4 chips in a 3D torus with optical circuit switches that reconfigure the topology per job and route around dead racks (TPU v4, ISCA 2023). Two vendors, one problem: interconnect.
Cloud instance choice is an interconnect decision. The current top-end training SKUs — AWS's P6e-GB200 UltraServers, Azure's ND GB200 v6 and Google Cloud's A4X, all built on NVIDIA's GB200 NVL72 as of mid-2026 and succeeding the 2023-era ND H100 v5 and A3 instances — sell the same accelerators; what distinguishes them is non-blocking InfiniBand or EFA between nodes. An engineer training a model pays for that fabric and packs every node into one availability zone; serving the same model, they do the opposite, because a request crossing an ocean has lost more latency than any kernel optimisation wins back. See model deployment and inference optimization.
Model architecture is now negotiated with the hardware. Mixture-of-Experts routing and quantization both cut the bytes read per decode step, and both are infrastructure decisions wearing the clothes of model design.
Key Concepts
- Model FLOPs Utilization: achieved arithmetic over peak — set by the fabric and the failure rate, not the chip, and convertible straight into calendar time.
- An idle accelerator costs what a busy one costs: a fleet averaging 40% utilisation pays 2.5× the marginal cost per token. That is the real unit economics of serving.
Challenges
Capacity planning goes wrong in a specific, expensive way. Teams divide expected token volume by an accelerator's advertised TFLOPS and find the fleet delivers a fraction of the projection. The correct denominator for decode is memory bandwidth, and the two rankings — most FLOPs, most bandwidth — are not the same list.
Failure is routine at cluster scale. Individual accelerators are reliable; multiplying them by sixteen thousand makes interruptions a design parameter. Checkpoint frequency, spare-node pools and topology-aware restart become architecture rather than hygiene, and each costs utilisation — see distributed training for failure rates from a real run.
Hardware arrives faster than the power to run it. Accelerators ship in months; substations and grid interconnection do not, which is why siting now precedes purchasing. HBM is contested too — the memory bought for weights is the memory the KV cache grows into, so raising the batch to buy arithmetic intensity eventually runs out of room.
Future Trends
The strongest current direction is disaggregation: prefill and decode on separate pools, each provisioned for the limit it actually hits. The same logic is growing scale-up domains — more accelerators inside one high-bandwidth island before traffic falls to the slow tier — which is what rack-scale products are for, and why co-packaged optics matters: electrical signalling is running out of reach at the bandwidths the fabric now needs.
The pacing item on the memory side is high-bandwidth memory and the advanced packaging that stacks it beside the die — a supply constraint set by semiconductor manufacturing rather than architecture. Bandwidth per chip has grown far more slowly than FLOPs per chip, so the gap this page is about widens even as absolute numbers improve. The counterweight is algorithmic: sparser models, KV-cache compression and speculative decoding all buy speed by moving fewer bytes. And expect training campuses and inference footprints near users — including edge AI on device — to keep diverging, because the two jobs want opposite machines.