--- source: 'https://howaiworks.ai/glossary/ai-infrastructure' section: glossary title: AI Infrastructure description: >- Why AI clusters are designed around memory bandwidth and interconnect rather than FLOPs, and what that changes when you size a training run or a serving fleet. tags: - AI Infrastructure - GPU - TPU - Data Centers - Memory Bandwidth category: Infrastructure & Hardware datePublished: '2026-01-30' lastUpdated: '2026-07-24' --- # AI Infrastructure > Why AI clusters are designed around memory bandwidth and interconnect rather than FLOPs, and what that changes when you size a training run or a serving fleet. ## Definition **AI infrastructure is the hardware and software models are trained and served on — and it is not designed around arithmetic. It is designed around moving bytes.** Every accelerator has three budgets, and whichever runs out first decides how fast your job goes. Take NVIDIA's H100 SXM as a worked example. By its [published specification](https://www.nvidia.com/en-us/data-center/h100/) it has **3.35 TB/s** of on-package memory bandwidth, **900 GB/s** of NVLink to the GPUs in its own server, and — through a typical 400 Gbps port — about **50 GB/s** to a GPU in another rack: a ratio of roughly **67 : 18 : 1**. A byte that stays in the accelerator's own memory arrives sixty-seven times faster than one crossing the data-center floor, and nearly every cluster design decision is an attempt to keep bytes in the fastest tier they can be kept in. Those absolute figures are a mid-2026 snapshot and will rot; the ordering of the tiers does not. Newer accelerators push every number up without flattening the hierarchy — NVIDIA's Blackwell **B200** carries **192 GB** of HBM3e at roughly **8 TB/s** ([NVIDIA](https://www.nvidia.com/en-us/data-center/hgx/)), and AMD's Instinct **MI300X** holds **192 GB** at **5.3 TB/s**, with the 2025 **MI355X** at **288 GB** ([AMD](https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html)) — but on-package bandwidth still beats intra-server NVLink, which still beats the cross-rack fabric by a wide multiple. It is the ratio, not any one chip's spec sheet, that this page is about. ## How It Works ### Decoding a token is a memory problem, not a maths problem Divide a chip's peak arithmetic rate by its memory bandwidth and you get its **break-even intensity**: the FLOPs it must perform per byte read to keep the tensor cores busy. For an H100 in FP8 that is `1,979 × 10¹² FLOP/s ÷ 3.35 × 10¹² bytes/s ≈ 590 FLOPs per byte`. Generating one token for one user comes nowhere near it. The forward pass reads every weight once and does about two floating-point operations per parameter, so its intensity is **2 FLOPs per byte** — nearly 300× under the line. Batching helps, because weights once read serve every request in the batch: intensity becomes `2 × batch`. At a batch of 64 that is 128 FLOPs per byte, about **22% of peak arithmetic**, and you would need roughly **295** concurrent requests merely to break even. The [memory wall](https://howaiworks.ai/glossary/memory-wall) page works this through in full. Processing the *prompt* sits on the opposite side: a 10,000-token prompt goes through in one pass, so each weight read does 10,000 tokens' worth of arithmetic — around 20,000 FLOPs per byte, firmly compute-bound. Prefill and decode are two workloads sharing one set of weights. ### What that means when you size a serving fleet The capacity of an inference machine is `batch × bandwidth ÷ bytes of weights`. Serve [Llama 3.1 70B](https://huggingface.co/meta-llama/Llama-3.1-70B) in 8-bit precision — 70.6 GB — across eight GPUs whose bandwidths add to 26.8 TB/s, and a decode step takes `70.6 ÷ 26,800 ≈ 2.6 ms`: about 380 steps per second, or roughly **24,000 tokens per second** at a batch of 64. Peak FLOPS appears nowhere in that calculation — nor in what erodes it: the [KV cache](https://howaiworks.ai/glossary/kv-cache), read from the same memory and growing with context and batch, and the all-reduce inside every layer of a split model. Size a fleet from TFLOPS and you will over-predict its throughput by orders of magnitude. ### Training makes the network the ceiling A training run costs about `6ND` — six [FLOPs](https://howaiworks.ai/glossary/flops) per parameter per token. Seventy billion parameters over 15 trillion tokens is `6 × 70×10⁹ × 15×10¹² = 6.3 × 10²⁴` FLOPs; 1,024 H100s at 989 dense BF16 TFLOPS each peak at `1.01 × 10¹⁸` FLOP/s. No run achieves peak, and the fraction it does achieve — **Model FLOPs Utilization** — decides everything: at 40% MFU that run takes **180 days**, at 60% it takes **120**. Twenty percentage points is two months of a thousand-GPU cluster, about **1.5 million GPU-hours**, on identical hardware. MFU is mostly lost to the fabric. [Distributed training](https://howaiworks.ai/glossary/distributed-training) is synchronous: every step ends with a gradient all-reduce, and the standard ring implementation moves about twice the gradient tensor in and out of each GPU. For a 70B model with BF16 gradients that is `2 × 141 GB = 282 GB` per GPU per step — **5.6 seconds** at 50 GB/s of cross-rack bandwidth, unless it hides under the backward pass. Published figures land accordingly: **46.2% MFU** for Google's 540B PaLM run ([PaLM](https://arxiv.org/abs/2204.02311)), roughly 40% for Meta's Llama 3 405B on 16,384 H100s ([The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783)). Half the silicon in a state-of-the-art training cluster is waiting, and faster chips do not fix it. ### Power decides whether the cluster can exist A rack of AI accelerators draws on the order of 120 kW against roughly 8 kW for a conventional one, and past about 30 kW air can no longer carry the heat away — hence capacity quoted in megawatts, and grid interconnection rather than chip supply setting the schedule. See [AI data center](https://howaiworks.ai/glossary/ai-data-center) and [AI energy consumption](https://howaiworks.ai/glossary/ai-energy-consumption). ## Real-World Applications **Rack-scale machines exist because of the byte hierarchy.** NVIDIA's GB200 NVL72 puts 72 GPUs into one NVLink domain and sells the rack as a single unit — a product category built to move traffic into the 900 GB/s tier instead of the 50 GB/s one. Google took a different route with [TPU](https://howaiworks.ai/glossary/tensor-processing-unit) pods, wiring 4,096 v4 chips in a 3D torus with **optical circuit switches** that reconfigure the topology per job and route around dead racks ([TPU v4, ISCA 2023](https://arxiv.org/abs/2304.01433)). Two vendors, one problem: interconnect. **Cloud instance choice is an interconnect decision.** The current top-end training SKUs — AWS's [P6e-GB200 UltraServers](https://aws.amazon.com/about-aws/whats-new/2025/07/amazon-p6e-gb200-ultraservers-gpu-performance-ec2), Azure's [ND GB200 v6](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/gpu-accelerated/nd-gb200-v6-series) and Google Cloud's [A4X](https://cloud.google.com/blog/products/compute/new-a4x-vms-powered-by-nvidia-gb200-gpus), all built on NVIDIA's GB200 NVL72 as of mid-2026 and succeeding the 2023-era ND H100 v5 and A3 instances — sell the same accelerators; what distinguishes them is non-blocking InfiniBand or EFA between nodes. An engineer training a model pays for that fabric and packs every node into one availability zone; serving the same model, they do the opposite, because a request crossing an ocean has lost more latency than any kernel optimisation wins back. See [model deployment](https://howaiworks.ai/glossary/model-deployment) and [inference optimization](https://howaiworks.ai/glossary/inference-optimization). **Model architecture is now negotiated with the hardware.** [Mixture-of-Experts](https://howaiworks.ai/glossary/mixture-of-experts) routing and [quantization](https://howaiworks.ai/glossary/quantization) both cut the bytes read per decode step, and both are infrastructure decisions wearing the clothes of model design. ## Key Concepts - **Model FLOPs Utilization**: achieved arithmetic over peak — set by the fabric and the failure rate, not the chip, and convertible straight into calendar time. - **An idle accelerator costs what a busy one costs**: a fleet averaging 40% utilisation pays 2.5× the marginal cost per token. That is the real unit economics of serving. ## Challenges **Capacity planning goes wrong in a specific, expensive way.** Teams divide expected token volume by an accelerator's advertised TFLOPS and find the fleet delivers a fraction of the projection. The correct denominator for decode is memory bandwidth, and the two rankings — most FLOPs, most bandwidth — are not the same list. **Failure is routine at cluster scale.** Individual accelerators are reliable; multiplying them by sixteen thousand makes interruptions a design parameter. Checkpoint frequency, spare-node pools and topology-aware restart become architecture rather than hygiene, and each costs utilisation — see [distributed training](https://howaiworks.ai/glossary/distributed-training) for failure rates from a real run. **Hardware arrives faster than the power to run it.** Accelerators ship in months; substations and grid interconnection do not, which is why siting now precedes purchasing. HBM is contested too — the memory bought for weights is the memory the [KV cache](https://howaiworks.ai/glossary/kv-cache) grows into, so raising the batch to buy arithmetic intensity eventually runs out of room. ## Future Trends The strongest current direction is **disaggregation**: prefill and decode on separate pools, each provisioned for the limit it actually hits. The same logic is growing **scale-up domains** — more accelerators inside one high-bandwidth island before traffic falls to the slow tier — which is what rack-scale products are for, and why **co-packaged optics** matters: electrical signalling is running out of reach at the bandwidths the fabric now needs. The pacing item on the memory side is [high-bandwidth memory](https://howaiworks.ai/glossary/high-bandwidth-memory) and the [advanced packaging](https://howaiworks.ai/glossary/advanced-packaging) that stacks it beside the die — a supply constraint set by [semiconductor manufacturing](https://howaiworks.ai/glossary/semiconductor-manufacturing) rather than architecture. Bandwidth per chip has grown far more slowly than FLOPs per chip, so the gap this page is about widens even as absolute numbers improve. The counterweight is algorithmic: sparser models, KV-cache compression and [speculative decoding](https://howaiworks.ai/glossary/speculative-decoding) all buy speed by moving fewer bytes. And expect training campuses and inference footprints near users — including [edge AI](https://howaiworks.ai/glossary/edge-ai) on device — to keep diverging, because the two jobs want opposite machines. ## Frequently Asked Questions ### What is the most important part of AI infrastructure? High-end data-center [GPUs](https://howaiworks.ai/glossary/gpu-computing) — NVIDIA's Hopper (H100/H200) and Blackwell (B200/GB200) lines, with AMD's Instinct MI300X and MI355X as the main alternative — are the most critical components. However, networking (connecting the GPUs) and memory bandwidth are equally important for large-scale training. ### Why is AI infrastructure so expensive? It requires cutting-edge silicon, massive amounts of electricity for cooling and power, and specialized networking hardware like InfiniBand that can handle the high-speed communication between thousands of processors. ### Is AI infrastructure limited by compute or by memory? Both, but not at the same time. Processing a prompt is compute-bound — one pass reuses each weight for thousands of tokens' worth of arithmetic. Generating each subsequent token is memory-bound: the accelerator reads every weight out of memory to produce one word, doing about two arithmetic operations per byte it read, so the tensor cores sit nearly idle while text streams out. ### What is MFU, and why do infrastructure teams care about it? Model FLOPs Utilization is the arithmetic a training run actually performs divided by what the hardware could perform at peak. Real large runs land somewhere around 40-50%, and the gap is mostly time spent waiting on the network and on stragglers. Because the cluster costs the same whether it is computing or waiting, MFU converts almost directly into wall-clock time and money: on a 1,024-GPU cluster, 40% versus 60% is the difference between a six-month run and a four-month one. ### How is inference infrastructure different from training infrastructure? Training is one synchronous job that has to finish, so it wants thousands of accelerators wired tightly together in one building, close to cheap power. Inference is a steady stream of independent, latency-sensitive requests, so it wants smaller units of hardware placed near users. The same chip is used for both, but almost every decision around it — network, region, batch size, redundancy — comes out differently. ## Related ### Related terms - [GPU Computing](https://howaiworks.ai/glossary/gpu-computing) - [Cloud Computing](https://howaiworks.ai/glossary/cloud-computing) - [Tensor Processing Unit (TPU)](https://howaiworks.ai/glossary/tensor-processing-unit) - [Scalable AI](https://howaiworks.ai/glossary/scalable-ai) - [Machine Learning (ML)](https://howaiworks.ai/glossary/machine-learning) --- Source: https://howaiworks.ai/glossary/ai-infrastructure — HowAIWorks.ai