--- source: 'https://howaiworks.ai/glossary/distributed-training' section: glossary title: Distributed Training description: >- How one model is trained across thousands of GPUs — data, tensor and pipeline parallelism, and why the network rather than the chip becomes the bottleneck. tags: - AI Infrastructure - GPU - Hardware - Scaling - Distributed Computing - Performance - NVIDIA - Efficiency category: Infrastructure & Hardware datePublished: '2026-07-13' lastUpdated: '2026-07-24' --- # Distributed Training > How one model is trained across thousands of GPUs — data, tensor and pipeline parallelism, and why the network rather than the chip becomes the bottleneck. ## Definition **Distributed training is what turns thousands of separate accelerators into a single machine training a single model — and the hard part is not the arithmetic, it is the talking.** Meta trained Llama 3.1 405B on **16,384 H100 GPUs**. Those GPUs were not each doing their own thing. They were executing one gradient-descent step in lockstep, and before any step could finish, every one of them had to stop and exchange data with the others. That is the whole subject. A 16,384-GPU cluster is not 16,384 computers. It is *one* computer with 16,384 processors and a synchronisation barrier at the end of every step, and what limits it is how fast those processors can **agree** — a property of the network, not of the chips. Here is the scale of the problem in one figure. Training a model with mixed-precision Adam costs about **16 bytes per parameter**: 4 for the FP32 master weight, 8 for Adam's two moment estimates, plus 2 each for the BF16 weights and gradients. For 405 billion parameters that is **6.5 TB** of state. The H100s that ran the job hold **80 GB** each ([NVIDIA](https://www.nvidia.com/en-us/data-center/h100/), 2024) — so the state is about **80× a single GPU's memory**, and it does not fit on one GPU, or on eighty. Newer silicon does not rescue this: a 2025 accelerator carrying **192 GB** (NVIDIA's B200, AMD's Instinct MI300X) still holds barely a thirtieth of one run's state. What moves is the absolute; the ratio — model state far larger than any one chip — is the durable fact. Splitting the model across machines is not an optimisation you reach for when you want to go faster — it is a precondition for the run existing at all. ## How It Works ### A step is a barrier Stochastic gradient descent is sequential by construction: step *n+1* uses the weights produced by step *n*. You cannot parallelise across steps. All you can parallelise is the work *inside* one step, and a step ends with a reconciliation — every device must end up holding the same updated weights, or the replicas drift apart and the run is no longer training one model. Two consequences follow immediately, and they are the source of nearly everything else on this page. **The slowest participant sets the step time**, so a single GPU running 10% slow makes the entire cluster 10% slow. And **a participant that dies stops the step**, so one failed GPU out of 16,384 idles the other 16,383. ### All-reduce: the operation everything is built on The reconciliation is a collective called an **all-reduce**: sum a tensor that every GPU holds a copy of, and leave the total on every GPU. In data-parallel training that tensor is the gradient — one number per parameter — and it is all-reduced *every step*. The standard ring implementation is bandwidth-optimal, and its cost is worth memorising. For a tensor of `S` bytes across `K` GPUs, each GPU sends and receives: ``` 2 × (K − 1)/K × S bytes ≈ 2S for large K ``` Note what is *not* in that formula: `K`. Adding GPUs does not increase the bytes each one must push — which is why the ring is the algorithm everybody uses. What it does increase is the number of hops the data must survive, and the number of things that can be slow. ### The network is a hierarchy, and the tiers differ by 18× Inside one of Meta's Grand Teton servers (2024), eight H100s talk over **[NVLink](https://howaiworks.ai/glossary/interconnect) at 900 GB/s** per GPU ([NVIDIA](https://www.nvidia.com/en-us/data-center/h100/)). Between servers, the Llama 3 cluster ran a RoCE fabric giving each GPU **400 Gbps** — which is **50 GB/s**. That is a factor of **18** — the *ratio*, not either absolute, is what to remember, because it is the single most consequential number in the design of a training run. It is why the Llama 3 paper orders its parallelism dimensions by appetite: *"The innermost parallelism requires the highest network bandwidth and lowest latency, and hence is usually constrained to within the same server. The outermost parallelism may spread across a multi-hop network and should tolerate higher network latency."* The FLOPs themselves — where they land, how they are counted, what MFU means — belong to [FLOPs](https://howaiworks.ai/glossary/flops), and the matmuls they run on belong to [matrix multiplication](https://howaiworks.ai/glossary/matrix-multiplication). This page is about the *18*. ## Types Three ways to cut a model across devices. This taxonomy is real, universal, and composable: a frontier run uses all three simultaneously, because each one runs out for a different reason. ### Data parallelism — replicate the model, all-reduce the gradients Every GPU holds a complete copy of the model and processes a different slice of the batch. At the end of the backward pass, the copies have computed different gradients, so they all-reduce them and take the average. It is the simplest strategy and it scales beautifully — until the model stops fitting on one device (6.5 TB, above), and until the all-reduce stops fitting in the gaps. Its communication cost is fixed by the model: **2 bytes per parameter, all-reduced, every step**, regardless of how much data you feed it. ### Tensor parallelism — split a single matmul, and pay inside every layer A transformer layer is a handful of large matrix multiplications. Tensor parallelism cuts each weight matrix into column or row shards and gives one shard to each GPU. Every GPU multiplies its shard by the activations; the partial results are then all-reduced to reconstruct the true output — **and this happens twice in the forward pass and twice in the backward pass of every single layer.** This is communication *inside* the computation, not after it, and it cannot be hidden behind anything. That is why tensor parallelism is confined to a fast domain: Llama 3 ran **TP=8**, exactly the eight GPUs inside one NVLink server. Stretch it across the RoCE fabric and you are doing per-layer collectives at 1/18 the bandwidth. The size of the NVLink island is therefore a hard architectural limit on how far tensor parallelism can go — which is precisely why NVIDIA's **GB200 NVL72** (2024) sells a **72-GPU NVLink domain**, 72 Blackwell GPUs wired at **1.8 TB/s per GPU** ([NVIDIA](https://www.nvidia.com/en-us/data-center/gb200-nvl72/): 130 TB/s of aggregate NVLink over 72 GPUs). That is nine times the eight-GPU H100 island. It is not a bigger server. It is a bigger island. ### Pipeline parallelism — split the layers, and pay in the bubble Give GPU 1 the first eight layers, GPU 2 the next eight, and so on. Communication is now tiny — one activation tensor handed from stage to stage — so pipeline parallelism happily crosses the slow network. Llama 3 ran **PP=16**. Its cost is idle time. Stage 2 cannot start until stage 1 has produced something, and stage 1 has nothing to do once it has passed its work along. That dead time is the **pipeline bubble**, and for the standard schedules it is: ``` bubble fraction = (p − 1) / m p = pipeline stages m = micro-batches per step ``` With 16 stages and 32 micro-batches, roughly **47%** of every GPU-second is spent waiting. Push to 256 micro-batches and it falls to **6%**. The only cure for the bubble is more micro-batches — which means a bigger batch — which, as the Challenges section explains, is a resource you can run out of. ### Worked example: the 810 GB that never happens Take the published Llama 3 405B configuration at 16,384 GPUs — **TP=8, PP=16, DP=128**, 8,192-token sequences, a global batch of **16M tokens**, and a measured **400 TFLOPs/GPU** — and price the gradient all-reduce two ways. **First, naively: pure data parallelism.** Every GPU holds all 405B parameters, so the gradient tensor is one BF16 number per parameter: ``` S = 4.05e11 params × 2 bytes = 8.1e11 B = 810 GB ring = 2 × S = 1.62 TB per GPU, per step time = 1.62e12 B / 5.0e10 B/s = 32.4 seconds ``` Now the compute. Each GPU runs 6ND FLOPs on the tokens flowing through its replica (16M tokens ÷ 128 replicas = 131,072 tokens each): ``` work = 6 × 4.05e11 × 1.31072e5 / 128 GPUs-per-replica = 2.49e15 FLOPs per GPU time = 2.49e15 / 4.0e14 FLOP/s = 6.2 seconds ``` **32.4 seconds of network against 6.2 seconds of maths.** The cluster would spend **84% of its life waiting on the wire**, and $500M of silicon would run at roughly one-sixth of its rated speed. (It would also never boot, since 6.5 TB does not fit in 80 GB — but the network alone already kills it.) **Now the real configuration.** TP=8 and PP=16 mean each GPU is responsible for only 1/128th of the parameters: ``` shard = 4.05e11 / 128 = 3.16e9 params S = 3.16e9 × 2 bytes = 6.33 GB ring = 2 × (127/128) × 6.33 GB = 12.6 GB per GPU, per step time = 1.26e10 B / 5.0e10 B/s = 0.25 seconds ``` **0.25 seconds against 6.2 seconds of compute — 4%**, small enough to hide entirely underneath the backward pass while it runs. The same all-reduce, over the same slow network, got **129× cheaper** because two other parallelisms had already shrunk what each GPU was accountable for. That is the argument for the whole apparatus. The three strategies are not a menu. They are a factorisation, chosen so that each communication pattern lands on a network fast enough to carry it. ## Real-World Applications **Every frontier model you have used.** Llama 3.1 405B is simply the run whose infrastructure was documented in the most detail — 16,384 H100s, 8 per server, 400 Gbps of RoCE per GPU, 41% BF16 MFU. The [Llama](https://howaiworks.ai/models/llama) family, and every closed frontier model alongside it, exists because this factorisation works. **Buying a cluster.** The 18× ratio is why an AI datacentre's cost is dominated by things that are not accelerators: NVLink switches, optics, and a non-blocking fabric. xAI's Colossus in Memphis put **100,000 H100s on a single RDMA fabric**, and the word doing the work there is *single* — a network across which one job can synchronise, not a hundred thousand chips in a warehouse. A cluster whose interconnect underdelivers is not a cheap cluster; it is a slow one, and you pay for the idle GPUs anyway. **Deciding what you can train at all.** The parallelism plan converts "we have 512 GPUs" into "we can train a model of size N on D tokens in W weeks". Get it wrong — tensor parallelism spilling across nodes, too few micro-batches for the pipeline depth — and MFU collapses from 40% to 15%, which is the same as having bought a third of the cluster. What that compute buys you in capability is the subject of [scaling laws](https://howaiworks.ai/glossary/scaling-laws). ## Key Concepts - **Overlap is the whole game.** Communication that finishes while the GPU is still computing is free; communication that outlasts the compute is a stall. Modern frameworks launch the gradient reduce for layer *n* the instant layer *n*'s backward pass completes, so the all-reduce unzips itself against the rest of the backward pass. This is why the 0.25 s above costs approximately nothing — and why a 0.25 s figure that grew to 7 s would cost everything. - **Bandwidth *between* chips, not within them.** A GPU starved of data by its own HBM is hitting the [memory wall](https://howaiworks.ai/glossary/memory-wall), a different problem with a different fix. Distributed training is bottlenecked on the wire *between* accelerators, where the bandwidth is roughly two orders of magnitude lower than HBM's. On the H100s of the Llama 3 run that gap was concrete: **3.35 TB/s** reading local memory ([NVIDIA](https://www.nvidia.com/en-us/data-center/h100/), 2024) against **0.05 TB/s** reaching across the rack — about 67×. Newer chips widen both numbers together — a B200 reads its HBM3e more than twice as fast — so the two-orders-of-magnitude gap, not the specific figures, is the part that persists. - **Sharding is not parallelism.** FSDP and ZeRO cut the *memory* bill by splitting optimizer states, gradients and weights across the data-parallel group, then re-gathering weights just in time. They let a model fit; they do not reduce the amount of talking, and they usually increase it. ## Challenges ### Reliability does not survive multiplication Meta published the numbers, and they are the most under-appreciated fact about large-scale training. During a **54-day** snapshot of Llama 3 405B pre-training there were **466 job interruptions** — 47 planned, **419 unexpected**. **78%** of the unexpected ones were confirmed or suspected hardware faults; GPU issues accounted for **58.7%**, and HBM3 memory failures alone for **72 interruptions (17.2%)**. Divide it out. 54 days is 1,296 hours, so **419 interruptions is one every 3.1 hours**. Now invert the scale: each interruption represents roughly `16,384 × 3.1 ≈ 51,000 GPU-hours` of trouble-free operation, which is **about 5.8 years per GPU**. The individual H100s were *fine*. Almost six years of mean uptime each — and the cluster still broke every three hours. That is the arithmetic nobody outside the labs internalises: **reliability does not survive multiplication.** Run the same per-device numbers on a 100,000-GPU cluster and you get an interruption roughly **every 30 minutes**. Meta's predecessor run tells the same story — OPT-175B, on 992 A100s, took **at least 35 manual restarts and cycled over 100 hosts** in two months, and its logbook is a public record of engineers babysitting a machine that would not stay up. So checkpointing, failure detection and automatic restart are not operational hygiene bolted on afterwards. They are load-bearing structure, and the metric that matters is **effective training time** — the fraction of wall-clock that was actually spent training. Meta reported **above 90%** for Llama 3, which is a very good number and also an admission that a tenth of a nine-figure hardware bill went to recovering from things breaking. ### Strong scaling dies, and you cannot batch your way out The obvious move — keep the batch, add GPUs, finish sooner — fails in a specific and instructive way. Doubling the GPU count at a fixed global batch **halves the tokens each GPU sees per step**. Its compute time halves. Its gradient all-reduce does not. Communication that used to hide comfortably under the backward pass gets exposed, the GPUs start waiting, and MFU falls. Buy twice the hardware, get 1.3× the speed, pay 2×. The apparent fix is to raise the global batch so each GPU has enough to chew on. This works, and then it stops working: past the **critical batch size**, extra examples per step stop improving the loss per token and you are simply burning compute for a slightly cleaner gradient you did not need. Between a network you cannot make faster and a batch size you cannot usefully grow, there is a ceiling on how many GPUs one model can absorb — and the frontier keeps arriving at it. ### Everything runs at the speed of the worst participant Because the step is a barrier, the cluster's throughput is the *minimum* over 16,384 devices, not the mean. A GPU thermally throttling, a NIC negotiating a degraded link, one pipeline stage handed slightly more layers than the others — any of these taxes every GPU in the job. Straggler-hunting is a permanent occupation in large-scale training precisely because the distribution's left tail, not its centre, is what you are paying for. ## Future Trends The interconnect is where the money and the engineering are going, because it is where the ceiling is. The direction is visible in the hardware: NVLink domains growing from 8 GPUs to 72, and rack-scale systems sold as a single accelerator. Enlarging the fast island is the most direct way to buy back the 18×, since it lets tensor parallelism — the greediest strategy — cover more of the model before it has to cross the slow fabric. The other direction is to shrink the traffic rather than widen the pipe: gradient compression, FP8 collectives, and [MoE](https://howaiworks.ai/glossary/mixture-of-experts) architectures — which add a communication pattern of their own, routing tokens to experts on other devices, trading a cheaper matmul for a harder network problem. Reliability engineering will only grow in importance, because the failure arithmetic above scales linearly with GPU count and the clusters are not getting smaller. The 100,000-GPU run does not need better GPUs. It needs a machine that can lose one every half hour and not notice. ## Frequently Asked Questions ### What is the difference between data, tensor and pipeline parallelism? Data parallelism gives every GPU a full copy of the model and a different slice of the batch, then averages the gradients. Tensor parallelism splits a single matrix multiplication across GPUs, so they must exchange activations inside every layer. Pipeline parallelism gives each GPU a different block of layers and passes activations down the chain. They are not alternatives — a large run uses all three at once, because each one hits a different limit. ### Why is the network the bottleneck in distributed training rather than the GPU? Because a synchronous training step ends with every GPU having to agree on the new weights, and agreement moves bytes. On Meta's Llama 3 cluster (2024) each H100 reached the seven other GPUs in its own server at 900 GB/s over NVLink, but a GPU in another rack at only 400 Gbps — 50 GB/s, about 18 times slower. That intra-node-versus-cross-rack gap is the durable point, not the exact figures: each hardware generation raises a GPU's FLOPS faster than it raises the cross-rack number, so the fraction of a step spent waiting on the network only grows. ### What is an all-reduce? It is the operation that sums a tensor held on every GPU and leaves the sum on every GPU. In data-parallel training it is how gradients get averaged: one number per parameter, summed across all replicas, every single step. The standard ring implementation moves about 2× the tensor's size in and out of each GPU, no matter how many GPUs there are. ### How often does hardware fail during a large training run? Constantly. Meta reported 466 interruptions during a 54-day snapshot of Llama 3 405B pre-training on 16,384 H100s — 419 of them unexpected, about one every three hours, with 78% attributed to hardware. Individual GPUs are reliable; it is multiplying them by 16,384 that makes failure routine. Checkpointing is therefore not hygiene, it is architecture. ### Why can't you just add more GPUs to make training faster? Because at a fixed global batch size, doubling the GPU count halves the tokens each GPU processes per step, which halves its compute time while leaving the gradient all-reduce unchanged. Communication that used to hide under the backward pass becomes exposed, and the GPUs idle. Raising the batch size to compensate runs into the critical batch size, beyond which extra tokens per step stop buying extra learning. ## Related ### Related terms - [Distributed Computing](https://howaiworks.ai/glossary/distributed-computing) - [Parallel Processing](https://howaiworks.ai/glossary/parallel-processing) - [Training](https://howaiworks.ai/glossary/training) - [GPU Computing](https://howaiworks.ai/glossary/gpu-computing) - [Scaling Laws](https://howaiworks.ai/glossary/scaling-laws) - [Model Size](https://howaiworks.ai/glossary/model-size) --- Source: https://howaiworks.ai/glossary/distributed-training — HowAIWorks.ai