Definition
Distributed training is what turns thousands of separate accelerators into a single machine training a single model — and the hard part is not the arithmetic, it is the talking. Meta trained Llama 3.1 405B on 16,384 H100 GPUs. Those GPUs were not each doing their own thing. They were executing one gradient-descent step in lockstep, and before any step could finish, every one of them had to stop and exchange data with the others.
That is the whole subject. A 16,384-GPU cluster is not 16,384 computers. It is one computer with 16,384 processors and a synchronisation barrier at the end of every step, and what limits it is how fast those processors can agree — a property of the network, not of the chips.
Here is the scale of the problem in one figure. Training a model with mixed-precision Adam costs about 16 bytes per parameter: 4 for the FP32 master weight, 8 for Adam's two moment estimates, plus 2 each for the BF16 weights and gradients. For 405 billion parameters that is 6.5 TB of state. An H100 holds 80 GB. The model does not fit on one GPU, or on eighty. Splitting it across machines is not an optimisation you reach for when you want to go faster — it is a precondition for the run existing at all.
How It Works
A step is a barrier
Stochastic gradient descent is sequential by construction: step n+1 uses the weights produced by step n. You cannot parallelise across steps. All you can parallelise is the work inside one step, and a step ends with a reconciliation — every device must end up holding the same updated weights, or the replicas drift apart and the run is no longer training one model.
Two consequences follow immediately, and they are the source of nearly everything else on this page. The slowest participant sets the step time, so a single GPU running 10% slow makes the entire cluster 10% slow. And a participant that dies stops the step, so one failed GPU out of 16,384 idles the other 16,383.
All-reduce: the operation everything is built on
The reconciliation is a collective called an all-reduce: sum a tensor that every GPU holds a copy of, and leave the total on every GPU. In data-parallel training that tensor is the gradient — one number per parameter — and it is all-reduced every step.
The standard ring implementation is bandwidth-optimal, and its cost is worth memorising. For a tensor
of S bytes across K GPUs, each GPU sends and receives:
2 × (K − 1)/K × S bytes ≈ 2S for large K
Note what is not in that formula: K. Adding GPUs does not increase the bytes each one must push —
which is why the ring is the algorithm everybody uses. What it does increase is the number of hops the
data must survive, and the number of things that can be slow.
The network is a hierarchy, and the tiers differ by 18×
Inside one of Meta's Grand Teton servers, eight H100s talk over NVLink at 900 GB/s per GPU. Between servers, the Llama 3 cluster ran a RoCE fabric giving each GPU 400 Gbps — which is 50 GB/s.
That is a factor of 18, and it is the single most consequential number in the design of a training run. It is why the Llama 3 paper orders its parallelism dimensions by appetite: "The innermost parallelism requires the highest network bandwidth and lowest latency, and hence is usually constrained to within the same server. The outermost parallelism may spread across a multi-hop network and should tolerate higher network latency."
The FLOPs themselves — where they land, how they are counted, what MFU means — belong to FLOPs, and the matmuls they run on belong to matrix multiplication. This page is about the 18.
Types
Three ways to cut a model across devices. This taxonomy is real, universal, and composable: a frontier run uses all three simultaneously, because each one runs out for a different reason.
Data parallelism — replicate the model, all-reduce the gradients
Every GPU holds a complete copy of the model and processes a different slice of the batch. At the end of the backward pass, the copies have computed different gradients, so they all-reduce them and take the average.
It is the simplest strategy and it scales beautifully — until the model stops fitting on one device (6.5 TB, above), and until the all-reduce stops fitting in the gaps. Its communication cost is fixed by the model: 2 bytes per parameter, all-reduced, every step, regardless of how much data you feed it.
Tensor parallelism — split a single matmul, and pay inside every layer
A transformer layer is a handful of large matrix multiplications. Tensor parallelism cuts each weight matrix into column or row shards and gives one shard to each GPU. Every GPU multiplies its shard by the activations; the partial results are then all-reduced to reconstruct the true output — and this happens twice in the forward pass and twice in the backward pass of every single layer.
This is communication inside the computation, not after it, and it cannot be hidden behind anything. That is why tensor parallelism is confined to a fast domain: Llama 3 ran TP=8, exactly the eight GPUs inside one NVLink server. Stretch it across the RoCE fabric and you are doing per-layer collectives at 1/18 the bandwidth. The size of the NVLink island is therefore a hard architectural limit on how far tensor parallelism can go — which is precisely why NVIDIA's GB200 NVL72 sells a 72-GPU NVLink domain at 1.8 TB/s per GPU. It is not a bigger server. It is a bigger island.
Pipeline parallelism — split the layers, and pay in the bubble
Give GPU 1 the first eight layers, GPU 2 the next eight, and so on. Communication is now tiny — one activation tensor handed from stage to stage — so pipeline parallelism happily crosses the slow network. Llama 3 ran PP=16.
Its cost is idle time. Stage 2 cannot start until stage 1 has produced something, and stage 1 has nothing to do once it has passed its work along. That dead time is the pipeline bubble, and for the standard schedules it is:
bubble fraction = (p − 1) / m
p = pipeline stages
m = micro-batches per step
With 16 stages and 32 micro-batches, roughly 47% of every GPU-second is spent waiting. Push to 256 micro-batches and it falls to 6%. The only cure for the bubble is more micro-batches — which means a bigger batch — which, as the Challenges section explains, is a resource you can run out of.
Worked example: the 810 GB that never happens
Take the published Llama 3 405B configuration at 16,384 GPUs — TP=8, PP=16, DP=128, 8,192-token sequences, a global batch of 16M tokens, and a measured 400 TFLOPs/GPU — and price the gradient all-reduce two ways.
First, naively: pure data parallelism. Every GPU holds all 405B parameters, so the gradient tensor is one BF16 number per parameter:
S = 4.05e11 params × 2 bytes = 8.1e11 B = 810 GB
ring = 2 × S = 1.62 TB per GPU, per step
time = 1.62e12 B / 5.0e10 B/s = 32.4 seconds
Now the compute. Each GPU runs 6ND FLOPs on the tokens flowing through its replica (16M tokens ÷ 128 replicas = 131,072 tokens each):
work = 6 × 4.05e11 × 1.31072e5 / 128 GPUs-per-replica = 2.49e15 FLOPs per GPU
time = 2.49e15 / 4.0e14 FLOP/s = 6.2 seconds
32.4 seconds of network against 6.2 seconds of maths. The cluster would spend 84% of its life waiting on the wire, and $500M of silicon would run at roughly one-sixth of its rated speed. (It would also never boot, since 6.5 TB does not fit in 80 GB — but the network alone already kills it.)
Now the real configuration. TP=8 and PP=16 mean each GPU is responsible for only 1/128th of the parameters:
shard = 4.05e11 / 128 = 3.16e9 params
S = 3.16e9 × 2 bytes = 6.33 GB
ring = 2 × (127/128) × 6.33 GB = 12.6 GB per GPU, per step
time = 1.26e10 B / 5.0e10 B/s = 0.25 seconds
0.25 seconds against 6.2 seconds of compute — 4%, small enough to hide entirely underneath the backward pass while it runs. The same all-reduce, over the same slow network, got 129× cheaper because two other parallelisms had already shrunk what each GPU was accountable for.
That is the argument for the whole apparatus. The three strategies are not a menu. They are a factorisation, chosen so that each communication pattern lands on a network fast enough to carry it.
Real-World Applications
Every frontier model you have used. Llama 3.1 405B is simply the run whose infrastructure was documented in the most detail — 16,384 H100s, 8 per server, 400 Gbps of RoCE per GPU, 41% BF16 MFU. The Llama family, and every closed frontier model alongside it, exists because this factorisation works.
Buying a cluster. The 18× ratio is why an AI datacentre's cost is dominated by things that are not accelerators: NVLink switches, optics, and a non-blocking fabric. xAI's Colossus in Memphis put 100,000 H100s on a single RDMA fabric, and the word doing the work there is single — a network across which one job can synchronise, not a hundred thousand chips in a warehouse. A cluster whose interconnect underdelivers is not a cheap cluster; it is a slow one, and you pay for the idle GPUs anyway.
Deciding what you can train at all. The parallelism plan converts "we have 512 GPUs" into "we can train a model of size N on D tokens in W weeks". Get it wrong — tensor parallelism spilling across nodes, too few micro-batches for the pipeline depth — and MFU collapses from 40% to 15%, which is the same as having bought a third of the cluster. What that compute buys you in capability is the subject of scaling laws.
Key Concepts
- Overlap is the whole game. Communication that finishes while the GPU is still computing is free; communication that outlasts the compute is a stall. Modern frameworks launch the gradient reduce for layer n the instant layer n's backward pass completes, so the all-reduce unzips itself against the rest of the backward pass. This is why the 0.25 s above costs approximately nothing — and why a 0.25 s figure that grew to 7 s would cost everything.
- Bandwidth between chips, not within them. A GPU starved of data by its own HBM is hitting the memory wall, a different problem with a different fix. Distributed training is bottlenecked on the wire between accelerators, where the bandwidth is roughly two orders of magnitude lower than HBM's — an H100 reads its own memory at 3.35 TB/s and reaches across the rack at 0.05 TB/s.
- Sharding is not parallelism. FSDP and ZeRO cut the memory bill by splitting optimizer states, gradients and weights across the data-parallel group, then re-gathering weights just in time. They let a model fit; they do not reduce the amount of talking, and they usually increase it.
Challenges
Reliability does not survive multiplication
Meta published the numbers, and they are the most under-appreciated fact about large-scale training. During a 54-day snapshot of Llama 3 405B pre-training there were 466 job interruptions — 47 planned, 419 unexpected. 78% of the unexpected ones were confirmed or suspected hardware faults; GPU issues accounted for 58.7%, and HBM3 memory failures alone for 72 interruptions (17.2%).
Divide it out. 54 days is 1,296 hours, so 419 interruptions is one every 3.1 hours. Now invert the
scale: each interruption represents roughly 16,384 × 3.1 ≈ 51,000 GPU-hours of trouble-free operation,
which is about 5.8 years per GPU. The individual H100s were fine. Almost six years of mean uptime
each — and the cluster still broke every three hours.
That is the arithmetic nobody outside the labs internalises: reliability does not survive multiplication. Run the same per-device numbers on a 100,000-GPU cluster and you get an interruption roughly every 30 minutes. Meta's predecessor run tells the same story — OPT-175B, on 992 A100s, took at least 35 manual restarts and cycled over 100 hosts in two months, and its logbook is a public record of engineers babysitting a machine that would not stay up.
So checkpointing, failure detection and automatic restart are not operational hygiene bolted on afterwards. They are load-bearing structure, and the metric that matters is effective training time — the fraction of wall-clock that was actually spent training. Meta reported above 90% for Llama 3, which is a very good number and also an admission that a tenth of a nine-figure hardware bill went to recovering from things breaking.
Strong scaling dies, and you cannot batch your way out
The obvious move — keep the batch, add GPUs, finish sooner — fails in a specific and instructive way. Doubling the GPU count at a fixed global batch halves the tokens each GPU sees per step. Its compute time halves. Its gradient all-reduce does not. Communication that used to hide comfortably under the backward pass gets exposed, the GPUs start waiting, and MFU falls. Buy twice the hardware, get 1.3× the speed, pay 2×.
The apparent fix is to raise the global batch so each GPU has enough to chew on. This works, and then it stops working: past the critical batch size, extra examples per step stop improving the loss per token and you are simply burning compute for a slightly cleaner gradient you did not need. Between a network you cannot make faster and a batch size you cannot usefully grow, there is a ceiling on how many GPUs one model can absorb — and the frontier keeps arriving at it.
Everything runs at the speed of the worst participant
Because the step is a barrier, the cluster's throughput is the minimum over 16,384 devices, not the mean. A GPU thermally throttling, a NIC negotiating a degraded link, one pipeline stage handed slightly more layers than the others — any of these taxes every GPU in the job. Straggler-hunting is a permanent occupation in large-scale training precisely because the distribution's left tail, not its centre, is what you are paying for.
Future Trends
The interconnect is where the money and the engineering are going, because it is where the ceiling is. The direction is visible in the hardware: NVLink domains growing from 8 GPUs to 72, and rack-scale systems sold as a single accelerator. Enlarging the fast island is the most direct way to buy back the 18×, since it lets tensor parallelism — the greediest strategy — cover more of the model before it has to cross the slow fabric.
The other direction is to shrink the traffic rather than widen the pipe: gradient compression, FP8 collectives, and MoE architectures — which add a communication pattern of their own, routing tokens to experts on other devices, trading a cheaper matmul for a harder network problem. Reliability engineering will only grow in importance, because the failure arithmetic above scales linearly with GPU count and the clusters are not getting smaller. The 100,000-GPU run does not need better GPUs. It needs a machine that can lose one every half hour and not notice.