Definition
An interconnect is the wiring that moves data between processors in an AI system — and, more usefully, the name for the hierarchy of such wiring, because there is no single kind. Data travels over three very different tiers: on-package memory glued to the chip itself, chip-to-chip links inside one server (Nvidia's NVLink is the best-known), and a network fabric between servers (InfiniBand or Ethernet-based RoCE). The one fact worth carrying away is that each rung down is several times to an order of magnitude slower than the one above it, and that gap — not the speed of the chips — decides how a large model is carved up and spread across a cluster.
That matters because a modern training or inference job is not one computer. Llama 3 405B was trained on up to 16,000 GPUs (Meta, 2024), and every one of them has to stay fed and, at the end of each step, has to agree with the others on the updated weights. Agreement moves bytes, and bytes move at the speed of whichever tier they have to cross. A job that keeps its heaviest traffic on the fastest tier runs; the same job with that traffic pushed onto the slow tier can spend most of its time waiting. The interconnect is where distributed AI is won or lost, and it is invisible in every framing that talks only about FLOPS.
How It Works
Think of a GPU as a very fast worker who is useless without a steady supply of numbers. The interconnect hierarchy is the set of supply routes, ordered from fastest to slowest.
Tier 1 — on-package memory. The numbers a GPU is working on live in High Bandwidth Memory (HBM) stacked millimetres away inside the same package. On an H100, that memory delivers 3.35 TB/s (NVIDIA H100 spec, as of 2026). This is the reference speed: everything else is measured against how much slower it is than reaching your own memory.
Tier 2 — inside the server (scale-up). When a GPU needs data another GPU holds, and that GPU is in the same chassis, it uses a direct chip-to-chip link. Nvidia's NVLink gives each H100 900 GB/s to its neighbours (NVIDIA H100 spec; corroborated on NVIDIA's NVLink page, both as of 2026). That is already about 3.7 times slower than reading local HBM (3,350 ÷ 900). It is, notably, seven times faster than the general-purpose PCIe Gen5 bus on the same card, which moves only 128 GB/s — which is exactly why a dedicated GPU-to-GPU link exists at all.
Tier 3 — between servers (scale-out). Once the data a GPU needs sits in a different server, possibly a different rack, it has to cross a network. On Meta's Llama 3 cluster each GPU had 400 Gbps of RoCE or InfiniBand — which is 50 GB/s (Meta, 2024: "Both RoCE and Infiniband clusters leverage 400 Gbps interconnects between GPUs"). Against NVLink's 900 GB/s that is 18 times slower, and against local HBM it is roughly 67 times slower — the full span of the hierarchy is close to two orders of magnitude.
The specific numbers above will be wrong within a hardware generation or two, and that is fine, because the durable content is the shape: a modest few-fold step from memory to intra-node link, then a large cliff — roughly an order of magnitude — at the edge of the server. That cliff at the node boundary is the single most consequential line in a GPU cluster, and everything below follows from crossing it as little as possible.
The operation the whole thing is built around: all-reduce
The reason bandwidth translates so directly into wall-clock time is a collective operation called an all-reduce. After a training step, every GPU has computed a slightly different set of gradient updates; to stay in sync they must sum those gradients and give every GPU the total. That sum-and-broadcast is the all-reduce, and it runs every step.
Its cost is set by the interconnect, not by arithmetic. The standard ring all-reduce arranges the GPUs in a loop and passes partial sums around it, and its communication volume has a clean, durable form: each GPU sends and receives about
2 × (N − 1) / N × M bytes
where M is the size of the tensor being reduced and N is the number of GPUs. The useful thing about that expression is what happens as N grows: (N − 1)/N heads toward 1, so per-GPU traffic converges on 2M and stops growing. Adding GPUs does not make the all-reduce move more data per GPU — it just spreads a fixed amount of work over a slower or faster wire.
Worked example: the same all-reduce on two tiers
Take an 8-billion-parameter model trained in BF16, so gradients are 2 bytes per parameter:
M = 8 × 10⁹ params × 2 bytes = 16 GB of gradients
A ring all-reduce over a large replica group moves about 2M per GPU, so each GPU pushes roughly 32 GB across the interconnect every step. Now time that transfer on the two tiers, using the 2026 figures above:
- On NVLink (900 GB/s):
32 GB ÷ 900 GB/s ≈ 36 ms - On the inter-node fabric (50 GB/s):
32 GB ÷ 50 GB/s ≈ 640 ms
Same bytes, same operation, an 18× difference in wall-clock time — the tier ratio, made concrete. And this is one collective per step. It survives on the slow fabric only because it happens once and can be overlapped with the backward pass: as soon as the last layers finish computing their gradients, the framework starts reducing them while the earlier layers are still busy, so a well-tuned 640 ms hides underneath a step that takes longer than that anyway.
The collective that cannot be hidden is the one that fires inside every layer. Tensor parallelism, which splits a single matrix multiply across GPUs, has to reconcile partial results dozens or hundreds of times per step, with no computation left to hide behind. Run those per-layer collectives at 50 GB/s and the job stalls. That is the placement rule the whole hierarchy forces: pin the high-frequency traffic (tensor parallelism) inside the NVLink domain, and reserve the slow inter-node fabric for the low-frequency, overlappable traffic (the gradient all-reduce of data parallelism). The interconnect does not just influence the parallelism strategy — it dictates it.
Types
The interconnect world divides cleanly along the line the hierarchy already drew — the edge of the server — and the industry has settled names for the two sides.
Scale-up fabrics connect GPUs inside a node or rack into a single tight domain, as if they were one large accelerator. Nvidia's NVLink is the dominant example; its fifth-generation switch wires 72 Blackwell GPUs into one domain at 1.8 TB/s per GPU and 130 TB/s of aggregate bandwidth (NVIDIA NVLink page, as of 2026). The open challenger is UALink, an industry standard finalized in 2025 to give non-Nvidia accelerators an equivalent link. The size of a scale-up domain is a hard architectural limit: it is the largest island in which tensor parallelism is cheap, so growing it from 8 GPUs to 72 changes what fraction of a model can be split the fast way.
Scale-out fabrics connect separate servers across a data center. InfiniBand is the long-standing high-performance option, purpose-built for low-latency collectives; RoCE (RDMA over Converged Ethernet) achieves similar behaviour on Ethernet hardware, which many operators already own — Meta ran Llama 3 on both. The open effort here is Ultra Ethernet, a 2025 specification retuning Ethernet specifically for AI collective traffic. Scale-out fabrics carry an order of magnitude less bandwidth per GPU than scale-up ones, which is precisely why the two tiers exist as separate networks with separate jobs rather than one uniform fabric.
Real-World Applications
Meta's Llama 3 cluster (2024) is the most fully documented example: up to 16,000 H100s, eight per server on an NVLink island, 400 Gbps of RoCE between servers, reaching 41% of peak BF16 utilization (Meta, 2024). The published interruption log and utilization figures are why this cluster, rather than a vendor slide, anchors the numbers on this page.
Nvidia's GB200 NVL72 turns the scale-up idea into a product: an entire rack sold as one NVLink domain of 72 GPUs, liquid-cooled, with 130 TB/s of aggregate NVLink (NVIDIA NVLink page). The pitch is not a faster chip; it is a far larger island in which the expensive kinds of parallelism stay cheap. It also shows the cost of the strategy — a rack that is one interconnect domain is also one thing that can fail as a unit.
Disaggregated inference is a newer application of the same arithmetic. Serving systems increasingly split a model's prefill stage (compute-heavy) onto one pool of GPUs and its decode stage (memory-heavy) onto another — a pattern often called disaggregated serving — which means the KV cache built during prefill has to be shipped over the interconnect to the decode pool. Whether that transfer is affordable is entirely a question of which tier connects the two pools, so techniques that shrink the KV cache, such as grouped-query attention, are in part interconnect-savings measures.
Key Concepts
A few ideas recur whenever the interconnect is the bottleneck:
- Bisection bandwidth — the aggregate bandwidth across the worst-case cut that splits the cluster in half. It is the honest measure of a fabric's capacity for all-to-all traffic, because a network can advertise a large per-GPU number and still choke when everyone communicates at once.
- Overlapping communication with computation — the reason a slow collective is sometimes free. If the interconnect transfer runs concurrently with arithmetic the GPU was going to do anyway, its cost disappears from the critical path. Most of the engineering in a training framework is arranging this overlap.
- Exposed communication — the failure of that overlap. When a collective takes longer than the computation it was meant to hide behind, the excess is "exposed" and the GPUs sit idle. This is what turns a small tier gap into a large slowdown.
Challenges
The hardest problem is structural and gets worse every generation: the slow tiers are improving more slowly than the chips. Each new GPU raises peak FLOPS faster than it raises the per-GPU inter-node bandwidth, so the fraction of a training step spent waiting on the network grows over time even as everything gets nominally faster. A cluster that is compute-bound today can be interconnect-bound two generations later on the same workload, purely because the ratio moved. This is the networking cousin of the memory wall.
Fault domains are the price of tight coupling. A 72-GPU NVLink domain behaves like one accelerator for performance — and, unfortunately, somewhat like one accelerator for failure: a lost link or switch can take a large block of GPUs out of a run at once, and at 16,000-GPU scale interruptions are frequent enough that checkpointing and fast recovery become first-order concerns rather than housekeeping.
Vendor lock-in is the commercial edge of the same coupling. The scale-up tier is where Nvidia's advantage is deepest, because NVLink is proprietary and a large NVLink domain is difficult to replicate — which is exactly what UALink and Ultra Ethernet exist to contest. An operator choosing a fabric today is also choosing how easily it can mix accelerators tomorrow.
Optics cost and power cap how far the fast tiers can reach. Copper links are cheap but short; extending high bandwidth across a rack or between racks means optical transceivers, which draw significant power and add cost that scales with the size of the fabric. Part of why the node boundary is such a sharp cliff is simply that copper stops working at that distance and optics get expensive fast.
Future Trends
The clearest direction is growing the scale-up domain. NVLink islands have already gone from 8 GPUs to 72, and the roadmap points at larger ones; each expansion makes a bigger slice of a model splittable the cheap way and pushes the expensive tier boundary further out. If the domain grows large enough, distinctions that matter today start to blur.
The second is open standards arriving at both tiers at once. UALink for scale-up and Ultra Ethernet for scale-out were both finalized in 2025, aimed squarely at the two layers where proprietary links currently dominate. Whether they erode that dominance depends less on raw specifications than on whether the surrounding software — the collective libraries that actually schedule an all-reduce — reaches parity, which historically has been the harder half.
The third is co-packaged optics: moving the optical engines onto the switch package itself to cut the power and cost of driving high bandwidth over distance. If it matures, it softens the node-boundary cliff by making the slow tier less slow and less power-hungry — which would, for the first time in a while, move the durable tier ratio in a favourable direction rather than the usual one.
Note: this term was last reviewed in July 2026. The specific bandwidths cited — NVLink at 900 GB/s and 1.8 TB/s, HBM at 3.35 TB/s, inter-node links at 400 Gbps, the 72-GPU domain — are stated as of that review and are the parts of this page most likely to age. The tier ratios they illustrate are the durable content and should outlast the figures.