Definition
AI training runs on NVIDIA hardware, and the reason is mostly not the hardware. Google trains its frontier models on its own TPUs — the Gemini technical report says plainly that "we trained Gemini models using TPUv5e and TPUv4" — which settles the question of whether the silicon is reproducible. It is. What has proved far harder to reproduce is everything wrapped around it: the kernel libraries, the collectives, the interconnect, the rack, and the fact that every framework's fast path was written against NVIDIA's software first.
Here is that claim as a number. NVIDIA's H100 SXM spec sheet advertises 1,979 teraFLOPS of BF16 tensor throughput. The footnote on that column reads "with sparsity", so the figure you can count on is 989. Meta then ran Llama 3.1 405B on a cluster of these and sustained 41% of it. You are sold one number, the silicon guarantees a second, and you receive a third that is roughly a fifth of the headline. Nothing about the die changed between those three figures. What determines where in that range you land is software and interconnect.
The H100 (2023) is the worked example throughout this page because it is the best-documented training generation on record. The parts you would actually buy in 2026 are NVIDIA's Blackwell — the B200 and the Blackwell Ultra GB300 — and AMD's Instinct MI300X and MI355X. The absolute teraFLOPS and the gigabytes move every generation; the three-way gap between marketed, dense and achieved does not. Read the ratios below rather than the numbers, and redo any absolute against today's datasheet — the shape is unchanged.
"NVIDIA GPU for AI" names the data-centre part: a large die with tensor cores, HBM stacked beside it in the package, and NVLink ports, sold not as a chip but as an eight-GPU board, a liquid-cooled rack, and a software stack. Low-power inference silicon in a phone or laptop is the NPU's subject. This page is the machine that trains the model, and the question it answers is what would have to change for that machine to come from somebody else.
How It Works
The workload is one operation, and the chip is shaped around it
More than 99% of a transformer's arithmetic is matrix multiplication — that page counts a Llama 3 layer operation by operation and finds the softmax, normalisation and activation functions together come to less than a fifth of one percent. Everything below follows from that census. A machine that is fast at one operation and mediocre at everything else is an excellent AI chip and a poor general computer, and NVIDIA has been moving die area in that direction for a decade.
What a tensor core does that an ordinary lane does not
An ordinary GPU lane — a "CUDA core" — performs one fused multiply-add per clock: read two numbers from registers, multiply, add to an accumulator. That is 2 floating-point operations per input value, and a warp of 32 threads issuing that instruction together produces 64 FLOPs.
A tensor core consumes a whole tile per instruction. NVIDIA's PTX ISA
documentation defines the matrix
multiply-accumulate operation as D = A × B + C, with shapes written MxNxK where A is M×K, B is
K×N and C and D are M×N, and it is "performed collectively by all threads in the warp". One of the
documented FP16 shapes is m16n8k16. Price it with the 2·M·N·K rule:
one mma.m16n8k16 instruction, issued once by a 32-thread warp
arithmetic 2 × 16 × 8 × 16 = 4,096 FLOPs
operands (16 × 16) + (16 × 8) = 384 input values
intensity 4,096 / 384 = 10.7 FLOPs per value loaded
the same warp doing scalar multiply-adds
arithmetic 32 threads × 2 = 64 FLOPs
intensity = 1 FLOP per value loaded
Sixty-four times the arithmetic from one instruction, and ten times more work per operand fetched. That second ratio is the durable half. Instruction issue, register-file bandwidth and scheduling are fixed overheads per instruction; amortising them across a tile instead of across a single multiply is what a matrix engine is. Widen the tile or narrow the number format and the ratio improves again, which is exactly what every hardware generation has done.
The consequence is visible on the same spec sheet: the dense BF16 figure above runs through the tensor cores, while ordinary FP32 vector throughput on the identical chip is 67 teraFLOPS. Work you cannot express as a matrix multiply runs on a small fraction of the chip you bought.
None of this, however, is a moat. Google's TPU is the same idea taken further, and every credible AI accelerator has a matrix engine. The tensor core is table stakes. The interesting question is why two chips with comparable matrix engines produce such different fractions of their rated speed.
Reading the spec sheet without being sold to
That same spec column runs from 67 teraFLOPS of FP32 up to 3,958 of FP8 — a spread of fifty-nine times across one chip, every entry truthfully "the H100's TFLOPS". Three things have to travel with any such figure or it means nothing:
The precision. FP8 and FP32 are two ends of that spread, and a FLOP count is only comparable to another FLOP count in the same format.
The sparsity assumption. Every tensor-core row on that table carries an asterisk reading "with sparsity" — 2:4 structured sparsity, in which half the weights are zero and the hardware skips them. Production training runs generally do not use it, so the dense figure is half the printed one.
The trap in the middle. Dense BF16 is 989.5 teraFLOPS. Sparse TF32 is 989 teraFLOPS. The same three digits appear twice on NVIDIA's own table meaning entirely different things, so "the H100 does 989 TFLOPS" is ambiguous before anyone has even started exaggerating. FLOPs owns the counting rules in detail; the point here is that the vendor's number is a ceiling under stated assumptions, and the assumptions are the content.
What one training run actually costs
This is the calculation that turns a spec sheet into a budget, and it works from entirely public numbers. A transformer costs about 6 FLOPs per parameter per training token once the backward pass is counted — the 6ND rule that large language models and FLOPs both derive. Meta published both inputs for its largest Llama 3 run — parameters and training tokens — and reported the utilisation it achieved:
work 6 × 4.05e11 params × 1.56e13 tokens = 3.79e25 FLOPs
rate 989e12 FLOP/s (dense BF16) × 0.41 MFU = 4.05e14 FLOP/s per GPU
effort 3.79e25 / 4.05e14 / 3600 = 26.0 million GPU-hours
schedule 26.0e6 / 16,384 GPUs = 1,585 hours = 66 days
bill 26.0e6 GPU-hours × $3.99/GPU-hour ≈ $104 million
The hourly rate is Lambda's published on-demand list price for an eight-GPU H100 SXM instance at the time of publication; committed multi-year capacity is bought for considerably less, so treat the dollar figure as an order of magnitude rather than a quote. The assumption that actually moves the answer is the 41%. State it out loud whenever you do this calculation, because everything else in the chain is arithmetic and that one number is an empirical claim about your cluster.
Watch what happens if you take the marketed sparse figure instead. Dividing the same work by it gives 5.3 million GPU-hours and a bill near $21 million — under a fifth of the real effort. That is not a rounding error; it is the wrong cluster, the wrong quarter and the wrong funding round. And the error runs the other way too: Meta's model card reports 30.84 million H100-hours actually spent, about a fifth above the 6ND estimate, because the invoice also pays for failed runs, restarts, evaluation and post-training. The estimate is a floor.
The lesson for this page is in which term did the damage. Peak-to-dense is a marketing convention. Dense-to-achieved is engineering — kernels, collectives, parallelism strategy, the network — and it is worth a factor of two and a half in money. That factor is what NVIDIA's software stack is selling, and it does not appear anywhere on the spec sheet.
Why the interconnect, not the chip, decides scaling
Mixed-precision Adam training costs roughly 16 bytes per parameter, so a 405-billion-parameter model carries about 6.5 TB of state. Set that against a single accelerator's memory and the generation barely changes the verdict: 80 GB on a 2023 H100, roughly 180 GB on a 2024 Blackwell B200 (NVIDIA ships the eight-GPU HGX B200 with 1.4 TB of HBM3e in total), 288 GB on a 2025 Blackwell Ultra GB300 or an AMD Instinct MI355X. Each generation roughly doubles the number and none of them holds the model, which does not fit on one GPU, or on eighty. Distributed training owns how it is cut into data, tensor and pipeline parallelism; what matters here is that one of those cuts is far greedier than the others. Tensor parallelism splits a single matrix multiply, so the GPUs must exchange activations inside every layer, several times per layer, with nothing to hide the traffic behind. It can only live where the wires are fast.
NVIDIA's product answer is NVLink, and its own spec tables give the two comparisons that matter. An H100 (2023) has 900 GB/s of NVLink against 128 GB/s of PCIe Gen5 on the same card — seven times; Blackwell's fifth-generation NVLink lifts the per-GPU figure to 1.8 TB/s (2024) without touching that ratio, which is the durable half. More consequentially, NVLink Switch grew the domain it can span: the fourth generation connects 8 GPUs at 7.2 TB/s aggregate, the fifth connects 72 at 130 TB/s. Nine times the GPUs in one island, eighteen times the bandwidth inside it.
Read that as a capability rather than a speed. A bigger NVLink domain does not make any individual GPU faster. It makes a much larger slice of the model splittable the greedy way before the traffic has to cross the slow fabric between racks — which is precisely the mechanism that raises achieved utilisation, the term that just decided the bill above. This is the part a merchant chip vendor cannot ship by shipping a chip. HBM comes from the same three qualified suppliers everyone uses; leading-edge wafers come from the same foundry. A switch ASIC, a cabled backplane, a liquid-cooled rack, a collectives library and tens of thousands of these deployed and debugged in customer data centres do not.
The software layer, in one paragraph
Almost nobody writes CUDA, and that page explains why the accumulated kernel stack rather than the language is the hard part. Two things belong here. First, the utilisation figure above was manufactured by cuBLAS and cuDNN kernels and by NCCL, the collectives library that performs the gradient all-reduce; none of it came from the die. Second, new techniques arrive as CUDA kernels first and are ported elsewhere later, if at all — so a competitor's stack is not merely behind, it is behind on a moving target.
Real-World Applications
Meta's Llama 3 cluster is the best-documented NVIDIA training deployment in existence. The 16,384 H100s in the calculation above sat eight to a server on an NVLink island, with 400 Gbps of RoCE per GPU between servers — a hierarchy whose fast tier and slow tier differ by more than an order of magnitude. Meta published the utilisation, the interruption log and the GPU-hours, which is why the Llama family is the standard worked example for cluster arithmetic and why every estimate on this page can be checked against something real.
xAI's Colossus in Memphis put 100,000 H100s on a single RDMA fabric — the word doing the work being single, since a fabric across which one job can synchronise is a different and far more expensive object than a hundred thousand chips in a warehouse. What it took to energise the building is the AI data center's subject, and it is the clearest demonstration that the chips arrive long before the electricity does.
Google's Gemini is the counter-example, and it is the most informative fact on this page. The technical report states that Gemini was trained on TPUv5e and TPUv4, deployed in "SuperPods of 4096 chips", each connected to an optical switch that can reconfigure the topology in around ten seconds. A frontier model family, trained without an NVIDIA GPU. It is proof the moat is crossable — and, in what it took, proof of how wide it is: a decade of in-house silicon, a compiler team, a proprietary interconnect and a collectives stack, for one company's workloads on one company's cloud.
AWS built the second existence proof. Project Rainier came online with, in Amazon's words, "nearly half a million Trainium2 chips", with Anthropic training and serving Claude on it. Again the buyer had to become a builder: custom silicon, custom interconnect, a custom compiler and toolchain.
And the aggregate shows up in a filing. NVIDIA's Q1 fiscal 2027 results, for the quarter ended 26 April 2026, report revenue of $81.6 billion with $75.2 billion of it from Data Center, at a GAAP gross margin of 74.9%. Read the margin rather than the revenue. Commodity hardware does not sustain three-quarters gross margin; that number is what a moat looks like when an accountant writes it down, and it is also precisely the incentive every large customer has to go and build its own.
Key Concepts
- Three TFLOPS numbers, and only the third is yours. Marketed (sparsity assumed), dense, and achieved — each roughly half the one before it on a very well-run frontier training job. A capacity plan built on the first is wrong by about five times.
- Scale-up and scale-out are different networks with different jobs. NVLink inside the rack carries the parallelism that talks inside every layer; InfiniBand or RoCE between racks carries the parallelism that talks once per step. Sizing the second correctly does not rescue a job whose tensor parallelism has spilled out of the first.
- You are buying a cluster, not an accelerator. The line items that decide whether the GPUs work together — switches, optics, cold plates, the collectives library, the scheduler — are most of the engineering and none of the datasheet.
- Bandwidth and capacity answer different questions. Capacity decides what fits; bandwidth decides how fast it runs. The memory wall explains why a chip's FLOPS figure does not appear anywhere in a token-rate calculation.
Challenges
Provisioning on peak TFLOPS. This is the most expensive mistake in the field and it is made with a spreadsheet. Take the biggest number in the vendor's table, divide your workload by it, and you will under-budget by roughly five times on training and by two or three orders of magnitude on single-stream inference, where the chip is waiting on memory rather than computing. The defence is mechanical: write the precision and the sparsity assumption next to every FLOPS figure you use, and make achieved utilisation on a reference workload an acceptance test for rented capacity rather than something you discover in month three.
The memory cliff is binary, it moves your whole architecture, and the edge itself moves every generation. Llama 3.1 70B has 70.6 billion parameters; in BF16 that is 141 GB of weights. On a 2023 H100's 80 GB that does not fit, so you need at least two GPUs, tensor parallelism, and therefore an NVLink domain. On a 2024 Blackwell B200 (roughly 180 GB) or an AMD Instinct MI300X (192 GB) the same weights now fit on one GPU with room to spare — the cliff did not disappear, it moved, and a larger model meets it again at the new capacity. Quantized to FP8 the same model is 70.6 GB, which fits even on the H100, with about 9 GB left over for the KV cache and activations. One decision about number format — and one about which generation you buy — changes the machine you need, the interconnect you need and the price. This arithmetic is worth doing before the purchase order, not after.
A chip being available is not a cluster being available. Three separate queues sit between an allocation and a working machine, and none of them is the GPU. Memory: HBM must be co-packaged with the logic die, only three suppliers are qualified, and advanced packaging capacity has been the binding constraint on accelerator output. Power: an AI data center is planned in megawatts, and the substation behind it can take longer to arrive than the hardware inside it will last. Networking: switches, optics and a non-blocking fabric have their own lead times, and a cluster whose interconnect underdelivers is not a cheap cluster — it is a slow one you pay full price for.
Lock-in is claimed at the framework level and lost at the kernel level. "It's just PyTorch" is true right up to the point where your inference server depends on a fused attention kernel, your training code calls a CUDA-specific operator, your throughput assumptions were measured against NCCL, and your checkpoint tooling knows about a particular device mesh. Nothing in that list appears in a portability audit, and all of it has to be rebuilt and re-benchmarked to move.
Future Trends
The credible competition is not a faster chip; it is a full stack, and two of them already exist. Google's TPU and AWS's Trainium are the working proofs, and what they demonstrate is the size of the entry fee: silicon, a compiler, an interconnect, a collectives library and years of workload-specific tuning. Both were built by firms large enough to be their own first customer. That is the shape the next entrant will have too — a buyer becoming a builder — rather than a merchant vendor selling a better die.
AMD's gap is narrowing on the common path and structurally open at the frontier. On the die and its memory AMD is already competitive on the metric that decides what fits: the Instinct MI300X shipped with 192 GB of HBM, and the 2025 MI355X (CDNA4) reaches 288 GB of HBM3e — matching or beating the Blackwell generation on memory per accelerator. ROCm's HIP layer translates CUDA source, and PyTorch ships ROCm builds that run most code unmodified; the CUDA page works through why a compatibility layer chasing a moving API is always second. The second front is the one this page is about: a merchant accelerator can match the die and the memory, but an equivalent to a rack-scale coherent domain is not a driver release. The industry's collective answer is an open standard — the UALink consortium's accelerator-to-accelerator interconnect — which by construction arrives after the thing it is answering, and which still has to be built into racks somebody ships.
Watch NVIDIA sell pieces of its own moat. Its NVLink page now advertises NVLink Fusion, which pairs NVIDIA interconnect and rack-scale architecture with "semi-custom ASICs or CPUs" so hyperscalers can build hybrid infrastructure. That is the company monetising the hardest-to-copy layer for customers who are shipping non-NVIDIA accelerators — a rational response to those customers building anyway, and a quiet admission about which layer is actually scarce.
The binding constraint has moved off the die entirely. HBM supply, advanced packaging throughput and grid interconnection now gate deployment more tightly than logic fabrication does. That reshapes the competitive question: an accelerator that is somewhat slower but can be manufactured, packaged, powered and cooled in volume this year beats one that is faster on a slide. Expect the arguments that matter to keep drifting away from teraFLOPS and toward tokens per second per dollar per watt, measured on your own workload — which is not a rating anyone can print, and is therefore the one number a vendor cannot inflate.