--- source: 'https://howaiworks.ai/glossary/nvidia-gpu-ai' section: glossary title: NVIDIA GPU for AI description: >- Why AI training runs on NVIDIA rather than cheaper silicon — tensor cores, NVLink, and the software that decides how much of the datasheet you actually get. tags: - NVIDIA - GPU - CUDA - AI Hardware - AI Infrastructure - GPU Computing - Hardware - Scaling category: Infrastructure & Hardware datePublished: '2025-09-23' lastUpdated: '2026-07-24' --- # NVIDIA GPU for AI > Why AI training runs on NVIDIA rather than cheaper silicon — tensor cores, NVLink, and the software that decides how much of the datasheet you actually get. ## Definition **AI training runs on NVIDIA hardware, and the reason is mostly not the hardware.** Google trains its frontier models on its own [TPUs](https://howaiworks.ai/glossary/tensor-processing-unit) — the Gemini technical report says plainly that "we trained Gemini models using TPUv5e and TPUv4" — which settles the question of whether the silicon is reproducible. It is. What has proved far harder to reproduce is everything wrapped around it: the kernel libraries, the collectives, the interconnect, the rack, and the fact that every framework's fast path was written against NVIDIA's software first. Here is that claim as a number. NVIDIA's [H100 SXM spec sheet](https://www.nvidia.com/en-us/data-center/h100/) advertises **1,979 teraFLOPS** of BF16 tensor throughput. The footnote on that column reads "with sparsity", so the figure you can count on is **989**. Meta then ran Llama 3.1 405B on a cluster of these and sustained **41%** of it. You are sold one number, the silicon guarantees a second, and you receive a third that is **roughly a fifth of the headline**. Nothing about the die changed between those three figures. What determines where in that range you land is software and interconnect. The H100 (2023) is the worked example throughout this page because it is the best-documented training generation on record. The parts you would actually buy in 2026 are NVIDIA's Blackwell — the [B200](https://www.nvidia.com/en-us/data-center/dgx-b200/) and the Blackwell Ultra GB300 — and AMD's [Instinct](https://www.amd.com/en/products/accelerators/instinct.html) MI300X and MI355X. The absolute teraFLOPS and the gigabytes move every generation; the three-way gap between marketed, dense and achieved does not. Read the ratios below rather than the numbers, and redo any absolute against today's datasheet — the shape is unchanged. "NVIDIA GPU for AI" names the data-centre part: a large die with tensor cores, [HBM](https://howaiworks.ai/glossary/high-bandwidth-memory) stacked beside it in the package, and NVLink ports, sold not as a chip but as an eight-GPU board, a liquid-cooled rack, and a software stack. Low-power inference silicon in a phone or laptop is the [NPU](https://howaiworks.ai/glossary/neural-processing-unit)'s subject. This page is the machine that trains the model, and the question it answers is what would have to change for that machine to come from somebody else. ## How It Works ### The workload is one operation, and the chip is shaped around it More than 99% of a transformer's arithmetic is [matrix multiplication](https://howaiworks.ai/glossary/matrix-multiplication) — that page counts a Llama 3 layer operation by operation and finds the softmax, normalisation and activation functions together come to less than a fifth of one percent. Everything below follows from that census. A machine that is fast at one operation and mediocre at everything else is an excellent AI chip and a poor general computer, and NVIDIA has been moving die area in that direction for a decade. ### What a tensor core does that an ordinary lane does not An ordinary GPU lane — a "CUDA core" — performs one fused multiply-add per clock: read two numbers from registers, multiply, add to an accumulator. That is **2 floating-point operations per input value**, and a warp of 32 threads issuing that instruction together produces **64 FLOPs**. A tensor core consumes a whole tile per instruction. NVIDIA's [PTX ISA documentation](https://docs.nvidia.com/cuda/parallel-thread-execution/) defines the matrix multiply-accumulate operation as `D = A × B + C`, with shapes written `MxNxK` where A is M×K, B is K×N and C and D are M×N, and it is "performed collectively by all threads in the warp". One of the documented FP16 shapes is `m16n8k16`. Price it with the 2·M·N·K rule: ``` one mma.m16n8k16 instruction, issued once by a 32-thread warp arithmetic 2 × 16 × 8 × 16 = 4,096 FLOPs operands (16 × 16) + (16 × 8) = 384 input values intensity 4,096 / 384 = 10.7 FLOPs per value loaded the same warp doing scalar multiply-adds arithmetic 32 threads × 2 = 64 FLOPs intensity = 1 FLOP per value loaded ``` **Sixty-four times the arithmetic from one instruction, and ten times more work per operand fetched.** That second ratio is the durable half. Instruction issue, register-file bandwidth and scheduling are fixed overheads per instruction; amortising them across a tile instead of across a single multiply is what a matrix engine *is*. Widen the tile or narrow the number format and the ratio improves again, which is exactly what every hardware generation has done. The consequence is visible on the same spec sheet: the dense BF16 figure above runs through the tensor cores, while ordinary FP32 vector throughput on the identical chip is **67 teraFLOPS**. Work you cannot express as a matrix multiply runs on a small fraction of the chip you bought. None of this, however, is a moat. Google's TPU is the same idea taken further, and every credible AI accelerator has a matrix engine. **The tensor core is table stakes.** The interesting question is why two chips with comparable matrix engines produce such different fractions of their rated speed. ### Reading the spec sheet without being sold to That same spec column runs from 67 teraFLOPS of FP32 up to **3,958** of FP8 — a spread of **fifty-nine times** across one chip, every entry truthfully "the H100's TFLOPS". Three things have to travel with any such figure or it means nothing: The **precision**. FP8 and FP32 are two ends of that spread, and a FLOP count is only comparable to another FLOP count in the same format. The **sparsity assumption**. Every tensor-core row on that table carries an asterisk reading "with sparsity" — 2:4 structured sparsity, in which half the weights are zero and the hardware skips them. Production training runs generally do not use it, so the dense figure is half the printed one. The **trap in the middle**. Dense BF16 is 989.5 teraFLOPS. Sparse TF32 is 989 teraFLOPS. The same three digits appear twice on NVIDIA's own table meaning entirely different things, so "the H100 does 989 TFLOPS" is ambiguous before anyone has even started exaggerating. [FLOPs](https://howaiworks.ai/glossary/flops) owns the counting rules in detail; the point here is that the vendor's number is a ceiling under stated assumptions, and the assumptions are the content. ### What one training run actually costs This is the calculation that turns a spec sheet into a budget, and it works from entirely public numbers. A transformer costs about 6 FLOPs per parameter per training token once the backward pass is counted — the **6ND** rule that [large language models](https://howaiworks.ai/glossary/large-language-model) and [FLOPs](https://howaiworks.ai/glossary/flops) both derive. Meta published both inputs for its largest Llama 3 run — parameters and training tokens — and reported the utilisation it achieved: ``` work 6 × 4.05e11 params × 1.56e13 tokens = 3.79e25 FLOPs rate 989e12 FLOP/s (dense BF16) × 0.41 MFU = 4.05e14 FLOP/s per GPU effort 3.79e25 / 4.05e14 / 3600 = 26.0 million GPU-hours schedule 26.0e6 / 16,384 GPUs = 1,585 hours = 66 days bill 26.0e6 GPU-hours × $3.99/GPU-hour ≈ $104 million ``` The hourly rate is Lambda's published on-demand list price for an eight-GPU H100 SXM instance at the time of publication; committed multi-year capacity is bought for considerably less, so treat the dollar figure as an order of magnitude rather than a quote. **The assumption that actually moves the answer is the 41%.** State it out loud whenever you do this calculation, because everything else in the chain is arithmetic and that one number is an empirical claim about your cluster. Watch what happens if you take the marketed sparse figure instead. Dividing the same work by it gives **5.3 million GPU-hours** and a bill near **$21 million** — under a fifth of the real effort. That is not a rounding error; it is the wrong cluster, the wrong quarter and the wrong funding round. And the error runs the other way too: Meta's model card reports **30.84 million H100-hours** actually spent, about a fifth above the 6ND estimate, because the invoice also pays for failed runs, restarts, evaluation and post-training. The estimate is a floor. The lesson for this page is in which term did the damage. Peak-to-dense is a marketing convention. **Dense-to-achieved is engineering** — kernels, collectives, parallelism strategy, the network — and it is worth a factor of two and a half in money. That factor is what NVIDIA's software stack is selling, and it does not appear anywhere on the spec sheet. ### Why the interconnect, not the chip, decides scaling Mixed-precision Adam training costs roughly 16 bytes per parameter, so a 405-billion-parameter model carries about **6.5 TB** of state. Set that against a single accelerator's memory and the generation barely changes the verdict: **80 GB** on a 2023 H100, roughly **180 GB** on a 2024 Blackwell B200 (NVIDIA ships the eight-GPU HGX B200 with **1.4 TB** of HBM3e in total), **288 GB** on a 2025 Blackwell Ultra GB300 or an AMD Instinct MI355X. Each generation roughly doubles the number and none of them holds the model, which does not fit on one GPU, or on eighty. [Distributed training](https://howaiworks.ai/glossary/distributed-training) owns how it is cut into data, tensor and pipeline parallelism; what matters here is that one of those cuts is far greedier than the others. Tensor parallelism splits a single matrix multiply, so the GPUs must exchange activations *inside every layer*, several times per layer, with nothing to hide the traffic behind. It can only live where the wires are fast. NVIDIA's product answer is NVLink, and its own spec tables give the two comparisons that matter. An H100 (2023) has **900 GB/s** of NVLink against **128 GB/s** of PCIe Gen5 on the same card — seven times; Blackwell's fifth-generation NVLink lifts the per-GPU figure to **1.8 TB/s** (2024) without touching that ratio, which is the durable half. More consequentially, [NVLink Switch](https://www.nvidia.com/en-us/data-center/nvlink/) grew the domain it can span: the fourth generation connects **8 GPUs** at **7.2 TB/s** aggregate, the fifth connects **72** at **130 TB/s**. Nine times the GPUs in one island, eighteen times the bandwidth inside it. Read that as a capability rather than a speed. A bigger NVLink domain does not make any individual GPU faster. It makes a much larger slice of the model splittable the greedy way before the traffic has to cross the slow fabric between racks — which is precisely the mechanism that raises achieved utilisation, the term that just decided the bill above. **This is the part a merchant chip vendor cannot ship by shipping a chip.** HBM comes from the same three qualified suppliers everyone uses; leading-edge wafers come from the same foundry. A switch ASIC, a cabled backplane, a liquid-cooled rack, a collectives library and tens of thousands of these deployed and debugged in customer data centres do not. ### The software layer, in one paragraph Almost nobody writes [CUDA](https://howaiworks.ai/glossary/cuda), and that page explains why the accumulated kernel stack rather than the language is the hard part. Two things belong here. First, the utilisation figure above was manufactured by cuBLAS and cuDNN kernels and by **NCCL**, the collectives library that performs the gradient all-reduce; none of it came from the die. Second, new techniques arrive as CUDA kernels first and are ported elsewhere later, if at all — so a competitor's stack is not merely behind, it is behind on a moving target. ## Real-World Applications **Meta's Llama 3 cluster is the best-documented NVIDIA training deployment in existence.** The 16,384 H100s in the calculation above sat eight to a server on an NVLink island, with **400 Gbps** of RoCE per GPU between servers — a hierarchy whose fast tier and slow tier differ by more than an order of magnitude. Meta published the utilisation, the interruption log and the GPU-hours, which is why the [Llama](https://howaiworks.ai/models/llama) family is the standard worked example for cluster arithmetic and why every estimate on this page can be checked against something real. **xAI's Colossus in Memphis put 100,000 H100s on a single RDMA fabric** — the word doing the work being *single*, since a fabric across which one job can synchronise is a different and far more expensive object than a hundred thousand chips in a warehouse. What it took to energise the building is the [AI data center](https://howaiworks.ai/glossary/ai-data-center)'s subject, and it is the clearest demonstration that the chips arrive long before the electricity does. **Google's Gemini is the counter-example, and it is the most informative fact on this page.** The technical report states that Gemini was trained on TPUv5e and TPUv4, deployed in "SuperPods of 4096 chips", each connected to an optical switch that can reconfigure the topology in around ten seconds. A frontier model family, trained without an NVIDIA GPU. It is proof the moat is crossable — and, in what it took, proof of how wide it is: a decade of in-house silicon, a compiler team, a proprietary interconnect and a collectives stack, for one company's workloads on one company's cloud. **AWS built the second existence proof.** Project Rainier came online with, in Amazon's words, "nearly half a million Trainium2 chips", with Anthropic training and serving Claude on it. Again the buyer had to become a builder: custom silicon, custom interconnect, a custom compiler and toolchain. **And the aggregate shows up in a filing.** NVIDIA's Q1 fiscal 2027 results, for the quarter ended 26 April 2026, report revenue of **$81.6 billion** with **$75.2 billion** of it from Data Center, at a GAAP gross margin of **74.9%**. Read the margin rather than the revenue. Commodity hardware does not sustain three-quarters gross margin; that number is what a moat looks like when an accountant writes it down, and it is also precisely the incentive every large customer has to go and build its own. ## Key Concepts - **Three TFLOPS numbers, and only the third is yours.** Marketed (sparsity assumed), dense, and achieved — each roughly half the one before it on a very well-run frontier training job. A capacity plan built on the first is wrong by about five times. - **Scale-up and scale-out are different networks with different jobs.** NVLink inside the rack carries the parallelism that talks inside every layer; InfiniBand or RoCE between racks carries the parallelism that talks once per step. Sizing the second correctly does not rescue a job whose tensor parallelism has spilled out of the first. - **You are buying a cluster, not an accelerator.** The line items that decide whether the GPUs work together — switches, optics, cold plates, the collectives library, the scheduler — are most of the engineering and none of the datasheet. - **Bandwidth and capacity answer different questions.** Capacity decides what fits; bandwidth decides how fast it runs. The [memory wall](https://howaiworks.ai/glossary/memory-wall) explains why a chip's FLOPS figure does not appear anywhere in a token-rate calculation. ## Challenges **Provisioning on peak TFLOPS.** This is the most expensive mistake in the field and it is made with a spreadsheet. Take the biggest number in the vendor's table, divide your workload by it, and you will under-budget by roughly five times on training and by two or three *orders of magnitude* on single-stream inference, where the chip is waiting on memory rather than computing. The defence is mechanical: write the precision and the sparsity assumption next to every FLOPS figure you use, and make achieved utilisation on a reference workload an acceptance test for rented capacity rather than something you discover in month three. **The memory cliff is binary, it moves your whole architecture, and the edge itself moves every generation.** Llama 3.1 70B has 70.6 billion parameters; in BF16 that is **141 GB** of weights. On a 2023 H100's **80 GB** that does not fit, so you need at least two GPUs, tensor parallelism, and therefore an NVLink domain. On a 2024 Blackwell B200 (roughly 180 GB) or an AMD Instinct MI300X (**192 GB**) the same weights now fit on one GPU with room to spare — the cliff did not disappear, it moved, and a larger model meets it again at the new capacity. [Quantized](https://howaiworks.ai/glossary/quantization) to FP8 the same model is **70.6 GB**, which fits even on the H100, with about 9 GB left over for the [KV cache](https://howaiworks.ai/glossary/kv-cache) and activations. One decision about number format — and one about which generation you buy — changes the machine you need, the interconnect you need and the price. This arithmetic is worth doing before the purchase order, not after. **A chip being available is not a cluster being available.** Three separate queues sit between an allocation and a working machine, and none of them is the GPU. Memory: [HBM](https://howaiworks.ai/glossary/high-bandwidth-memory) must be co-packaged with the logic die, only three suppliers are qualified, and [advanced packaging](https://howaiworks.ai/glossary/advanced-packaging) capacity has been the binding constraint on accelerator output. Power: an [AI data center](https://howaiworks.ai/glossary/ai-data-center) is planned in megawatts, and the substation behind it can take longer to arrive than the hardware inside it will last. Networking: switches, optics and a non-blocking fabric have their own lead times, and a cluster whose interconnect underdelivers is not a cheap cluster — it is a slow one you pay full price for. **Lock-in is claimed at the framework level and lost at the kernel level.** "It's just PyTorch" is true right up to the point where your inference server depends on a fused attention kernel, your training code calls a CUDA-specific operator, your throughput assumptions were measured against NCCL, and your checkpoint tooling knows about a particular device mesh. Nothing in that list appears in a portability audit, and all of it has to be rebuilt and re-benchmarked to move. ## Future Trends **The credible competition is not a faster chip; it is a full stack, and two of them already exist.** Google's TPU and AWS's Trainium are the working proofs, and what they demonstrate is the size of the entry fee: silicon, a compiler, an interconnect, a collectives library and years of workload-specific tuning. Both were built by firms large enough to be their own first customer. That is the shape the next entrant will have too — a buyer becoming a builder — rather than a merchant vendor selling a better die. **AMD's gap is narrowing on the common path and structurally open at the frontier.** On the die and its memory AMD is already competitive on the metric that decides what fits: the Instinct MI300X shipped with **192 GB** of HBM, and the 2025 MI355X (CDNA4) reaches **288 GB** of HBM3e — matching or beating the Blackwell generation on memory per accelerator. ROCm's HIP layer translates CUDA source, and PyTorch ships ROCm builds that run most code unmodified; the [CUDA](https://howaiworks.ai/glossary/cuda) page works through why a compatibility layer chasing a moving API is always second. The second front is the one this page is about: a merchant accelerator can match the die and the memory, but an equivalent to a rack-scale coherent domain is not a driver release. The industry's collective answer is an open standard — the [UALink](https://ualinkconsortium.org/) consortium's accelerator-to-accelerator interconnect — which by construction arrives after the thing it is answering, and which still has to be built into racks somebody ships. **Watch NVIDIA sell pieces of its own moat.** Its NVLink page now advertises **NVLink Fusion**, which pairs NVIDIA interconnect and rack-scale architecture with "semi-custom ASICs or CPUs" so hyperscalers can build hybrid infrastructure. That is the company monetising the hardest-to-copy layer for customers who are shipping non-NVIDIA accelerators — a rational response to those customers building anyway, and a quiet admission about which layer is actually scarce. **The binding constraint has moved off the die entirely.** HBM supply, advanced packaging throughput and grid interconnection now gate deployment more tightly than logic fabrication does. That reshapes the competitive question: an accelerator that is somewhat slower but can be manufactured, packaged, powered and cooled in volume this year beats one that is faster on a slide. Expect the arguments that matter to keep drifting away from teraFLOPS and toward **tokens per second per dollar per watt, measured on your own workload** — which is not a rating anyone can print, and is therefore the one number a vendor cannot inflate. ## Frequently Asked Questions ### Why does AI training run on NVIDIA GPUs specifically? Mostly not because the chips are faster. A matrix engine is table stakes — every serious AI accelerator has one. What NVIDIA sells that competitors do not is the whole machine: tuned kernel libraries, a collectives library, an NVLink domain that lets 72 GPUs behave as one, and racks you can actually buy in volume. The chip is the smallest part of the purchase. ### What does a tensor core actually do differently? An ordinary GPU lane does one multiply-add per clock on two numbers. A tensor core takes a whole tile per instruction: NVIDIA's PTX shape m16n8k16 multiplies a 16×16 matrix by a 16×8 one, which is 4,096 floating-point operations from a single instruction issued by one warp, against 64 for the same warp doing scalar multiply-adds. ### Is an H100 really 1,979 teraFLOPS? Only with an asterisk that NVIDIA prints. The 1,979 BF16 figure on the H100 SXM spec sheet assumes 2:4 structured sparsity; the dense number is 989. A large real training run then sustains around 40% of that. Budget on the marketed figure and you will be wrong by roughly five times in both money and months. ### How much does it cost to train a frontier model on NVIDIA hardware? For Llama 3.1 405B the arithmetic is public: 6ND gives 3.79×10²⁵ FLOPs, and at 989 dense BF16 teraFLOPS times Meta's reported 41% utilisation that is about 26 million GPU-hours, or 66 days on 16,384 H100s. At a published on-demand rate near $4 per GPU-hour that is on the order of $100 million, and Meta's own invoice was higher. ### Could AMD, Google or Amazon replace NVIDIA? Google already has, for itself — Gemini was trained on TPUs, not on NVIDIA hardware — and Anthropic runs Claude on AWS Trainium. Both took years of in-house compiler, interconnect and collectives work for one company's workloads. That is the price of leaving, and it is why it has been paid by firms large enough to build a stack rather than buy one. ### Does a bigger NVLink domain matter more than a faster chip? For training, often yes. Tensor parallelism exchanges activations inside every layer, so it can only live inside the fast domain. Going from an 8-GPU NVLink island to a 72-GPU one does not make any GPU quicker; it makes a much larger slice of the model splittable the expensive way, which is what raises the fraction of peak you actually reach. ## Related ### Related terms - [CUDA](https://howaiworks.ai/glossary/cuda) - [GPU Computing](https://howaiworks.ai/glossary/gpu-computing) - [Matrix Multiplication (GEMM)](https://howaiworks.ai/glossary/matrix-multiplication) - [Distributed Training](https://howaiworks.ai/glossary/distributed-training) - [High Bandwidth Memory (HBM)](https://howaiworks.ai/glossary/high-bandwidth-memory) - [Tensor Processing Unit (TPU)](https://howaiworks.ai/glossary/tensor-processing-unit) --- Source: https://howaiworks.ai/glossary/nvidia-gpu-ai — HowAIWorks.ai