Definition
A FLOP is one arithmetic operation on a decimal number. FLOPS — with the S — is FLOPs per second. One is an amount of work, like kilowatt-hours; the other is a rate, like kilowatts. That single letter separates the two most-confused numbers in AI infrastructure, and once you keep them apart, most compute questions become division.
Training a model has a size, measured in FLOPs: Meta's Llama 3.1 405B took roughly 3.8×10^25 FLOPs of arithmetic. A chip has a speed, measured in FLOPS: an NVIDIA H100 is rated at 989 teraFLOPS of dense BF16 throughput, meaning about 9.89×10^14 operations every second. Divide the job by the machine and you get a schedule; multiply the schedule by a price and you get a budget. Nearly every "how much would it cost to train that?" question is this one division, done carefully.
Carefully is the operative word: both numbers in that division are routinely misquoted — the workload's FLOP count by people using the wrong parameter count, and the chip's FLOPS rating by the marketing department.
How It Works
Counting the work: the 6ND rule
A transformer's forward pass costs about 2 FLOPs per parameter per token: each weight is multiplied by an activation (one operation) and the result is added to a running sum (a second). The backward pass, which computes gradients with respect to both the activations and the weights, costs about twice as much again. Add them up and you get roughly 6 FLOPs per parameter per token — the estimate introduced in the original scaling-law work and used ever since:
C ≈ 6 × N × D
C = total training FLOPs
N = parameters
D = training tokens
Where all that arithmetic physically lands is matrix multiplication — matmul is where well over 90% of these FLOPs are spent, and that page owns the details. What the compute budget buys you in capability is the subject of scaling laws. This page is about the counting.
Worked example: Llama 3.1 405B, from parameters to days
Meta trained Llama 3.1 405B on 15.6 trillion tokens. Take those two published numbers and turn the crank:
N = 4.05 × 10^11 parameters
D = 1.56 × 10^13 tokens
C = 6 × 4.05e11 × 1.56e13
= 3.79 × 10^25 FLOPs
Meta reports 3.8×10^25 for that run. The rule reproduces the published figure to two significant figures, which is why it is worth trusting on models whose compute was never disclosed.
Now convert work into time. An H100 SXM is rated at 989 teraFLOPS dense BF16 — but that is the peak, the number the silicon hits only if every tensor core is fed every cycle, which no real training job achieves. Meta reported a sustained 41% MFU on its 16,384-GPU configuration, so each GPU actually delivers:
9.89e14 FLOP/s × 0.41 = 4.06 × 10^14 FLOP/s sustained
GPU-seconds = 3.79e25 / 4.06e14 = 9.34 × 10^10 s
GPU-hours = 9.34e10 / 3600 ≈ 26 million GPU-hours
wall-clock = 26e6 / 16,384 GPUs ≈ 1,585 hours ≈ 66 days
Meta's model card reports 30.84 million H100-hours for the 405B. Our 26 million is 19% low, and that gap is the honest one: 6ND counts only the arithmetic the model requires, while the invoice also pays for failed runs, checkpoint restarts, evaluation and post-training. An estimate that lands within 20% of a nine-figure hardware bill, from two numbers on a model card, is doing real work.
Why the marketed number is not the number you divide by
Now do the same division with the figure on the marketing slide. The H100's headline BF16 rating is 1,979 teraFLOPS — twice the 989 above, because it assumes 2:4 structured sparsity, a mode in which half the weights are zero and the hardware skips them. Almost no production training run uses it:
3.79e25 / 1.979e15 FLOP/s = 1.92e10 s ≈ 5.3 million GPU-hours
5.3 million versus the 30.84 million actually spent. Believing the datasheet would have under-budgeted this run by nearly a factor of six — the wrong cluster, the wrong quarter, the wrong funding round. The same inflation runs through rack-scale claims: NVIDIA's headline "1.4 exaFLOPS" for a GB200 NVL72 is an FP4 sparse figure; the dense FP4 number is 720 petaFLOPS, and a BF16 training job sees a small fraction of even that.
Two rules follow. A FLOPS rating is meaningless without its precision and its sparsity assumption — FP4, FP8, BF16 and FP64 ratings on the same chip differ by more than an order of magnitude, and a FLOP count is only comparable to another FLOP count at the same numeric format (see quantization). And peak is not sustained.
MFU: the number that says what you are actually getting
Model FLOPs Utilization is the fraction of peak that a run converts into useful work — observed throughput divided by what the hardware would do at its rated speed. It is the honesty metric of a GPU cluster, and it is unforgiving:
MFU = (6 × N × tokens/second) / (peak FLOPS per GPU × number of GPUs)
Meta reported 38–43% BF16 MFU across Llama 3's pre-training, dropping from 43% at 8,192 GPUs to 41% at 16,384 as communication overhead grew. Google reported 46.2% for PaLM 540B on TPU v4, which was a headline result at the time. So a very good large-scale run wastes more than half of the FLOPS it is paying for — to gradient synchronisation across nodes, to parallelism that leaves some devices waiting, to memory traffic, and to the parts of a transformer that are not big matmuls.
MFU has a sibling, HFU (hardware FLOPs utilization), which counts every operation the chip executed — including activation recomputation, work that is done twice to save memory. HFU is always the higher number, which is why it is the one people quote when the MFU is embarrassing.
And often, FLOPs are not the constraint at all
Generating tokens one at a time is usually limited by how fast weights can be pulled out of memory, not by how fast the tensor cores can multiply them — so a GPU can sit at 5% MFU while running as fast as it physically can. That is the memory wall, and it is that page's subject, not this one's.
Real-World Applications
Budgeting a training run before you buy the cluster. The chain above — 6ND to FLOPs, FLOPs over (peak × MFU) to GPU-hours, GPU-hours over GPU count to weeks — is how a run gets sized and how the board paper justifying it gets written. Get the MFU assumption wrong by 2× and the schedule slips by months.
Regulation. Compute thresholds are now law in two jurisdictions, and both are FLOP counts. The EU AI Act (Article 51) presumes a general-purpose model has systemic risk once its cumulative training compute exceeds 10^25 FLOP, triggering evaluation, red-teaming and incident-reporting duties; a separate indicative criterion treats 10^23 FLOP as the floor for being a general-purpose model at all. California's SB 53, signed in September 2025, defines a frontier model as one trained with more than 10^26 integer or floating-point operations — and, notably, counts subsequent fine-tuning and reinforcement learning toward that total. The United States had a 10^26 federal reporting threshold in Executive Order 14110 (October 2023); it was revoked in January 2025.
Regulators reached for FLOPs for a reason. A capability benchmark can be gamed, contaminated, or disputed, and it can only be measured after the model exists. Training compute can be estimated before the run, forecast from a purchase order, and audited afterwards from cluster logs and electricity bills. It is the only property of a frontier model that is both roughly proportional to capability and physically hard to hide. Whether the line is in the right place is another question — Epoch AI counts 33 models publicly known to have been trained at or above 10^25 FLOP, with GPT-4 estimated at 2.1×10^25.
Accepting a cluster. MFU on a reference workload is the standard acceptance test for rented GPUs: compute achieved FLOPs over rated FLOPs, and find out whether the interconnect you were sold actually exists. A cluster that benchmarks at 15% MFU on a job that should hit 40% is not cheap — it is 2.7× more expensive than the price list says.
Key Concepts
- Inference costs ~2N FLOPs per token, not 6N — there is no backward pass. For a 405B model that is roughly 810 GFLOPs per generated token. Dividing the training budget by that, 3.79e25 / 8.1e11 ≈ 47 trillion tokens must be generated before serving the model has cost as much arithmetic as training it — about three times its entire training corpus. For a popular model, inference passes training; for most, it never does.
- For Mixture-of-Experts, N means active parameters. DeepSeek-V3 has 671B total parameters but activates 37B per token, so its 14.8T-token run cost about 6 × 3.7e10 × 1.48e13 ≈ 3.3×10^24 FLOPs. Plug the headline 671B in instead and you get 6.0×10^25 — an 18× overestimate that would place the model on the wrong side of the EU's 10^25 line.
- 6ND ignores attention, whose cost grows with the square of sequence length. For typical pre-training this is a few percent and safely inside the rule's error bars; at very long context it is not, and the approximation quietly stops holding.
Challenges
The unit is stable; every number attached to it rots. A FLOP is a physical count and will mean the same thing in ten years. Peak FLOPS ratings, precision formats and MFU records change every hardware generation, so a budget that hard-codes a chip's throughput is wrong within a year while the arithmetic around it stays right. Write the formula down; look the rate up.
A threshold measures input, not capability. The efficiency with which FLOPs are converted into intelligence improves every year, so a fixed 10^25 line steadily catches less capable models over time while the frontier moves past it in other ways: reasoning models spend enormous compute at inference, which no training-FLOP threshold counts, and distillation can compress a threshold-crossing model's behaviour into a small one that legally never crossed it. SB 53's decision to count fine-tuning and RL toward the total is an attempt to close part of this gap; it does not close all of it.
High MFU and fast training are not the same goal. It is entirely possible to raise MFU by choosing a parallelism strategy that does more redundant arithmetic, or by picking a model shape that suits the hardware rather than the task. The metric measures how busy the tensor cores are, not how much learning happened — and optimising it directly is how teams end up with a beautifully utilised cluster training a worse model.
Future Trends
Marketed FLOPS numbers will keep inflating faster than real throughput, because each hardware generation introduces a narrower numeric format — FP16 gave way to FP8, FP8 to FP4 — and the headline is always quoted in the narrowest one, with sparsity assumed. Expect "exaflops" to appear in more press releases and to mean progressively less. The defensible comparison is drifting toward tokens per second per dollar on your workload, which is a measurement rather than a rating.
The harder shift is regulatory. Compute thresholds were drawn when capability was essentially a function of pre-training compute, and that link is loosening: test-time compute, post-training and distillation all decouple what a model can do from what it cost to pre-train. A line that a 671B-parameter model walks under while a distilled 8B model inherits frontier behaviour is a line under pressure. The likely direction is compound triggers — compute plus capability evaluations plus deployment scale — which are harder to game and much harder to audit. FLOPs will stay in the statute regardless, because nothing else about a model is as countable.