--- source: 'https://howaiworks.ai/glossary/flops' section: glossary title: FLOPs (Floating Point Operations) description: >- The unit AI compute is measured in — how a training run's FLOP budget is counted, why FLOP and FLOPS are not the same, and why regulators use the number. tags: - AI Infrastructure - GPU - Hardware - Scaling - Efficiency - Inference - Performance - Regulation category: Infrastructure & Hardware datePublished: '2026-07-13' lastUpdated: '2026-07-24' --- # FLOPs (Floating Point Operations) > The unit AI compute is measured in — how a training run's FLOP budget is counted, why FLOP and FLOPS are not the same, and why regulators use the number. ## Definition **A FLOP is one arithmetic operation on a decimal number. FLOPS — with the S — is FLOPs *per second*.** One is an amount of work, like kilowatt-hours; the other is a rate, like kilowatts. That single letter separates the two most-confused numbers in AI infrastructure, and once you keep them apart, most compute questions become division. Training a model has a *size*, measured in FLOPs: Meta's Llama 3.1 405B took roughly **3.8×10^25 FLOPs** of arithmetic. A chip has a *speed*, measured in FLOPS — and that speed climbs with every hardware generation. As of 2026 a mainstream current-generation part, NVIDIA's Blackwell **B200**, is rated at about **2,250 teraFLOPS** — 2.25 PFLOP/s, or 2.25×10^15 operations every second — of dense BF16 throughput ([NVIDIA](https://www.nvidia.com/en-us/data-center/dgx-b200/)), roughly 2.3× the **989 teraFLOPS** of the H100 it succeeds. Divide the job by the machine and you get a schedule; multiply the schedule by a price and you get a budget. Nearly every "how much would it cost to train that?" question is this one division, done carefully — and the speed you divide by is the one number here that goes obsolete within a generation, so date it and look it up rather than hard-coding it. Carefully is the operative word: both numbers in that division are routinely misquoted — the workload's FLOP count by people using the wrong parameter count, and the chip's FLOPS rating by the marketing department. ## How It Works ### Counting the work: the 6ND rule A transformer's forward pass costs about **2 FLOPs per parameter per token**: each weight is multiplied by an activation (one operation) and the result is added to a running sum (a second). The backward pass, which computes gradients with respect to both the activations and the weights, costs about twice as much again. Add them up and you get roughly **6 FLOPs per parameter per token** — the estimate introduced in the original scaling-law work and used ever since: ``` C ≈ 6 × N × D C = total training FLOPs N = parameters D = training tokens ``` Where all that arithmetic physically lands is matrix multiplication — [matmul](https://howaiworks.ai/glossary/matrix-multiplication) is where well over 90% of these FLOPs are spent, and that page owns the details. What the compute budget *buys* you in capability is the subject of [scaling laws](https://howaiworks.ai/glossary/scaling-laws). This page is about the counting. ### Worked example: Llama 3.1 405B, from parameters to days Meta trained Llama 3.1 405B on 15.6 trillion tokens. Take those two published numbers and turn the crank: ``` N = 4.05 × 10^11 parameters D = 1.56 × 10^13 tokens C = 6 × 4.05e11 × 1.56e13 = 3.79 × 10^25 FLOPs ``` Meta reports **3.8×10^25** for that run. The rule reproduces the published figure to two significant figures, which is why it is worth trusting on models whose compute was never disclosed. Now convert work into time — using the H100s Meta actually ran this job on, not a newer chip, since that is what the invoice was paid in. An H100 SXM is rated at 989 teraFLOPS dense BF16 — but that is the *peak*, the number the silicon hits only if every tensor core is fed every cycle, which no real training job achieves. Meta reported a sustained **41% MFU** on its 16,384-GPU configuration, so each GPU actually delivers: ``` 9.89e14 FLOP/s × 0.41 = 4.06 × 10^14 FLOP/s sustained GPU-seconds = 3.79e25 / 4.06e14 = 9.34 × 10^10 s GPU-hours = 9.34e10 / 3600 ≈ 26 million GPU-hours wall-clock = 26e6 / 16,384 GPUs ≈ 1,585 hours ≈ 66 days ``` Meta's model card reports **30.84 million H100-hours** for the 405B. Our 26 million is 19% low, and that gap is the honest one: 6ND counts only the arithmetic the model requires, while the invoice also pays for failed runs, checkpoint restarts, evaluation and post-training. An estimate that lands within 20% of a nine-figure hardware bill, from two numbers on a model card, is doing real work. ### Why the marketed number is not the number you divide by Now do the same division with the figure on the marketing slide. The H100's headline BF16 rating is **1,979 teraFLOPS** — twice the 989 above, because it assumes **2:4 structured sparsity**, a mode in which half the weights are zero and the hardware skips them. Almost no production training run uses it: ``` 3.79e25 / 1.979e15 FLOP/s = 1.92e10 s ≈ 5.3 million GPU-hours ``` **5.3 million versus the 30.84 million actually spent.** Believing the datasheet would have under-budgeted this run by nearly a factor of six — the wrong cluster, the wrong quarter, the wrong funding round. The same inflation runs through rack-scale claims: NVIDIA's headline "1.4 exaFLOPS" for a GB200 NVL72 is an FP4 sparse figure; the dense FP4 number is 720 petaFLOPS, and a BF16 training job sees a small fraction of even that. Two rules follow. **A FLOPS rating is meaningless without its precision and its sparsity assumption** — FP4, FP8, BF16 and FP64 ratings on the same chip differ by more than an order of magnitude, and a FLOP count is only comparable to another FLOP count at the same numeric format (see [quantization](https://howaiworks.ai/glossary/quantization)). And **peak is not sustained.** ### MFU: the number that says what you are actually getting **Model FLOPs Utilization** is the fraction of peak that a run converts into useful work — observed throughput divided by what the hardware would do at its rated speed. It is the honesty metric of a GPU cluster, and it is unforgiving: ``` MFU = (6 × N × tokens/second) / (peak FLOPS per GPU × number of GPUs) ``` Meta reported **38–43% BF16 MFU** across Llama 3's pre-training, dropping from 43% at 8,192 GPUs to 41% at 16,384 as communication overhead grew. Google reported **46.2%** for PaLM 540B on TPU v4, which was a headline result at the time. So a *very good* large-scale run wastes more than half of the FLOPS it is paying for — to gradient synchronisation across nodes, to [parallelism](https://howaiworks.ai/glossary/parallel-processing) that leaves some devices waiting, to memory traffic, and to the parts of a transformer that are not big matmuls. MFU has a sibling, **HFU** (hardware FLOPs utilization), which counts every operation the chip executed — including activation recomputation, work that is done twice to save memory. HFU is always the higher number, which is why it is the one people quote when the MFU is embarrassing. ### And often, FLOPs are not the constraint at all Generating tokens one at a time is usually limited by how fast weights can be pulled out of memory, not by how fast the tensor cores can multiply them — so a GPU can sit at 5% MFU while running as fast as it physically can. That is the [memory wall](https://howaiworks.ai/glossary/memory-wall), and it is that page's subject, not this one's. ## Real-World Applications **Budgeting a training run before you buy the cluster.** The chain above — 6ND to FLOPs, FLOPs over (peak × MFU) to GPU-hours, GPU-hours over GPU count to weeks — is how a run gets sized and how the board paper justifying it gets written. Get the MFU assumption wrong by 2× and the schedule slips by months. **Regulation.** Compute thresholds are now law in two jurisdictions, and both are FLOP counts. The EU AI Act (Article 51) presumes a general-purpose model has **systemic risk** once its cumulative training compute exceeds **10^25 FLOP**, triggering evaluation, red-teaming and incident-reporting duties; a separate indicative criterion treats **10^23 FLOP** as the floor for being a general-purpose model at all. California's SB 53, signed in September 2025, defines a **frontier model** as one trained with more than **10^26** integer or floating-point operations — and, notably, counts subsequent fine-tuning and reinforcement learning toward that total. The United States had a **10^26** federal reporting threshold in Executive Order 14110 (October 2023); it was revoked in January 2025. Regulators reached for FLOPs for a reason. A capability benchmark can be gamed, contaminated, or disputed, and it can only be measured after the model exists. Training compute can be estimated *before* the run, forecast from a purchase order, and audited afterwards from cluster logs and electricity bills. It is the only property of a frontier model that is both roughly proportional to capability and physically hard to hide. Whether the line is in the right place is another question — Epoch AI counts **33 models** publicly known to have been trained at or above 10^25 FLOP, with GPT-4 estimated at **2.1×10^25**. **Accepting a cluster.** MFU on a reference workload is the standard acceptance test for rented GPUs: compute achieved FLOPs over rated FLOPs, and find out whether the interconnect you were sold actually exists. A cluster that benchmarks at 15% MFU on a job that should hit 40% is not cheap — it is 2.7× more expensive than the price list says. ## Key Concepts - **Inference costs ~2N FLOPs per token, not 6N** — there is no backward pass. For a 405B model that is roughly 810 GFLOPs per generated token. Dividing the training budget by that, **3.79e25 / 8.1e11 ≈ 47 trillion tokens** must be generated before serving the model has cost as much arithmetic as training it — about three times its entire training corpus. For a popular model, inference passes training; for most, it never does. - **For [Mixture-of-Experts](https://howaiworks.ai/glossary/mixture-of-experts), N means *active* parameters.** DeepSeek-V3 has 671B total parameters but activates 37B per token, so its 14.8T-token run cost about 6 × 3.7e10 × 1.48e13 ≈ **3.3×10^24 FLOPs**. Plug the headline 671B in instead and you get 6.0×10^25 — an 18× overestimate that would place the model on the wrong side of the EU's 10^25 line. - **6ND ignores attention**, whose cost grows with the square of sequence length. For typical pre-training this is a few percent and safely inside the rule's error bars; at very long [context](https://howaiworks.ai/glossary/context-window) it is not, and the approximation quietly stops holding. ## Challenges **The unit is stable; every number attached to it rots.** A FLOP is a physical count and will mean the same thing in ten years. Peak FLOPS ratings, precision formats and MFU records change every hardware generation: dense BF16 throughput jumped from the H100's 989 teraFLOPS to the B200's ~2,250 in a single step, and the headline precision has narrowed from BF16 through FP8 to FP4 over the same span. A budget that hard-codes a chip's throughput is wrong within a year while the arithmetic around it stays right. Write the formula down; look the rate up. **A threshold measures input, not capability.** The efficiency with which FLOPs are converted into intelligence improves every year, so a fixed 10^25 line steadily catches *less capable* models over time while the frontier moves past it in other ways: reasoning models spend enormous compute at inference, which no training-FLOP threshold counts, and [distillation](https://howaiworks.ai/glossary/knowledge-distillation) can compress a threshold-crossing model's behaviour into a small one that legally never crossed it. SB 53's decision to count fine-tuning and RL toward the total is an attempt to close part of this gap; it does not close all of it. **High MFU and fast training are not the same goal.** It is entirely possible to raise MFU by choosing a parallelism strategy that does more redundant arithmetic, or by picking a model shape that suits the hardware rather than the task. The metric measures how busy the tensor cores are, not how much learning happened — and optimising it directly is how teams end up with a beautifully utilised cluster training a worse model. ## Future Trends Marketed FLOPS numbers will keep inflating faster than real throughput, because each hardware generation introduces a narrower numeric format — FP16 gave way to FP8, FP8 to FP4 — and the headline is always quoted in the narrowest one, with sparsity assumed. Expect "exaflops" to appear in more press releases and to mean progressively less. The defensible comparison is drifting toward *tokens per second per dollar on your workload*, which is a measurement rather than a rating. The harder shift is regulatory. Compute thresholds were drawn when capability was essentially a function of pre-training compute, and that link is loosening: test-time compute, post-training and distillation all decouple what a model can do from what it cost to pre-train. A line that a 671B-parameter model walks under while a distilled 8B model inherits frontier behaviour is a line under pressure. The likely direction is compound triggers — compute *plus* capability evaluations *plus* deployment scale — which are harder to game and much harder to audit. FLOPs will stay in the statute regardless, because nothing else about a model is as countable. ## Frequently Asked Questions ### What is the difference between FLOP and FLOPS? A FLOP is one floating-point operation — a count of work done. FLOPS is floating-point operations per second — a rate of work. Training Llama 3.1 405B took about 3.8×10^25 FLOPs of work; the H100 GPUs it ran on do up to 989 teraFLOPS (9.89×10^14 FLOP/s) of dense BF16 work each, while a current-generation Blackwell B200 does about 2,250 teraFLOPS as of 2026. One is the size of the job, the other is the speed of the machine — and the speed keeps rising each hardware generation — so dividing the first by the second is how you get a schedule. ### How do you estimate the compute cost of training a model? Use the 6ND rule: total training FLOPs ≈ 6 × parameters × training tokens. The 6 comes from roughly 2 FLOPs per parameter per token in the forward pass and about twice that in the backward pass. For a 405B-parameter model trained on 15.6T tokens, that gives 6 × 4.05×10^11 × 1.56×10^13 ≈ 3.8×10^25 FLOPs — which is what Meta reported. ### What is MFU (Model FLOPs Utilization)? MFU is the fraction of a chip's peak FLOPS that a training run actually converts into useful model FLOPs. Large runs land in the 35–50% range: Meta reported 38–43% for Llama 3, and Google reported 46.2% for PaLM 540B. A cluster running below that is losing money to communication stalls, memory bandwidth, or a bad parallelism configuration. ### Why do regulators use a FLOP threshold to define a frontier model? Because training compute is one of very few properties of a model that can be estimated before it is built and audited afterwards from hardware invoices and cluster logs. The EU AI Act presumes systemic risk above 10^25 FLOP; California's SB 53 defines a frontier model at more than 10^26 operations. It is a proxy for capability, not a measure of it. ### Does the 6ND rule work for Mixture-of-Experts models? Only if you use active parameters, not total ones. DeepSeek-V3 has 671B total parameters but activates 37B per token, so its training compute is about 6 × 3.7×10^10 × 1.48×10^13 ≈ 3.3×10^24 FLOPs — not the 6×10^25 you get from the headline parameter count. That is an 18× error, and it straddles a regulatory threshold. ### Are FLOPs the main bottleneck in AI workloads? Often not. Generating tokens one at a time is usually limited by how fast weights can be read from memory rather than by arithmetic throughput, which is why a GPU can report low utilization while running flat out. ## Related ### Related terms - [GPU Computing](https://howaiworks.ai/glossary/gpu-computing) - [Model Size](https://howaiworks.ai/glossary/model-size) - [Training](https://howaiworks.ai/glossary/training) - [Inference](https://howaiworks.ai/glossary/inference) - [Parallel Processing](https://howaiworks.ai/glossary/parallel-processing) - [Performance](https://howaiworks.ai/glossary/performance) --- Source: https://howaiworks.ai/glossary/flops — HowAIWorks.ai