Introduction
There is a habit in AI commentary of treating training and inference as two phases of one business: first you build the model, then you sell it. That framing survives contact with almost no part of the machine.
Training and serving have opposite bottlenecks — one waits on the network, the other waits on memory. They have opposite cost structures — one is a capital expense that ends, the other an operating expense that grows with every user you win. And they scale with different things — one with your ambition, the other with your popularity. A chip optimised for one is not the right chip for the other, a capacity plan built from one will not size the other, and an argument about the hardware market that only counts the first will get the second exactly backwards.
That last mistake was made in public in 2024, at scale, and it is worth understanding why.
This piece is the economic half of the story that From Token to Transistor tells physically. The mechanisms are linked; the argument is the arithmetic that connects them.
Two Machines With Opposite Bottlenecks
Training waits on the network
Meta trained Llama 3.1 405B on 16,384 H100 GPUs. Those GPUs were not each doing their own thing. Gradient descent is sequential — step n+1 needs the weights produced by step n — so the only thing you can parallelise is the work inside a step, and every step ends with a synchronisation barrier where all 16,384 devices must agree on the new weights.
Agreement moves bytes, and this is where the ceiling is. An H100 talks to the seven GPUs in its own server over NVLink at 900 GB/s; it reaches a GPU in another rack at 400 Gbps, which is 50 GB/s — roughly 18 times slower. That single ratio dictates the entire architecture of a training run, and distributed training is the craft of arranging three kinds of parallelism so that each communication pattern lands on a network fast enough to carry it. Meta reported a sustained 41% MFU on that cluster, which is a good number: more than half the FLOPS being paid for went to synchronisation, stalls and idle devices.
Training is compute-hungry and network-limited. Nothing on this page is about memory bandwidth.
Serving waits on memory
Now generate a token. The model must multiply your vector by every weight matrix it has, which means it must read every weight out of memory — and then it does about two arithmetic operations per byte it read.
Two FLOPs per byte. An H100 needs roughly 590 FLOPs per byte to keep its tensor cores fed (1,979 dense FP8 TFLOPS divided by 3.35 TB/s of bandwidth). Decoding is not slightly below that line; it is about 295 times below it. The arithmetic units run at 0.34% of capacity while the chatbot "generates fast". This is the memory wall, and it is the entire reason the second machine exists.
So: training is thousands of chips held at a barrier by a wire. Serving is one chip, mostly idle, waiting on its own memory. Calling both of these "AI compute" and adding them up is how the reasoning goes wrong.
The hardware admits it
You do not have to take the theory on faith, because the vendors have stopped hedging.
NVIDIA's H100 and H200 have identical FP8 tensor-core throughput — 3,958 TFLOPS. The H200 simply has faster, larger memory: 4.8 TB/s and 141 GB against 3.35 TB/s and 80 GB. Same arithmetic, 1.43× the bandwidth — and NVIDIA advertises up to 1.9× the Llama 2 70B inference performance. Zero extra FLOPS bought that. On a training run, the same upgrade would be worth far less.
In September 2025 NVIDIA went further and announced Rubin CPX, a GPU carrying 128 GB of GDDR7 rather than HBM and 30 petaFLOPS of NVFP4 — deliberately trading bandwidth for compute and capacity, because prompt processing is compute-bound even though token generation is not. And in April 2025 Google introduced its seventh-generation TPU under the headline "the first Google TPU for the age of inference". When a vendor builds a separate chip for a workload, it has conceded that the workload is separate.
One Cost Is a Purchase. The Other Is a Subscription You Sell.
Meta's model card puts the 405B run at 30.84 million H100-hours. That number is finished. It does not change if ten people download Llama or ten billion do. Training cost scales with ambition — with how big a model you decided to build — and with nothing else.
Inference cost scales with usage. Every user, every query, every day, forever. Which means the cheapest model in the world to serve is one that nobody wants, and the most expensive thing that can happen to an AI company is that its product works.
That is not a figure of speech. When OpenAI shipped native image generation in March 2025 and it went viral, Sam Altman wrote that the company's "GPUs are melting" and imposed rate limits — three free generations a day. No model had failed. No training run had overrun. Demand was the incident.
The growth rates say the same thing more coldly. Google reported at I/O in May 2026 that it was processing over 3.2 quadrillion tokens per month across its products, up roughly 7× in a year from about 480 trillion. That is not a curve driven by anyone's training budget. It is driven by people using things.
The Crossover: 47 Trillion Tokens
Here is the number the whole argument turns on, and it is checkable in three lines.
FLOPs gives the two rules. Training costs about 6 FLOPs per parameter per token (forward pass plus backward pass). Inference costs about 2N — there is no backward pass. So for Llama 3.1 405B:
training = 3.8e25 FLOPs (Meta's published figure)
per token = 2 x 4.05e11 = 8.1e11 FLOPs
crossover = 3.8e25 / 8.1e11 = 4.7e13 tokens
About 47 trillion tokens. Three times the model's entire 15.6-trillion-token training corpus must pass through it — as prompts read or answers written — before serving it has cost as much arithmetic as building it.
That sounds enormous. Now price it in real traffic. At DevDay in October 2025, Altman said OpenAI was processing over 6 billion tokens per minute on the API alone — 8.6 trillion a day. If those tokens were served at a 405B-scale cost per token, OpenAI's API would burn an entire frontier pre-training run's worth of compute every 5.4 days. Run the same counterfactual against Google's 3.2 quadrillion tokens a month and the crossover arrives in about eleven hours.
Or count it per person. Artificial Analysis, measuring at the time of publication, puts one hard reasoning task at roughly 43,000 output tokens. Divide:
4.7e13 / 4.3e4 = 1.1 billion reasoning tasks
ChatGPT had 800 million weekly active users in October 2025. One hard reasoning question each, and you have spent a frontier pre-training run.
Those last figures are counterfactuals — nobody serves all their traffic on a 405B dense model, and the mix of model sizes is not public. Read them as scale, not as accounting. The scale is the point: the crossover is not a distant milestone that a successful model might reach one day. It is a weekly event, and then it happens again.
Why the 2024 Chip-Demand Argument Was Backwards
In December 2024, Ilya Sutskever told NeurIPS that "pre-training as we know it will end" — compute keeps growing, data does not, "because we have but one internet." The inference many drew from that was straightforward: if pre-training saturates, demand for accelerators falls.
The premise was largely right. The conclusion was backwards, and not because pre-training secretly kept scaling. It was backwards because it counted only the first economy.
What happened instead is that the compute moved. Reasoning models spend their budget while answering rather than while training: the model writes a long internal chain of thought, you are billed for every hidden token, and the mechanism belongs to test-time compute, which derives it properly. Three months after Sutskever's talk, NVIDIA's Jensen Huang told GTC 2025 that reasoning and agentic AI needed "easily 100 times more" compute than the company had assumed a year earlier, decomposing it as roughly ten times more tokens, generated ten times faster to stay responsive.
Look at what that did to the ledger. A pre-training run is a capital expense that ends. Test-time compute is an operating expense that compounds with usage — every user, every query, forever. The 2024 argument watched a capex line flatten and concluded the industry was done buying chips, at the exact moment the demand was migrating to the one line that grows with success. Deloitte's 2026 predictions put inference at roughly two-thirds of all AI compute in 2026, up from a third in 2023 and half in 2025.
The lesson is not that a forecast was wrong. It is that you cannot reason about the hardware market from the training economy alone, because the training economy is the one that has a ceiling.
Training Buys Capability. Inference Buys Margin.
The two machines also buy different things, and this is where the money actually is.
Training buys a capability, once. You either got the model you wanted or you did not. Efficiency work here buys you a faster run — a better parallelism plan, a higher MFU, a cheaper next model. It does not touch the price of your product.
Inference buys a margin, repeatedly — and every unit of efficiency converts directly into money, because it lowers the cost of a good you sell over and over.
The levers are well understood and they are all attacks on the memory wall. Inference optimization is mostly batching: read the weights once, use them for many users at the same time. On one eight-GPU H100 node running Llama 3.1 70B in FP8, that single dial moves the cost of a million tokens from about $23.80 at batch size 1 to about 51 cents at batch 384 — a 47× swing, with no change to the model, the code or the hardware. Quantization halves the bytes and therefore roughly halves the read. Speculative decoding has a small model guess five tokens and the big model verify all of them for about 1.4% more time, because the big model was going to read all its weights anyway.
What that adds up to, DeepSeek published. For the 24 hours to 28 February 2025 it disclosed its serving meter: an average of 226.75 H800 nodes, $87,072 of compute per day at an assumed $2 per GPU-hour, against $562,027 per day of tokens at list price — a stated 545% cost profit margin, which DeepSeek itself notes is theoretical, since much of that traffic was discounted or free. Even discounted heavily, that is the shape of the second economy: a gross margin that is, in the end, a scheduler setting.
What Breaks
You reason about the hardware market from training alone. You rank accelerators by FLOPS, conclude that a plateau in pre-training means a plateau in demand, and miss that the growth had moved to a workload where FLOPS is nearly irrelevant and bandwidth is the product. The FLOPS ranking and the tokens-per-second ranking are not the same list — the H100 and H200 have identical FLOPS.
You plan capacity from your training budget. This is the one that kills companies. The board paper prices the run: so many GPU-hours, so many weeks, a number with an end. Then the product works. Users arrive, and each one attaches a permanent, recurring compute cost to your P&L that appears in no line of that paper. Teams discover after launch that their gross margin is negative and that the fix is not an engineering breakthrough but a batch size — and that raising it makes the product feel slower to the users who are the reason they are in trouble.
A model nobody uses is cheap to serve. That is the whole warning, and it is not a comforting one.
Conclusion
Two machines. Training: thousands of accelerators, a barrier every step, a network that sets the ceiling, a bill that arrives once and is then over. Serving: one chip at a time, arithmetic units idle, waiting on memory, a bill that arrives every day and grows with your success.
They are not two phases of one business. They are two businesses that happen to share a supplier, and the industry has spent the last two years moving its centre of gravity from the first to the second — not because pre-training stopped working, but because scaling laws were never the only curve, and the other one is paid per query.
If you only remember one number, remember 47 trillion. It is what a frontier training run costs in served tokens — and for anything people actually use, it is a number you pass, and then pass again.
Sources
- Meta, Llama 3.1 405B model card — 30.84M H100-hours, 15.6T training tokens, 3.8×10²⁵ FLOPs. See also Llama.
- The Llama 3 Herd of Models — 16,384 H100s, NVLink 900 GB/s vs 400 Gbps RoCE, 41% MFU.
- NVIDIA H100 and NVIDIA H200 — identical 3,958 FP8 TFLOPS; 3.35 TB/s vs 4.8 TB/s; up to 1.9× Llama 2 70B inference.
- NVIDIA unveils Rubin CPX (9 September 2025) — 128 GB GDDR7, 30 PFLOPS NVFP4, built for context processing.
- Ironwood: The first Google TPU for the age of inference (April 2025).
- Sundar Pichai, Google I/O 2026 keynote — over 3.2 quadrillion tokens/month, ~7× year on year.
- Sam Altman at DevDay, 6 October 2025 — 800M weekly ChatGPT users; over 6 billion tokens per minute on the API.
- Sam Altman, 27 March 2025 — "our GPUs are melting"; image-generation rate limits.
- Jensen Huang at GTC 2025 — "easily 100× more" compute for reasoning and agentic AI.
- Deloitte, TMT Predictions 2026 — inference ≈ two-thirds of all compute in 2026.
- DeepSeek V3/R1 inference system overview (1 March 2025) — 226.75 H800 nodes, $87,072/day cost, $562,027/day theoretical revenue, 545% stated margin.