--- source: 'https://howaiworks.ai/blog/training-vs-inference-two-economies' section: blog title: 'Training vs Inference: Two Economies, Two Opposite Bottlenecks' description: >- Training is network-bound and paid once. Serving is memory-bound and paid forever. The crossover point, and why the 2024 chip-demand call was backwards. date: '2026-07-13' author: HowAIWorks Team tags: - AI Infrastructure - Inference - Model Training - Scaling Laws - AI Hardware - GPU - Efficiency - Reasoning - LLM category: AI Fundamentals readingTime: 9 minutes isReviewed: false --- # Training vs Inference: Two Economies, Two Opposite Bottlenecks > Training is network-bound and paid once. Serving is memory-bound and paid forever. The crossover point, and why the 2024 chip-demand call was backwards. ## Introduction There is a habit in AI commentary of treating training and inference as two phases of one business: first you build the model, then you sell it. That framing survives contact with almost no part of the machine. Training and serving have **opposite bottlenecks** — one waits on the network, the other waits on memory. They have **opposite cost structures** — one is a capital expense that ends, the other an operating expense that grows with every user you win. And they **scale with different things** — one with your ambition, the other with your popularity. A chip optimised for one is not the right chip for the other, a capacity plan built from one will not size the other, and an argument about the hardware market that only counts the first will get the second exactly backwards. That last mistake was made in public in 2024, at scale, and it is worth understanding why. This piece is the economic half of the story that [From Token to Transistor](https://howaiworks.ai/blog/how-ai-runs-on-silicon) tells physically. The mechanisms are linked; the argument is the arithmetic that connects them. ## Two Machines With Opposite Bottlenecks ### Training waits on the network Meta trained Llama 3.1 405B on **16,384 H100 GPUs**. Those GPUs were not each doing their own thing. Gradient descent is sequential — step *n+1* needs the weights produced by step *n* — so the only thing you can parallelise is the work *inside* a step, and every step ends with a synchronisation barrier where all 16,384 devices must agree on the new weights. Agreement moves bytes, and this is where the ceiling is. An H100 talks to the seven GPUs in its own server over NVLink at **900 GB/s**; it reaches a GPU in another rack at **400 Gbps**, which is **50 GB/s** — roughly **18 times slower**. That single ratio dictates the entire architecture of a training run, and [distributed training](https://howaiworks.ai/glossary/distributed-training) is the craft of arranging three kinds of parallelism so that each communication pattern lands on a network fast enough to carry it. Meta reported a sustained **41% MFU** on that cluster, which is a *good* number: more than half the FLOPS being paid for went to synchronisation, stalls and idle devices. Training is compute-hungry and network-limited. Nothing on this page is about memory bandwidth. ### Serving waits on memory Now generate a token. The model must multiply your vector by every weight matrix it has, which means it must **read every weight out of memory** — and then it does about two arithmetic operations per byte it read. Two FLOPs per byte. An H100 needs roughly **590 FLOPs per byte** to keep its tensor cores fed (1,979 dense FP8 TFLOPS divided by 3.35 TB/s of bandwidth). Decoding is not slightly below that line; it is about **295 times** below it. The arithmetic units run at **0.34% of capacity** while the chatbot "generates fast". This is the [memory wall](https://howaiworks.ai/glossary/memory-wall), and it is the entire reason the second machine exists. So: training is thousands of chips held at a barrier by a wire. Serving is one chip, mostly idle, waiting on its own memory. Calling both of these "AI compute" and adding them up is how the reasoning goes wrong. ### The hardware admits it You do not have to take the theory on faith, because the vendors have stopped hedging. NVIDIA's H100 and H200 have **identical FP8 tensor-core throughput — 3,958 TFLOPS**. The H200 simply has faster, larger memory: **4.8 TB/s and 141 GB** against **3.35 TB/s and 80 GB**. Same arithmetic, 1.43× the bandwidth — and NVIDIA advertises **up to 1.9× the Llama 2 70B inference performance**. Zero extra FLOPS bought that. On a training run, the same upgrade would be worth far less. In September 2025 NVIDIA went further and announced **Rubin CPX**, a GPU carrying **128 GB of GDDR7 rather than HBM** and **30 petaFLOPS of NVFP4** — deliberately trading bandwidth for compute and capacity, because prompt *processing* is compute-bound even though token *generation* is not. And in April 2025 Google introduced its seventh-generation TPU under the headline "**the first Google TPU for the age of inference**". When a vendor builds a separate chip for a workload, it has conceded that the workload is separate. ## One Cost Is a Purchase. The Other Is a Subscription You Sell. Meta's model card puts the 405B run at **30.84 million H100-hours**. That number is *finished*. It does not change if ten people download Llama or ten billion do. Training cost scales with **ambition** — with how big a model you decided to build — and with nothing else. Inference cost scales with **usage**. Every user, every query, every day, forever. Which means the cheapest model in the world to serve is one that nobody wants, and the most expensive thing that can happen to an AI company is that its product works. That is not a figure of speech. When OpenAI shipped native image generation in March 2025 and it went viral, Sam Altman wrote that the company's **"GPUs are melting"** and imposed rate limits — three free generations a day. No model had failed. No training run had overrun. *Demand* was the incident. The growth rates say the same thing more coldly. Google reported at I/O in May 2026 that it was processing **over 3.2 quadrillion tokens per month** across its products, up roughly **7× in a year** from about 480 trillion. That is not a curve driven by anyone's training budget. It is driven by people using things. ## The Crossover: 47 Trillion Tokens Here is the number the whole argument turns on, and it is checkable in three lines. [FLOPs](https://howaiworks.ai/glossary/flops) gives the two rules. Training costs about **6 FLOPs per parameter per token** (forward pass plus backward pass). Inference costs about **2N** — there is no backward pass. So for Llama 3.1 405B: ``` training = 3.8e25 FLOPs (Meta's published figure) per token = 2 x 4.05e11 = 8.1e11 FLOPs crossover = 3.8e25 / 8.1e11 = 4.7e13 tokens ``` **About 47 trillion tokens.** Three times the model's entire 15.6-trillion-token training corpus must pass through it — as prompts read or answers written — before serving it has cost as much arithmetic as building it. That sounds enormous. Now price it in real traffic. At DevDay in October 2025, Altman said OpenAI was processing **over 6 billion tokens per minute on the API** alone — 8.6 trillion a day. If those tokens were served at a 405B-scale cost per token, OpenAI's API would burn an entire frontier pre-training run's worth of compute **every 5.4 days**. Run the same counterfactual against Google's 3.2 quadrillion tokens a month and the crossover arrives in about **eleven hours**. Or count it per person. Artificial Analysis, measuring at the time of publication, puts one hard reasoning task at roughly **43,000 output tokens**. Divide: ``` 4.7e13 / 4.3e4 = 1.1 billion reasoning tasks ``` ChatGPT had **800 million weekly active users** in October 2025. **One hard reasoning question each, and you have spent a frontier pre-training run.** Those last figures are counterfactuals — nobody serves all their traffic on a 405B dense model, and the mix of model sizes is not public. Read them as scale, not as accounting. The scale is the point: the crossover is not a distant milestone that a successful model might reach one day. It is a **weekly** event, and then it happens again. ## Why the 2024 Chip-Demand Argument Was Backwards In December 2024, Ilya Sutskever told NeurIPS that *"pre-training as we know it will end"* — compute keeps growing, data does not, "because we have but one internet." The inference many drew from that was straightforward: if pre-training saturates, demand for accelerators falls. The premise was largely right. The conclusion was backwards, and not because pre-training secretly kept scaling. It was backwards because it counted only the first economy. What happened instead is that the compute **moved**. Reasoning models spend their budget while answering rather than while training: the model writes a long internal chain of thought, you are billed for every hidden token, and the mechanism belongs to [test-time compute](https://howaiworks.ai/glossary/test-time-compute), which derives it properly. Three months after Sutskever's talk, NVIDIA's Jensen Huang told GTC 2025 that reasoning and agentic AI needed **"easily 100 times more"** compute than the company had assumed a year earlier, decomposing it as roughly **ten times more tokens, generated ten times faster** to stay responsive. Look at what that did to the ledger. A pre-training run is a capital expense that **ends**. Test-time compute is an operating expense that **compounds with usage** — every user, every query, forever. The 2024 argument watched a capex line flatten and concluded the industry was done buying chips, at the exact moment the demand was migrating to the one line that grows with success. Deloitte's 2026 predictions put inference at roughly **two-thirds of all AI compute in 2026**, up from a third in 2023 and half in 2025. The lesson is not that a forecast was wrong. It is that **you cannot reason about the hardware market from the training economy alone**, because the training economy is the one that has a ceiling. ## Training Buys Capability. Inference Buys Margin. The two machines also buy different things, and this is where the money actually is. **Training buys a capability, once.** You either got the model you wanted or you did not. Efficiency work here buys you a *faster run* — a better parallelism plan, a higher MFU, a cheaper next model. It does not touch the price of your product. **Inference buys a margin, repeatedly** — and every unit of efficiency converts directly into money, because it lowers the cost of a good you sell over and over. The levers are well understood and they are all attacks on the memory wall. [Inference optimization](https://howaiworks.ai/glossary/inference-optimization) is mostly **batching**: read the weights once, use them for many users at the same time. On one eight-GPU H100 node running Llama 3.1 70B in FP8, that single dial moves the cost of a million tokens from about **$23.80 at batch size 1 to about 51 cents at batch 384** — a 47× swing, with no change to the model, the code or the hardware. [Quantization](https://howaiworks.ai/glossary/quantization) halves the bytes and therefore roughly halves the read. [Speculative decoding](https://howaiworks.ai/glossary/speculative-decoding) has a small model guess five tokens and the big model verify all of them for about **1.4% more time**, because the big model was going to read all its weights anyway. What that adds up to, DeepSeek published. For the 24 hours to 28 February 2025 it disclosed its serving meter: an average of **226.75 H800 nodes**, **$87,072 of compute per day** at an assumed $2 per GPU-hour, against **$562,027 per day** of tokens at list price — a stated **545% cost profit margin**, which DeepSeek itself notes is theoretical, since much of that traffic was discounted or free. Even discounted heavily, that is the shape of the second economy: a gross margin that is, in the end, a scheduler setting. ## What Breaks **You reason about the hardware market from training alone.** You rank accelerators by FLOPS, conclude that a plateau in pre-training means a plateau in demand, and miss that the growth had moved to a workload where FLOPS is nearly irrelevant and bandwidth is the product. The FLOPS ranking and the tokens-per-second ranking are not the same list — the H100 and H200 have identical FLOPS. **You plan capacity from your training budget.** This is the one that kills companies. The board paper prices the run: so many GPU-hours, so many weeks, a number with an end. Then the product works. Users arrive, and each one attaches a permanent, recurring compute cost to your P&L that appears in no line of that paper. Teams discover after launch that their gross margin is negative and that the fix is not an engineering breakthrough but a batch size — and that raising it makes the product feel slower to the users who are the reason they are in trouble. A model nobody uses is cheap to serve. That is the whole warning, and it is not a comforting one. ## Conclusion Two machines. Training: thousands of accelerators, a barrier every step, a network that sets the ceiling, a bill that arrives once and is then over. Serving: one chip at a time, arithmetic units idle, waiting on memory, a bill that arrives every day and grows with your success. They are not two phases of one business. They are two businesses that happen to share a supplier, and the industry has spent the last two years moving its centre of gravity from the first to the second — not because pre-training stopped working, but because [scaling laws](https://howaiworks.ai/glossary/scaling-laws) were never the only curve, and the other one is paid per query. If you only remember one number, remember 47 trillion. It is what a frontier training run costs in served tokens — and for anything people actually use, it is a number you pass, and then pass again. ## Sources - [Meta, Llama 3.1 405B model card](https://huggingface.co/meta-llama/Llama-3.1-405B) — 30.84M H100-hours, 15.6T training tokens, 3.8×10²⁵ FLOPs. See also [Llama](https://howaiworks.ai/models/llama). - [The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783) — 16,384 H100s, NVLink 900 GB/s vs 400 Gbps RoCE, 41% MFU. - [NVIDIA H100](https://www.nvidia.com/en-us/data-center/h100/) and [NVIDIA H200](https://www.nvidia.com/en-us/data-center/h200/) — identical 3,958 FP8 TFLOPS; 3.35 TB/s vs 4.8 TB/s; up to 1.9× Llama 2 70B inference. - [NVIDIA unveils Rubin CPX](https://nvidianews.nvidia.com/news/nvidia-unveils-rubin-cpx-a-new-class-of-gpu-designed-for-massive-context-inference) (9 September 2025) — 128 GB GDDR7, 30 PFLOPS NVFP4, built for context processing. - [Ironwood: The first Google TPU for the age of inference](https://blog.google/innovation-and-ai/infrastructure-and-cloud/google-cloud/ironwood-tpu-age-of-inference/) (April 2025). - [Sundar Pichai, Google I/O 2026 keynote](https://blog.google/innovation-and-ai/sundar-pichai-io-2026/) — over 3.2 quadrillion tokens/month, ~7× year on year. - [Sam Altman at DevDay, 6 October 2025](https://techcrunch.com/2025/10/06/sam-altman-says-chatgpt-has-hit-800m-weekly-active-users/) — 800M weekly ChatGPT users; over 6 billion tokens per minute on the API. - [Sam Altman, 27 March 2025](https://fortune.com/2025/03/28/sam-altman-chatgpt-gpus-melting-ai-images/) — "our GPUs are melting"; image-generation rate limits. - [Jensen Huang at GTC 2025](https://www.eetimes.com/huang-talks-tokens-reveals-roadmap-at-gtc-2025/) — "easily 100× more" compute for reasoning and agentic AI. - [Deloitte, TMT Predictions 2026](https://www.deloitte.com/us/en/insights/industry/technology/technology-media-and-telecom-predictions/2026/compute-power-ai.html) — inference ≈ two-thirds of all compute in 2026. - [DeepSeek V3/R1 inference system overview](https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md) (1 March 2025) — 226.75 H800 nodes, $87,072/day cost, $562,027/day theoretical revenue, 545% stated margin. ## Frequently Asked Questions ### What is the difference between AI training and AI inference? Training is the one-off process of producing a model: thousands of accelerators run a synchronised gradient-descent step in lockstep, and the network between them usually sets the ceiling. Inference is serving that model to users: a chip generating a token spends nearly all its time reading weights out of memory, with its arithmetic units almost idle. They are compute- and network-bound versus memory-bandwidth-bound — opposite limits on the same silicon. ### How many tokens must a model serve before inference costs as much compute as training it? For Meta's Llama 3.1 405B: training took about 3.8x10^25 FLOPs, and generating or reading one token costs roughly 2 x 405 billion = 8.1x10^11 FLOPs. Divide and you get about 47 trillion tokens — three times the model's entire 15.6-trillion-token training corpus. For a popular model that is a matter of days; for an unused one it never arrives. ### Why did the 2024 argument that pre-training was saturating not reduce chip demand? Because the compute moved rather than disappeared. Reasoning models spend their budget while answering rather than while training, so the demand reattached to serving — where it scales with the number of users instead of with a single training run. NVIDIA's Jensen Huang told GTC in March 2025 that reasoning and agentic AI needed 'easily 100 times more' compute than the company had assumed a year earlier. ### Is a chip that is good at training also good at inference? Not necessarily, and the vendors have stopped pretending otherwise. NVIDIA's H100 and H200 have identical FP8 tensor-core throughput; the H200 simply has faster memory, and NVIDIA advertises up to 1.9x the Llama 2 70B inference performance on the strength of that alone. Google called its Ironwood TPU 'the first Google TPU for the age of inference' in April 2025. ### Why can a successful AI product be more dangerous to a company than an expensive training run? Because a training run is a capital expense with a known end, while serving is an operating expense that grows with every user, every day. A model nobody uses is nearly free to serve. When OpenAI's image generation went viral in March 2025, Sam Altman said the company's 'GPUs are melting' and imposed rate limits — success, not failure, was the thing that broke the capacity plan. --- Source: https://howaiworks.ai/blog/training-vs-inference-two-economies — HowAIWorks.ai