--- source: 'https://howaiworks.ai/glossary/scaling-laws' section: glossary title: Scaling Laws description: >- The empirical curves that make AI capex rational: model loss falls predictably with compute, data and parameters — and what they do not promise. tags: - Scaling Laws - Deep Learning - LLM - Model Training - AI Research - AI Infrastructure - Machine Learning category: Deep Learning datePublished: '2026-07-13' lastUpdated: '2026-07-13' --- # Scaling Laws > The empirical curves that make AI capex rational: model loss falls predictably with compute, data and parameters — and what they do not promise. ## Definition **Scaling laws are a handful of fitted curves saying that a language model's [loss](https://howaiworks.ai/glossary/loss-function) falls as a *power law* in the compute, the data and the parameters you spend on it — smoothly, predictably, across more than seven orders of magnitude.** They are the reason a company will commit to a multi-billion-dollar cluster for a model that does not exist yet: the curve says, approximately, what will come out. They are also weaker than the word "law" suggests. They predict **loss** — the model's average surprise at the next [token](https://howaiworks.ai/glossary/token) — and loss is not what anyone actually wants. Nobody buys a model because its cross-entropy is 1.97. And the returns are brutal: on the replicated Chinchilla fit, the *reducible* part of the loss falls as roughly **C^-0.17**, meaning each **10× increase in compute removes about a third of the error that is left**. Two-thirds of it survives, and you pay ten times as much to attack it again. That single sentence is most of what the AI industry is arguing about. ## How It Works Take a transformer, train it, and record the loss. Do it again with a bigger model, or more tokens, or a longer run. Plot loss against each of those on log-log axes and you get straight lines. A straight line on log-log axes is a power law, and a power law can be extrapolated. Kaplan et al. fitted the original three, and they are worth writing out because their exponents are the whole story: - loss versus parameters: **L(N) = (8.8 × 10^13 / N)^0.076** - loss versus tokens: **L(D) = (5.4 × 10^13 / D)^0.095** - loss versus compute: **L(C) = (3.1 × 10^8 / C)^0.050** Those exponents are tiny. An exponent of 0.05 means a 10× increase in compute multiplies the loss by 10^-0.05 ≈ 0.89 — an 11% improvement for an order of magnitude of money. This is not a bug in the finding; it *is* the finding. Scaling works, and it works expensively. How a training budget is counted in [FLOPs](https://howaiworks.ai/glossary/flops) is a separate topic, and so is the question of where the compute physically goes (almost all of it into [matrix multiplication](https://howaiworks.ai/glossary/matrix-multiplication)). For this page, one identity is enough: **C ≈ 6ND** — training a model with N parameters on D tokens costs about six FLOPs per parameter per token. ### Kaplan said "build it bigger". Chinchilla said "train it longer". Kaplan's 2020 paper also asked how to *split* a fixed budget, and answered: put nearly all of it into model size. Their allocation was N ∝ C^0.73 and D ∝ C^0.27 — so a 100× budget increase should buy a **28.8× bigger model** trained on only **3.5× more data**. The field believed this, and built accordingly. GPT-3: 175B parameters, 300B tokens. Gopher: 280B parameters, 300B tokens — barely more than one token per parameter. In 2022 DeepMind trained over 400 models and found the opposite. **Model size and training tokens should be scaled equally: every doubling of parameters wants a doubling of tokens too.** At the budgets they tested this lands near **20 tokens per parameter** — and the models everyone had just spent hundreds of millions of dollars on were *undertrained by a factor of about four*. ### Worked example: what Gopher's budget actually bought Gopher's training budget was about **5.8 × 10^23 FLOPs**. Hold that fixed and use C ≈ 6ND to see what the two rules do with it. **Kaplan-era split (what DeepMind built the first time).** Fix N = 280B parameters. Then the tokens you can afford are > D = C / 6N = 5.8 × 10^23 / (6 × 2.8 × 10^11) ≈ **345 billion tokens** — about **1.2 tokens per > parameter**. **Chinchilla split.** Impose D = 20N and solve. Substituting into C = 6ND gives C = 120N², so > N = √(C / 120) = √(5.8 × 10^23 / 120) = √(4.83 × 10^21) ≈ **69.5 billion parameters**, > and D = 20N ≈ **1.39 trillion tokens**. DeepMind built exactly that: **Chinchilla, 70B parameters on 1.4T tokens** — same budget, quarter the size, four times the data. The arithmetic reproduces the model. Now the payoff. Feeding both into the replicated Chinchilla loss curve, L(N, D) = 1.817 + 482 / N^0.348 + 2085 / D^0.366: | | parameters | tokens | predicted loss | perplexity | MMLU | |---|---|---|---|---|---| | Gopher | 280B | 300B | **2.00** | 7.39 | 60.0% | | Chinchilla | 70B | 1.4T | **1.97** | 7.20 | **67.6%** | The loss gap is **0.026 nats**. It looks like nothing. It was worth **7.6 points of MMLU** and beat Gopher, GPT-3, Jurassic-1 and Megatron-Turing NLG across the board — from a model a quarter the size, which is also four times cheaper to serve. **This is what breaks if you get it wrong.** Getting the split wrong does not slow you down; it *spends your entire budget on a worse model*. Gopher's compute was not recoverable. Every lab that had scaled parameters-first between 2020 and 2022 had bought a model that a four-times-smaller one could beat — and had to pay again to find out. ### What the next 10× buys Run the same compute-optimal allocation forward from Chinchilla's budget and the curve stops flattering anyone. The floor term, **E ≈ 1.82**, is irreducible — no amount of compute goes below it — so only the other 0.157 nats are on the table: | budget | parameters | tokens | loss | reducible loss left | |---|---|---|---|---| | 1× (5.8e23) | 70B | 1.4T | 1.974 | 0.157 | | 10× | 220B | 4.4T | 1.921 | 0.104 | | 100× | 695B | 13.9T | 1.886 | 0.069 | | 1,000× | 2.2T | 44T | 1.863 | 0.046 | A **thousand-fold** increase in spending removes **71%** of the reducible loss. That has always been the deal on offer. Nothing broke; people read the first factor of ten and extrapolated the *feeling* rather than the exponent. ## Real-World Applications **Every frontier pretraining run is a bet placed on these curves.** Meta's Llama 3 405B was trained on 15.6T tokens for 3.8 × 10^25 FLOPs, and Meta states the 405B size was approximately compute-optimal *for their own fitted scaling law* — which put the ratio at **38.5 tokens per parameter**, not 20. Chinchilla's 20:1 is a fit on one dataset with one recipe; it is not a constant of nature, and every serious lab now refits it before a big run rather than quoting it. **Deliberate overtraining, because compute-optimal is not deployment-optimal.** Llama 3 8B was trained on 15T tokens — **1,875 tokens per parameter, roughly 94× the Chinchilla ratio**. By the training curve that is waste. By the *serving* bill it is obvious: you pay to train once and pay to [infer](https://howaiworks.ai/glossary/inference) forever, so a smaller model pushed far past its compute-optimal point is the cheaper artefact if enough people use it. Chinchilla-optimal answers "cheapest model I can train"; almost no one is actually asking that question. **The capex argument.** The largest cloud providers are committing hundreds of billions of dollars a year to AI infrastructure, and there is exactly one *quantitative* reason to believe the next tranche of that spending buys a better model than the last: a set of curves fitted to a few hundred training runs on datasets nobody outside the labs has seen. That is the entire empirical basis. When commentators ask whether "scaling has hit a wall", these curves are the object under discussion — not a metaphor, and not a figure of speech about progress in general. ## Key Concepts - **Compute-optimal ≠ best.** A compute-optimal model is the one that minimises *training* loss for a training budget. It ignores inference cost, latency, memory footprint and how many users you have. The frontier optimises a different objective than the paper does. - **The exponents are properties of the dataset, not the universe.** Change the data mixture, the tokenizer, the architecture ([MoE](https://howaiworks.ai/glossary/mixture-of-experts) versus dense) and the fit moves. Kaplan's and Chinchilla's disagreement was partly a learning-rate-schedule artefact, not a discovery about intelligence. - **The published Chinchilla fit was wrong, and the correction matters.** Hoffmann et al.'s reported coefficients (E = 1.69, A = 406.4, B = 410.7, α = 0.34, β = 0.28) imply an optimum near **70** tokens per parameter — inconsistent with the 20:1 they used to build Chinchilla and with their own two other methods. A 2024 replication refitted them to E = 1.82, A = 482, B = 2085, α = 0.348, β = 0.366, which restores ~20:1. This page uses the corrected fit. - **Loss is measured in nats, and nats are deceptive.** Three hundredths of a nat separated Chinchilla from Gopher. Read loss differences as ratios of the *reducible* part, never as absolute distances. ## Challenges **Scaling laws predict loss, and nobody wants loss.** This is the central limitation and it is usually glossed. OpenAI predicted GPT-4's final loss from models trained with up to **10,000×** less compute, and its pass rate on a HumanEval subset from models with 1,000× less — genuinely impressive extrapolation. But the GPT-4 report is equally clear that other capabilities remained unpredictable, and the mapping from a loss number to *"can it write a correct multi-file patch"* is not part of any law. A curve that tells you the loss and not the capability is a budgeting tool, not a theory of intelligence. The related fight is over **emergence** — abilities that seem to appear abruptly at scale. A NeurIPS 2023 award paper argued these are largely an artefact of the metric: score with a discontinuous measure like exact-match accuracy and you see a cliff; score the *same outputs* with a continuous measure and the improvement is smooth all along. Either way, the loss curve does not tell you when your product starts working. **The data wall is arithmetic, not opinion.** Compute-optimal training needs D = 20N tokens, and N grows as √C, so **D grows as √C too**. Push the compute-optimal recipe to 10^28 FLOPs — a few hundred times Llama 3 405B — and it asks for **√(10^28/120) × 20 ≈ 183 trillion tokens.** Epoch AI puts the entire stock of public human-generated text at roughly **300 trillion tokens**, and projects it being fully consumed somewhere between **2026 and 2032**. The compute curve does not care that the data runs out; it simply stops being achievable, and you fall off the compute-optimal frontier into the overtrained-on-repeats regime, where the exponents are worse. **Extrapolation is a promise about a region nobody has visited.** Every scaling law is fitted below the frontier and used above it. Kaplan's held for two years and then bent — at the exact moment it was most expensive to be wrong about. ## Future Trends The live question is whether a second axis rescues the first. **Test-time scaling** — spending compute at [inference](https://howaiworks.ai/glossary/inference) by letting a model think longer, sample more and verify itself — has its own empirical curve, and it buys accuracy from a budget that is not pretraining compute and does not touch the data wall. It is a different law with different economics (you pay it per query, forever), and it is why the "scaling is over" argument and the "scaling is fine" argument can both cite real numbers. On the data side the responses are synthetic generation, multimodal corpora, and squeezing more out of the same tokens. Note what they share: each tries to move the *constants* of the curve, because the exponents have proven very hard to move. A better dataset and 20× the compute both help. Only one of them is for sale. ## Frequently Asked Questions ### What are scaling laws in AI? They are fitted curves showing that a language model's training loss falls as a power law in the compute, data and parameters you spend on it. The relationship held across more than seven orders of magnitude in the original 2020 study, which is why a lab will commit to a cluster before the model exists — the curve says roughly what it will get. ### What is the Chinchilla 20 tokens per parameter rule? DeepMind's 2022 result found that for a fixed compute budget, model size and training tokens should grow together, landing near 20 training tokens per parameter at the budgets they tested. It is a fitted optimum on one dataset, not a constant of nature: Meta's own fit for Llama 3 put the ratio nearer 39. ### Do scaling laws mean bigger models are always better? No — that was the Kaplan-era reading, and it was wrong. Chinchilla showed that a 70B model trained on 1.4T tokens beat a 280B model trained on 300B tokens using the same compute. Above a fixed budget, spending it all on parameters wastes it. ### Do scaling laws predict capability? They predict loss — average surprise per token — and nothing else. The step from loss to a benchmark score is not part of the law. Chinchilla's loss advantage over Gopher was about 0.03 nats and its MMLU advantage was 7.6 points; no curve told anyone that exchange rate in advance. ### Are we running out of data to scale on? Epoch AI estimates the stock of public human-generated text at roughly 300 trillion tokens and projects it being fully used somewhere between 2026 and 2032. Compute-optimal training at 10^28 FLOPs would want about 183 trillion tokens, so the constraint is arithmetic, not rhetoric. ### Has scaling stopped working? The curves have not broken; they were always this shape. What changed is that the cheap part of the curve is spent — the reducible loss falls roughly as compute to the power -0.17, so each 10x of spending removes about a third of the error that remains, and the returns look worse every time you pay them. ## Related ### Related terms - [Model Size](https://howaiworks.ai/glossary/model-size) - [Training](https://howaiworks.ai/glossary/training) - [Foundation Models](https://howaiworks.ai/glossary/foundation-models) - [Deep Learning](https://howaiworks.ai/glossary/deep-learning) - [Large Language Model (LLM)](https://howaiworks.ai/glossary/large-language-model) - [Benchmark](https://howaiworks.ai/glossary/benchmark) --- Source: https://howaiworks.ai/glossary/scaling-laws — HowAIWorks.ai