Scaling Laws

Definition

Scaling laws are a handful of fitted curves saying that a language model's loss falls as a power law in the compute, the data and the parameters you spend on it — smoothly, predictably, across more than seven orders of magnitude. They are the reason a company will commit to a multi-billion-dollar cluster for a model that does not exist yet: the curve says, approximately, what will come out.

They are also weaker than the word "law" suggests. They predict loss — the model's average surprise at the next token — and loss is not what anyone actually wants. Nobody buys a model because its cross-entropy is 1.97. And the returns are brutal: on the replicated Chinchilla fit, the reducible part of the loss falls as roughly C^-0.17, meaning each 10× increase in compute removes about a third of the error that is left. Two-thirds of it survives, and you pay ten times as much to attack it again.

That single sentence is most of what the AI industry is arguing about.

How It Works

Take a transformer, train it, and record the loss. Do it again with a bigger model, or more tokens, or a longer run. Plot loss against each of those on log-log axes and you get straight lines. A straight line on log-log axes is a power law, and a power law can be extrapolated.

Kaplan et al. fitted the original three, and they are worth writing out because their exponents are the whole story:

• loss versus parameters: L(N) = (8.8 × 10^13 / N)^0.076
• loss versus tokens: L(D) = (5.4 × 10^13 / D)^0.095
• loss versus compute: L(C) = (3.1 × 10^8 / C)^0.050

Those exponents are tiny. An exponent of 0.05 means a 10× increase in compute multiplies the loss by 10^-0.05 ≈ 0.89 — an 11% improvement for an order of magnitude of money. This is not a bug in the finding; it is the finding. Scaling works, and it works expensively.

How a training budget is counted in FLOPs is a separate topic, and so is the question of where the compute physically goes (almost all of it into matrix multiplication). For this page, one identity is enough: C ≈ 6ND — training a model with N parameters on D tokens costs about six FLOPs per parameter per token.

Kaplan said "build it bigger". Chinchilla said "train it longer".

Kaplan's 2020 paper also asked how to split a fixed budget, and answered: put nearly all of it into model size. Their allocation was N ∝ C^0.73 and D ∝ C^0.27 — so a 100× budget increase should buy a 28.8× bigger model trained on only 3.5× more data.

The field believed this, and built accordingly. GPT-3: 175B parameters, 300B tokens. Gopher: 280B parameters, 300B tokens — barely more than one token per parameter.

In 2022 DeepMind trained over 400 models and found the opposite. Model size and training tokens should be scaled equally: every doubling of parameters wants a doubling of tokens too. At the budgets they tested this lands near 20 tokens per parameter — and the models everyone had just spent hundreds of millions of dollars on were undertrained by a factor of about four.

Worked example: what Gopher's budget actually bought

Gopher's training budget was about 5.8 × 10^23 FLOPs. Hold that fixed and use C ≈ 6ND to see what the two rules do with it.

Kaplan-era split (what DeepMind built the first time). Fix N = 280B parameters. Then the tokens you can afford are

D = C / 6N = 5.8 × 10^23 / (6 × 2.8 × 10^11) ≈ 345 billion tokens — about 1.2 tokens per parameter.

Chinchilla split. Impose D = 20N and solve. Substituting into C = 6ND gives C = 120N², so

N = √(C / 120) = √(5.8 × 10^23 / 120) = √(4.83 × 10^21) ≈ 69.5 billion parameters, and D = 20N ≈ 1.39 trillion tokens.

DeepMind built exactly that: Chinchilla, 70B parameters on 1.4T tokens — same budget, quarter the size, four times the data. The arithmetic reproduces the model.

Now the payoff. Feeding both into the replicated Chinchilla loss curve, L(N, D) = 1.817 + 482 / N^0.348 + 2085 / D^0.366:

The loss gap is 0.026 nats. It looks like nothing. It was worth 7.6 points of MMLU and beat Gopher, GPT-3, Jurassic-1 and Megatron-Turing NLG across the board — from a model a quarter the size, which is also four times cheaper to serve.

This is what breaks if you get it wrong. Getting the split wrong does not slow you down; it spends your entire budget on a worse model. Gopher's compute was not recoverable. Every lab that had scaled parameters-first between 2020 and 2022 had bought a model that a four-times-smaller one could beat — and had to pay again to find out.

What the next 10× buys

Run the same compute-optimal allocation forward from Chinchilla's budget and the curve stops flattering anyone. The floor term, E ≈ 1.82, is irreducible — no amount of compute goes below it — so only the other 0.157 nats are on the table:

A thousand-fold increase in spending removes 71% of the reducible loss. That has always been the deal on offer. Nothing broke; people read the first factor of ten and extrapolated the feeling rather than the exponent.

Real-World Applications

Every frontier pretraining run is a bet placed on these curves. Meta's Llama 3 405B was trained on 15.6T tokens for 3.8 × 10^25 FLOPs, and Meta states the 405B size was approximately compute-optimal for their own fitted scaling law — which put the ratio at 38.5 tokens per parameter, not 20. Chinchilla's 20:1 is a fit on one dataset with one recipe; it is not a constant of nature, and every serious lab now refits it before a big run rather than quoting it.

Deliberate overtraining, because compute-optimal is not deployment-optimal. Llama 3 8B was trained on 15T tokens — 1,875 tokens per parameter, roughly 94× the Chinchilla ratio. By the training curve that is waste. By the serving bill it is obvious: you pay to train once and pay to infer forever, so a smaller model pushed far past its compute-optimal point is the cheaper artefact if enough people use it. Chinchilla-optimal answers "cheapest model I can train"; almost no one is actually asking that question.

The capex argument. The largest cloud providers are committing hundreds of billions of dollars a year to AI infrastructure, and there is exactly one quantitative reason to believe the next tranche of that spending buys a better model than the last: a set of curves fitted to a few hundred training runs on datasets nobody outside the labs has seen. That is the entire empirical basis. When commentators ask whether "scaling has hit a wall", these curves are the object under discussion — not a metaphor, and not a figure of speech about progress in general.

Key Concepts

• Compute-optimal ≠ best. A compute-optimal model is the one that minimises training loss for a training budget. It ignores inference cost, latency, memory footprint and how many users you have. The frontier optimises a different objective than the paper does.
• The exponents are properties of the dataset, not the universe. Change the data mixture, the tokenizer, the architecture (MoE versus dense) and the fit moves. Kaplan's and Chinchilla's disagreement was partly a learning-rate-schedule artefact, not a discovery about intelligence.
• The published Chinchilla fit was wrong, and the correction matters. Hoffmann et al.'s reported coefficients (E = 1.69, A = 406.4, B = 410.7, α = 0.34, β = 0.28) imply an optimum near 70 tokens per parameter — inconsistent with the 20:1 they used to build Chinchilla and with their own two other methods. A 2024 replication refitted them to E = 1.82, A = 482, B = 2085, α = 0.348, β = 0.366, which restores ~20:1. This page uses the corrected fit.
• Loss is measured in nats, and nats are deceptive. Three hundredths of a nat separated Chinchilla from Gopher. Read loss differences as ratios of the reducible part, never as absolute distances.

Challenges

Scaling laws predict loss, and nobody wants loss. This is the central limitation and it is usually glossed. OpenAI predicted GPT-4's final loss from models trained with up to 10,000× less compute, and its pass rate on a HumanEval subset from models with 1,000× less — genuinely impressive extrapolation. But the GPT-4 report is equally clear that other capabilities remained unpredictable, and the mapping from a loss number to "can it write a correct multi-file patch" is not part of any law. A curve that tells you the loss and not the capability is a budgeting tool, not a theory of intelligence.

The related fight is over emergence — abilities that seem to appear abruptly at scale. A NeurIPS 2023 award paper argued these are largely an artefact of the metric: score with a discontinuous measure like exact-match accuracy and you see a cliff; score the same outputs with a continuous measure and the improvement is smooth all along. Either way, the loss curve does not tell you when your product starts working.

The data wall is arithmetic, not opinion. Compute-optimal training needs D = 20N tokens, and N grows as √C, so D grows as √C too. Push the compute-optimal recipe to 10^28 FLOPs — a few hundred times Llama 3 405B — and it asks for √(10^28/120) × 20 ≈ 183 trillion tokens. Epoch AI puts the entire stock of public human-generated text at roughly 300 trillion tokens, and projects it being fully consumed somewhere between 2026 and 2032. The compute curve does not care that the data runs out; it simply stops being achievable, and you fall off the compute-optimal frontier into the overtrained-on-repeats regime, where the exponents are worse.

Extrapolation is a promise about a region nobody has visited. Every scaling law is fitted below the frontier and used above it. Kaplan's held for two years and then bent — at the exact moment it was most expensive to be wrong about.

Future Trends

The live question is whether a second axis rescues the first. Test-time scaling — spending compute at inference by letting a model think longer, sample more and verify itself — has its own empirical curve, and it buys accuracy from a budget that is not pretraining compute and does not touch the data wall. It is a different law with different economics (you pay it per query, forever), and it is why the "scaling is over" argument and the "scaling is fine" argument can both cite real numbers.

On the data side the responses are synthetic generation, multimodal corpora, and squeezing more out of the same tokens. Note what they share: each tries to move the constants of the curve, because the exponents have proven very hard to move. A better dataset and 20× the compute both help. Only one of them is for sale.