Model Size

Definition

Model size is the number of parameters a model contains — the individual numbers, held in matrices, that training adjusted. Meta's Llama 3 8B holds 8.03 billion of them; GPT-3 held 175 billion; Moonshot's Kimi K3, the largest open-weight model as of 2026, holds 2.8 trillion. No other figure on a model card is quoted as often, because that single integer answers three different questions at once.

It tells you how much memory you need, since weights occupy the parameter count times the bytes each number takes: 405.8 billion parameters at two bytes each is 812 GB before anything else is loaded. It tells you what a token costs, because generating one token runs roughly two floating-point operations per parameter and training runs about six per parameter per training token. And it tells you, loosely, how capable the model is, because scaling laws relate parameter count to loss.

The first two are arithmetic and hold exactly. The third is a tendency, and it is where people get hurt. A 405-billion-parameter model is not automatically better than a 70-billion-parameter one: DeepMind's 70B Chinchilla outperformed the 280B Gopher trained on the same compute budget, and a 2024 8-billion-parameter model saw fifty times more training text than GPT-3 did in 2020. Parameter count orders models within a family trained the same way. Across families and across years it does not order them at all.

If you came here to find out how big the current frontier models are, the honest answer is that the closed ones do not say. OpenAI's GPT-4 technical report states outright that it "contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar," and nothing since has reversed that. Every specific number below therefore comes from a model whose weights and configuration are public, which is the only place model size is still a checkable fact.

How It Works

The number is derived, not declared

A parameter count is not chosen by a marketing team; it falls out of about six architectural integers, and anyone can reproduce it. The Llama 3 paper publishes the whole configuration for its flagship: 126 layers, a model dimension of 16,384, a feed-forward dimension of 53,248, 128 attention heads, 8 key/value heads (head dimension 128), over a 128,000-token vocabulary.

Work through one layer. The query and output projections are each 16,384 × 16,384; grouped-query attention shrinks the key and value projections to 16,384 × 1,024, because 8 key/value heads of dimension 128 is 1,024 rather than the full 16,384. Attention therefore holds 570,425,344 parameters. The SwiGLU feed-forward block is three matrices of 16,384 × 53,248, or 2,617,245,696 — about 82% of the layer. One layer is 3.19 billion parameters; 126 of them are 401.6 billion; the input embedding and output projection over a 128,000-token vocabulary add 4.19 billion more.

The total is 405,840,855,040. That is why Meta calls it 405B, and at two bytes per parameter it is the 810-odd GB of weights the model is usually described as occupying. Run the identical sum on the 8B row of the same table — 32 layers, dimension 4,096, feed-forward 14,336 — and you get 8,027,897,856. On the 70B row, 70.5 billion. The names on model cards are this arithmetic, rounded.

That is worth knowing for a practical reason. If a release ships weights and a config.json but no headline figure, you can compute the figure yourself. And when a vendor's headline number and its configuration disagree, the configuration is the one you have to fit into memory.

What the count buys, part one: memory

Weights occupy the parameter count multiplied by the bytes each parameter takes — 4 for float32, 2 for bfloat16, 1 for FP8, half a byte for 4-bit integers. The Llama 3 family at its native bfloat16 therefore needs 16.1 GB, 141 GB and 812 GB of weights for its 8B, 70B and 405B models. Against the 80 GB H100 that Meta itself trained on, that is one GPU, two GPUs and eleven GPUs, before a single byte of KV cache or activation memory.

Quantization changes the bytes, never the count. This is the confusion that costs people the most time. A 70-billion-parameter model served at 4 bits still has 70 billion parameters — the architecture is untouched, every matrix is the same shape, and the model is exactly as "large" as it was. What changed is that each number now occupies half a byte instead of two, so the weights fall from 141 GB to 35 GB and the model moves from two GPUs to one. "Smaller model" and "smaller file" are different claims about different quantities, and only pruning, distillation and compression touch the first one.

The distinction is not academic. Moonshot trained Kimi K3 with quantization-aware training so that its weights are meant to ship in MXFP4 — a 4-bit-class format — rather than at bfloat16, which is how the download of a multi-trillion-parameter model can be a fraction of what those same parameters at bfloat16 would occupy. The parameter count on the card and the size on disk are answering different questions.

Part two: compute, and the ceiling on tokens per second

A forward pass costs roughly two floating-point operations per parameter per token, one multiply and one add. Llama 3 405B therefore spends about 812 GFLOPs to generate a single token, and the 8B model about 16. Training costs about six FLOPs per parameter per training token once the backward pass is counted, so the whole pretraining budget is roughly 6ND. Check it: 6 × 405.8 billion × 15.6 trillion tokens is 3.80 × 10²⁵ FLOPs, and Meta reports pretraining the flagship with 3.8 × 10²⁵. Two public numbers reproduce a datacentre-scale budget to two significant figures.

But neither figure is what makes a chat feel fast. Generating a token requires reading every parameter out of memory once, and on a single stream that read, not the arithmetic, is the binding constraint — the memory wall in its plainest form. The ceiling is memory bandwidth divided by weight bytes: an 8-billion-parameter model at bfloat16 pulls 16.1 GB per token, and at 4 bits it pulls 4.0 GB. Same model, same parameter count, same hardware, four times the ceiling. That is why people running models locally care about precision more than almost anything else, and why the parameter count alone never predicted speed.

Part three: capability, and the number that always travels with it

Size is one half of a pair. DeepMind's 2022 result is the clearest statement of the other half: training over 400 models from 70 million to 16 billion parameters, they concluded that "for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled." They then trained Chinchilla — 70 billion parameters on 1.4 trillion tokens, twenty tokens per parameter — on the same compute as the 280-billion-parameter Gopher, and it won.

Line up the training ratios of models people actually compare and the point becomes hard to miss. GPT-3 saw 300 billion tokens for 175 billion parameters, 1.7 tokens per parameter. Chinchilla sat at 20. Llama 3 405B, on 15.6 trillion tokens, sits at 38.5. And Llama 3 8B was trained on that same 15.6 trillion tokens — roughly 1,950 tokens per parameter, about a hundred times past compute-optimal.

That last one is deliberate and it is the reason the size ladder no longer works as a quality ranking. Compute-optimal is a rule for spending a training budget; it is not a rule for shipping a product, because training is paid once and inference is paid forever. Overtraining a small model wastes training compute to buy permanently cheaper serving. So a 2020 model with 175 billion parameters and a 2024 model with 8 billion are not points on the same curve — one read 300 billion tokens and the other 15.6 trillion. Comparing them by parameter count compares the wrong number.

Total and active are two different sizes

For a Mixture-of-Experts model, "size" splits into two figures that behave nothing alike. DeepSeek-V3's model card gives 671B total parameters against 37B activated per token. The total sets the memory bill, because any expert may be needed for the next token and all of them must be resident. The active count sets the compute bill and the latency. That case is worked out in full on the Mixture-of-Experts page, down to the 1,342 GB of bfloat16 weights and the seventeen H100s it implies — the arithmetic is not repeated here, but the rule is: quote both numbers or you have not quoted the size. The newest models sharpen the point by refusing to reduce to one: Moonshot describes Kimi K3, the largest open-weight model as of 2026, as "effectively activating 16 out of 896 experts" — a 2.8-trillion-parameter total whose sparsity is stated as a fraction of experts rather than an active-parameter count at all.

Real-World Applications

Model size is, in practice, a hardware question. The parameter count decides what class of machine a model can run on at all, and every deployment decision downstream of that follows from it. The figures below are all taken from the models' own papers, cards or configuration files.

Read the first two rows together and the durability of the arithmetic is obvious: BERT-base and GPT-3 Small, six years and two labs apart, are within 15 million parameters of each other and both fit in a quarter of a gigabyte. The formula that put them there is the same one that puts DeepSeek-V3 on seventeen GPUs.

Meta's Llama 3 release is the clearest illustration of size as a product decision. One architecture and one training corpus were shipped at three sizes deliberately chosen to land on three different machines — a laptop-class 8B, a single-node 70B, and a 405B that Meta trained on up to 16,000 H100s and that essentially nobody self-hosts. The 8B and the 405B are not different products for different tasks; they are the same product at three price points, where the price is hardware.

The same logic runs in the other direction at the small end. BERT-base at 110 million parameters is a decade old and still deployed at scale for classification and retrieval ranking, because a task that needs a label rather than an essay does not need a hundred billion parameters, and the 640× gap between its 0.22 GB of weights and a 70B model's 141 GB shows up in every single request. Choosing a model size is choosing what you are willing to pay per call, forever.

At the top of the table, the reason the "what it takes to serve" column matters more than the parameter column is that the two large MoE entries advertise a per-token cost — DeepSeek-V3's 37B active, and K3's mere 16 of 896 experts — that would fit on a single GPU, while their totals do not fit on a node. That gap is the single most expensive misreading in the field.

Key Concepts

The nameplate is not the count. "8B" means 8,027,897,856, and "405B" means 405,840,855,040. The rounding is harmless for conversation and dangerous for capacity planning, because the difference between 405 and 405.8 billion parameters is 1.6 GB of memory you did not budget for.

Model size is not context length. These are routinely conflated, including by model cards, and they are unrelated quantities. Parameter count is a fact about the weights and fixed at training time; the context window is a fact about how many tokens the model will attend to at once, and its memory cost lives in the KV cache, not in the weights. A model can grow its context by an order of magnitude without gaining a single parameter.

Parameters are not the only thing in GPU memory. The weights are the fixed cost. On top of them sit the KV cache, which grows with context length and with the number of concurrent requests, plus activations and framework overhead. A 70.5-billion-parameter model at 141 GB does not run on exactly 141 GB of hardware, and a serving stack sized to the weights alone will fall over on its first long conversation.

Training memory is several times inference memory. Serving needs the weights. Training needs the weights, a full-precision master copy, the gradients, and two optimizer moments per parameter — which is why a model you can comfortably run on one card may need a small cluster to fully fine-tune. The parameters page works that multiplier out.

Challenges

Provisioning against the active count of an MoE. Reading "32B active" and buying a 32B-dense-sized machine understates the memory requirement by roughly the sparsity ratio — a factor of 18 for DeepSeek-V3, and larger still for the sparser trillion-scale MoEs that followed it (Kimi K3 fires only 16 of its 896 experts per token). Sparse activation reduces the arithmetic per token and does nothing whatsoever to the number of bytes that must be resident, because the router may send the very next token to any expert.

Treating parameter count as a benchmark. Emergent abilities — capabilities that appear abruptly past some size — are the usual argument for size as a proxy for quality, and the evidence is weaker than it looks. Schaeffer et al. argued in 2023 that "emergent abilities appear due to the researcher's choice of metric rather than due to fundamental changes in model behavior with scale": nonlinear, all-or-nothing metrics such as exact-match accuracy produce apparent discontinuities where continuous metrics on the same model outputs show smooth improvement. Sharp capability jumps at a parameter threshold may be a property of the ruler.

Comparing sizes across training regimes. Because token-to-parameter ratios now vary by three orders of magnitude — 1.7 for GPT-3, 20 for Chinchilla, 1,950 for Llama 3 8B — a parameter count only means something alongside the training-token count. A ranking of models by size alone is a ranking by an unspecified mixture of two variables.

No number at all for the models most people use. With frontier closed models publishing nothing, "how big is this model" is answerable only for open weights. Third-party estimates for closed models circulate widely and are unverifiable; treating them as data is how a plausible number becomes a citation.

Quantized weights hide the count. A repository shipped at 4 bits has a download size that no longer implies its parameter count, and two models with the same file size can differ fourfold in parameters. If you need the count, read the configuration and do the multiplication — do not infer it from the size of the download.

Future Trends

Totals rise while active counts stay flat. The MoE line has gone from Mixtral's 46.7B total to DeepSeek-V3's 671B to Kimi K3's 2.8T, while the per-token compute has stayed comparatively flat: Mixtral activated 12.9B parameters, DeepSeek-V3 37B, and K3 fires just 16 of its 896 experts. The consequence is that "model size" is steadily becoming two numbers moving in opposite directions, and any single figure quoted for a frontier open model is now ambiguous by default.

Small models keep absorbing large ones. The Qwen3 report describes exactly this route — "by leveraging the knowledge from the flagship models, we significantly reduce the computational resources required to build smaller-scale models, while ensuring their highly competitive performance." Knowledge distillation plus heavy overtraining means the capability available at a fixed parameter count keeps rising, which steadily devalues size as a cross-generation comparison.

A second axis that is not size at all. Test-time compute buys accuracy by letting a model think for longer rather than by making it bigger, so two deployments of one checkpoint — identical parameter count, identical memory footprint — can differ substantially in both quality and cost per answer. Whatever "how big is this model" was a proxy for, it is now a proxy for less of it.

Code Example

The parameter counts on Llama 3's model cards are not published figures you have to take on trust — they are a five-line function over the hyperparameters in Table 3 of the paper.

# Llama 3, from Table 3 of the paper: vocabulary 128,000, SwiGLU feed-forward,
# grouped-query attention with 8 key/value heads and a head dimension of 128.
def count(layers, d_model, d_ffn, n_heads, n_kv_heads=8, head_dim=128, vocab=128_000):
    d_kv = n_kv_heads * head_dim
    attn = 2 * d_model * (n_heads * head_dim) + 2 * d_model * d_kv   # Q, O, K, V
    ffn = 3 * d_model * d_ffn                                        # gate, up, down
    return 2 * vocab * d_model + layers * (attn + ffn)               # embed + unembed

for name, cfg in [("Llama 3 8B",   (32,  4_096, 14_336,  32)),
                  ("Llama 3 70B",  (80,  8_192, 28_672,  64)),
                  ("Llama 3 405B", (126, 16_384, 53_248, 128))]:
    n = count(*cfg)
    print(f"{name:14} {n/1e9:7.1f} B params "
          f"| bf16 {n*2/1e9:6.0f} GB | int4 {n*0.5/1e9:5.0f} GB "
          f"| {2*n/1e9:6.0f} GFLOPs/token")

Output:

Llama 3 8B         8.0 B params | bf16     16 GB | int4     4 GB |     16 GFLOPs/token
Llama 3 70B       70.5 B params | bf16    141 GB | int4    35 GB |    141 GFLOPs/token
Llama 3 405B     405.8 B params | bf16    812 GB | int4   203 GB |    812 GFLOPs/token

Three lines that reproduce three model names, and with them the whole memory and compute story. The function omits the normalisation parameters, which are a few hundred thousand per model and round away — everything that matters about model size is in the two matrix shapes and the layer count.