Parameters

Definition

A parameter is one number that a model learned during training, and the number of them tells you — by a multiplication you can do in your head — how much memory the model needs. Take the parameter count, multiply by the bytes each number takes up, and you have the size of the model when it is loaded onto a GPU. That single conversion is why the parameter count is the headline of every model announcement: it is the figure from which the hardware bill follows.

Llama 3.1 70B does not have 70 billion parameters. It has exactly 70,553,706,496 of them, and they ship as 16-bit numbers, two bytes each. That is 141 GB of weights — which does not fit in the 80 GB of an NVIDIA H100. Everything else on this page is a consequence of that sentence.

How It Works

A parameter is an entry in a matrix

A neural network turns an input into an output by multiplying it by a series of large tables of numbers. Each of those tables is a matrix; each individual number inside one is a parameter, also called a weight. At the start of training the numbers are random and the model's output is noise. Training nudges every one of them, over and over, toward values that make the output less wrong. When training stops, the numbers freeze, and those frozen numbers are the model — a checkpoint file is nothing but a list of them plus the recipe for which matrix each belongs to.

The distinction that trips up newcomers: parameters are learned, hyperparameters are chosen. How many layers, how wide they are, the learning rate — those are decisions a human made before training. The 70 billion numbers that filled the resulting matrices are what training produced.

Where the 70 billion actually are

They are not spread evenly, and you can count them yourself. Llama 3.1 70B is a transformer with 80 identical layers, a hidden width of 8,192, a feed-forward width of 28,672, and 8 key/value heads sharing 64 query heads (model card; the Llama 3 paper gives the same architecture table). Each layer holds:

• the attention projections: 8192×8192 for queries, 8192×1024 each for keys and values, 8192×8192 for the output — 151 million parameters;
• the feed-forward block, three matrices of 8192×28672 — 705 million parameters.

So one layer is 856 million parameters, and the feed-forward block, not attention, holds 82% of them. Multiply by 80 layers and add the two 128,256 × 8,192 embedding tables that convert tokens to vectors and back:

80 × 855,638,016 = 68,451,041,280 + 2 × 1,050,673,152 = 70,552,387,584 + 1,318,912 normalisation scales = 70,553,706,496

That is the exact number in the checkpoint. There is no rounding and no mystery: parameters are a property of the architecture, fixed the moment someone chose the layer count and the widths.

The conversion that matters: parameters × bytes = memory

Every parameter occupies some number of bytes, and that number is a choice made at deployment time. Quantization owns the formats; all you need here is the exchange rate. For our 70.55B parameters:

An H100 has 80 GB of memory; an H200 has 141 GB — which the BF16 weights would exactly exhaust, leaving nothing for the KV cache or activations, so in practice it does not fit either. A 24 GB RTX 4090 cannot hold this model in any format. This is the entire reason quantization exists, and the reason the same model can be described as "needs a data centre" or "runs on one card" depending on nothing but the bytes-per-parameter you picked.

The bytes are also what you read, every token, which is why they set your speed as well as your capacity — see the memory wall and high-bandwidth memory.

Training needs about eight times the memory of inference

Running a model holds the weights. Training one holds the weights, a gradient for every parameter, and the optimizer's running statistics for every parameter — with mixed-precision Adam that is roughly 16 bytes per parameter (ZeRO, Rajbhandari et al., 2019), against 2 for inference. For 70.55B parameters:

70.55 × 10⁹ × 16 bytes ≈ 1,129 GB ≈ 1.1 TB

before a single activation is stored. That is fifteen H100s' worth of memory to train a model that serves from one, and it is why distributed training is not an optimisation but a precondition. It is also why full fine-tuning is so much more expensive than people expect, and why LoRA-style methods — which train a few million new parameters and freeze the rest — took over.

Total parameters and active parameters are different numbers

A dense model uses all of its parameters on every token. A Mixture-of-Experts model does not: a router sends each token to a small subset of the network, and the rest of the weights sit idle for that token.

DeepSeek-V3 has 671B total parameters and activates 37B per token (technical report). Both numbers are real, and they answer different questions:

• Memory follows the total. All 671B parameters must be resident, because the router may pick any of them for the next token. In the FP8 the model natively ships in, that is ~671 GB — nine H100s' worth of capacity, before the KV cache.
• Compute follows the active count. Each token does about 2 × 37 × 10⁹ = 74 GFLOP of arithmetic, not 2 × 671 × 10⁹ = 1,342 GFLOP. It is an 18× smaller sum. (The FLOPs page owns that estimate and the 6ND training budget it belongs to.)

The names make this worse, not better. Mixtral 8x7B sounds like 56B parameters; it has 46.7B total and 12.9B active (Mistral), because the "8x" applies only to the feed-forward blocks and the attention layers are shared. Kimi K2 is 1T total, 32B active (Moonshot AI). No arithmetic on the model's name will recover either figure. You have to read both numbers from the model card.

Real-World Applications

Sizing hardware is the parameter count's day job. Before anyone runs a model, someone converts its parameter count into a GPU order — and that conversion is the table above. It is why an "open weights" release with 70B parameters is genuinely usable by a small team and a 671B release is not, even though both are free downloads. The gap between them is not licensing. It is 600 GB.

Choosing a precision is choosing a machine. A team that needs Llama 3.1 70B on one GPU is not choosing between models; it is choosing between 8-bit (fits, 9 GB spare, tight KV cache) and 4-bit (fits, 44 GB spare, room to serve many users at once, some accuracy given up). Both options are the same model. The decision is made in bytes per parameter, and inference optimization is largely the study of making that trade well.

Regulators count compute, and compute is estimated from parameters. The EU AI Act presumes a general-purpose model has "systemic risk" when its cumulative training compute exceeds 10²⁵ FLOPs (Article 51). Nobody watches the training run with a meter; the figure is estimated, and the standard estimate is a function of the parameter count and the tokens trained on. Which parameter count you use for a sparse model therefore has legal consequences, and it is the active one that enters the arithmetic.

The headline number is a marketing artefact — and it moved. GPT-3's launch in 2020 made "175 billion parameters" the entire story (Brown et al.). Frontier labs mostly stopped publishing the figure afterwards, because scaling laws showed that a smaller model trained on far more data beats a bigger one trained on less. Parameter count survives as the headline in open-weights releases for the practical reason that it is the number you need to know before you can run the thing.

Key Concepts

• Parameter count is a cost, not a score. It sets capacity — how much the model could learn — and the memory bill you will definitely pay. What it actually learned depends on data and training compute, which is why an 8B model from 2026 outperforms a 175B model from 2020 on almost everything.
• "B" means the parameters, never the file size. A 7B model is not a 7 GB download unless it happens to ship at one byte per parameter. The unit is numbers, and the bytes are a separate, changeable fact about the format.
• The two numbers on a sparse model are not interchangeable. Total parameters buy you a GPU; active parameters buy you a token. Quote one where the other belongs and every estimate downstream is wrong.

Challenges

What breaks if you get total-versus-active backwards is every downstream number at once, by an order of magnitude. Size a DeepSeek-V3 deployment on its 37B active parameters and you provision about 37 GB of memory for a model that needs 671 GB — it will not load, and no amount of tuning will make it. Make the mirror-image error and estimate its speed or its training FLOPs from 671B and you will predict a system 18× slower and 18× more expensive to train than the one that actually exists. The same mistake in a regulatory filing puts a model on the wrong side of a compute threshold. There is no version of this confusion that produces a small error; MoE ratios are 10–30×, so the mistake is always an order of magnitude.

Parameter counts are not comparable across architectures. Comparing a 671B sparse model with a 70B dense one tells you almost nothing: the first is far larger in memory and roughly half the arithmetic per token. "Bigger model" has quietly stopped being a single ordering, and any leaderboard sorted by parameter count is sorting two different quantities into one column.

A published count may not be the count you get. Quantized redistributions, merged LoRA adapters, pruned variants and multi-token-prediction modules all move the number: DeepSeek-V3's Hugging Face repository is 685B parameters, not 671B, because it includes a 14B MTP module the base model does not use at inference (model card). If the memory arithmetic has to be right, count the checkpoint you actually downloaded.

Most frontier models do not tell you. GPT-5-class and Claude-class parameter counts are not published. Every "GPT-4 is 1.8T parameters" figure in circulation is a leak or an inference, and building a cost model on one is building on a rumour.

Future Trends

The direction of travel is that the total count keeps climbing while the active count barely moves — 671B/37B, 1T/32B — because memory is comparatively cheap to add and bandwidth is not. Sparsity is how a lab buys capacity without buying the per-token cost of using it, and it is turning the single "size" of a model into at least two numbers that must be quoted together.

The second shift is that even those numbers no longer determine how good an answer you get, because a model can now spend a variable amount of computation at inference time on a hard question. When that happens, capability stops being a property of the checkpoint alone and becomes a property of the checkpoint plus the compute budget you gave it — see test-time compute. The parameter count still tells you exactly what the model costs to hold in memory. It tells you less and less about what the model can do.