Definition
A small language model is one small enough to run on the device that uses it, rather than in a datacentre you call over the network. In 2026 that means roughly 0.3 to 5 billion parameters, because that is the range that fits into a phone or laptop's spare memory once quantised.
The definition is deliberately about deployment rather than a parameter count, because the count that qualifies keeps moving and the constraint does not. What makes a model "small" is that the memory it occupies and the bandwidth it consumes per token both fit inside a budget someone else is also using — for their browser, their photos, their operating system.
The part that surprises people is why these models became good, and it has almost nothing to do with clever architecture. It is a deliberate trade: spend enormously more on training in order to buy permanently cheaper inference. The arithmetic below makes that concrete, and it is the single thing worth understanding about the category.
How It Works
Why the size limit exists: two numbers, not one
The obvious constraint is memory. A model's weights occupy parameters × bytes per parameter. At FP16 that is 2 bytes each; at INT4, after quantization, it is roughly 0.5 bytes.
| Model size | FP16 | INT4 |
|---|---|---|
| 350M | 0.7 GB | 0.18 GB |
| 1B | 2.0 GB | 0.5 GB |
| 3B | 6.0 GB | 1.5 GB |
| 7B | 14.0 GB | 3.5 GB |
| 70B | 140 GB | 35 GB |
A phone with 8 GB of RAM might make 2–3 GB available to an application. That table tells you immediately why 7B is roughly where phones stop participating, and why 70B is not a rounding error away from feasible but two orders of magnitude away.
The less obvious constraint, and the one that actually governs how fast the model feels, is memory bandwidth. Generating each token in a dense model requires reading essentially every weight from memory. So:
tokens per second ≈ memory bandwidth ÷ bytes of weights
Phone-class LPDDR5X delivers on the order of 50 GB/s. Run the division:
- 1B at INT4 (0.5 GB): 50 ÷ 0.5 = 100 tokens/second
- 3B at INT4 (1.5 GB): 50 ÷ 1.5 ≈ 33 tokens/second
- 7B at INT4 (3.5 GB): 50 ÷ 3.5 ≈ 14 tokens/second
These are ceilings — real throughput is lower — but the ratios hold, and they are the reason the category exists. A 7B model on a phone is not slightly slower than a 1B model. It is seven times slower, on hardware where 14 tokens/second reads as sluggish and 100 reads as instant. This is the memory wall at consumer scale, and no amount of NPU compute fixes it, because the bottleneck is moving weights rather than multiplying them.
Why they got good: trained far past compute-optimal
The Chinchilla scaling result says that for a fixed training budget, the compute-optimal split is roughly 20 training tokens per parameter. For a 350M model that is about 7 billion tokens.
Liquid AI's LFM2.5-350M was trained on 28 trillion tokens.
That is 80,000 tokens per parameter — about 4,000 times the compute-optimal ratio. Using the standard 6ND estimate for training FLOPs:
6 × 350e6 params × 28e12 tokens ≈ 5.9e22 FLOPs
Chinchilla-optimal for the same model would have been about 6 × 350e6 × 7e9 ≈ 1.5e19 FLOPs. The actual training run spent roughly 4,000× more compute than the "optimal" recipe prescribes.
This looks irrational until you notice that Chinchilla optimises the wrong thing. It minimises the cost of training one model. A model that will serve billions of requests on a billion devices should minimise the cost of inference, and inference cost is set by parameter count, which is fixed once you choose it. So you pour compute into training until the small model is as good as it can possibly be at that size, because every token it ever generates is cheap forever.
Scaling laws did not stop applying. The industry started optimising a different objective under them.
The supporting techniques
Three others matter, in rough order of impact:
- Quantization to INT4. The 4× memory reduction in the table above, and therefore a 4× bandwidth reduction. Meta's MobileLLM-Pro reports near-lossless INT4, which is what makes the arithmetic above a real deployment rather than a thought experiment.
- Knowledge distillation. Training the small model against a larger teacher's output distribution rather than only against hard labels. Still used, but less central than it was — the strongest small models are now trained from scratch to be small.
- Architectural efficiency. Grouped-query attention, sliding-window attention and similar reduce the KV cache, which is what actually breaks on-device long context: the weights are a fixed cost, but the cache grows with every token in the conversation.
Real-World Applications
The category is defined by shipped models rather than by research, and the specific deployments show what the size buys.
Meta's MobileLLM-Pro is a 1B model from Reality Labs with a 128k context window and near-lossless INT4 quantisation, built for on-device inference. The 128k figure is the interesting one: it says the target is processing a long local document — an email thread, a transcript — without any of it leaving the device.
Liquid AI's LFM2.5 family targets agentic loops and data extraction at 350M and 1.2B, explicitly for edge deployment, with a sub-500 MB footprint at the 350M size. The stated use cases are extraction and tool-calling loops rather than conversation, which is the honest positioning for the size.
Google's Gemma 4 spans 2.3B to 30.7B under Apache-2.0. Only the bottom of that range is an SLM by the definition here, and its inclusion in the same family as a 30.7B model is a reminder that "small" is a deployment claim, not a brand.
The workloads these are actually used for share a shape: narrow, high-volume, latency-sensitive, or privacy-constrained.
Extraction — pulling structured fields out of a document — is the archetype, because the task is well-defined, the output is short, and running it a million times through a frontier API would cost more than the value extracted. Classification and routing follow the same logic; a small model deciding which of six queues a support ticket belongs in does not need frontier reasoning. Autocomplete and on-device assistants need latency below the threshold where a person notices, which a network round trip cannot guarantee. And anything touching health records, private messages or camera frames has a regulatory reason to never leave the device at all.
Challenges
The capability ceiling is real and is not closing. A 1B model is not a slightly worse frontier model; it is a different tool. Multi-step reasoning, long-horizon planning and tasks requiring broad world knowledge degrade sharply, and no amount of training tokens fixes a capacity limit. The correct response is task selection, not optimism.
Long context is a cache problem, not a weights problem. The memory table above covers weights only. A conversation's KV cache grows linearly with tokens and can exceed the model itself — which is how a 1B model advertised with 128k context runs out of memory at 20k tokens on a real phone. Check the cache arithmetic separately; the parameter count will not warn you.
Quantization damage is uneven and hard to detect. "Near-lossless" is measured on benchmarks. INT4 damage concentrates in low-frequency behaviour — rare languages, unusual formats, precise arithmetic — which is exactly what an aggregate score averages away. A model that scores within a point of its FP16 version can be materially worse at the one thing you need.
Thermals and battery bound sustained use. The bandwidth calculation above assumes the memory subsystem runs at full rate. A phone generating tokens continuously heats up and throttles, so the tokens-per-second a device demonstrates for ten seconds is not what it sustains for ten minutes.
Fragmented deployment targets. A datacentre model targets CUDA. An on-device model targets some combination of Apple Neural Engine, Qualcomm Hexagon, Arm CPUs and a dozen runtime formats, each with different quantization support. The engineering cost of shipping is frequently larger than the cost of the model.
Future Trends
The trend most likely to matter is the one already visible in the training budgets: token counts continue past any point scaling laws would recommend, because the economics of inference-optimal training only get more favourable as deployment volume grows. Expect the ratio of training tokens to parameters to keep rising at the small end, and expect the returns to keep diminishing without stopping.
The second is that the boundary of the category is set by consumer hardware, and consumer memory bandwidth improves slowly — much more slowly than datacentre bandwidth. If phone bandwidth roughly doubles over several years, the ceiling moves from about 3B to about 6B at INT4. That is a modest, predictable expansion, not a step change, which means "small" will keep meaning roughly what it means now for longer than the pace of the rest of the field would suggest.
The third is architectural: sparse and mixture-of-experts approaches break the assumption behind the bandwidth arithmetic, because they read only a fraction of weights per token. An MoE model with 8B total and 1B active parameters has the memory footprint of an 8B model and the bandwidth cost of a 1B one — attractive on a laptop with spare RAM, useless on a phone that cannot hold the full model at all. Which constraint binds decides whether that trade helps.
Note: this term was last reviewed in August 2026. The specific models named, and the memory-bandwidth figures for current consumer hardware, are stated as of that review. The arithmetic — bytes per parameter, bandwidth divided by model size, and 6ND — does not age.