Term tag: Quantization

Low-Rank Adaptation (LoRA)

LoRA freezes a model's weights and trains two thin matrices beside each one: 98,304 numbers instead of 151 million, and no added inference latency.

Model Compression

Making a trained model smaller and cheaper to run — quantization, pruning, distillation and low-rank methods, and which one your problem actually needs.

Model Size

Model size is a model's parameter count. What that one number predicts — memory, cost per token, capability — worked on real models from 110M to trillions.

Neural Processing Unit (NPU)

An NPU is a fixed-function matrix engine inside a phone or laptop chip that runs AI models at a few watts — and why TOPS is the wrong number to…

Performance

AI performance is not one number but a frontier of quality, speed, and cost—improving one axis (via quantization or batching) usually trades off another.

Quantization

Running a model in fewer bits — FP16, FP8, INT4 — to cut memory and cost, what accuracy it actually costs, and why a memory-bound model also gets faster.

Small Language Model (SLM)

A language model small enough to run on the device that uses it — roughly under 5B parameters. Trained far past compute-optimal to buy cheap inference.