Term tag: Efficiency

AI Energy Consumption

How much energy an AI query really uses, how training compares with serving, and how to read the numbers without believing either the hype or the panic.

Application-Specific Integrated Circuit (ASIC)

A chip wired for one job. Why Google builds TPUs and bitcoin abandoned GPUs — the volume-and-time bet behind custom silicon, with the break-even maths.

Diffusion Language Models (DLMs)

Text models that unmask many tokens per pass instead of writing one at a time — what that actually buys in speed, and what it costs in quality.

Distributed Training

How one model is trained across thousands of GPUs — data, tensor and pipeline parallelism, and why the network rather than the chip becomes the bottleneck.

FLOPs (Floating Point Operations)

The unit AI compute is measured in — how a training run's FLOP budget is counted, why FLOP and FLOPS are not the same, and why regulators use the number.

Inference Optimization

How a served LLM is made cheap: continuous batching, the prefill/decode split, and the throughput-versus-latency tradeoff behind every API price.

Knowledge Distillation

How a small model learns what a big one knows: the teacher's full probability distribution carries structure the hard labels do not — Hinton's dark knowledge.

Low-Rank Adaptation (LoRA)

LoRA freezes a model's weights and trains two thin matrices beside each one: 98,304 numbers instead of 151 million, and no added inference latency.

Mixture-of-Experts (MoE)

The architecture behind "671B total, 37B active": hundreds of expert sub-networks, a router that picks a few per token, and a memory bill that never shrinks.

Model Compression

Making a trained model smaller and cheaper to run — quantization, pruning, distillation and low-rank methods, and which one your problem actually needs.

Model Routing

Model routing sends each request to the cheapest capable model — a small model handles the easy majority and hard queries escalate to a frontier model.

Performance

AI performance is not one number but a frontier of quality, speed, and cost—improving one axis (via quantization or batching) usually trades off another.

Prompt Caching

Repeated prompt prefixes bill at a fraction of the input price. How prefix matching works, what silently breaks it, and where the break-even sits.

Quantization

Running a model in fewer bits — FP16, FP8, INT4 — to cut memory and cost, what accuracy it actually costs, and why a memory-bound model also gets faster.

Small Language Model (SLM)

A language model small enough to run on the device that uses it — roughly under 5B parameters. Trained far past compute-optimal to buy cheap inference.

Speculative Decoding

A small model drafts tokens and a large one verifies them in parallel — how LLM inference gets several times faster with mathematically identical output.