Term tag: Inference

AI Architecture

How a production AI system is assembled: retrieval, context assembly, the model call, tool execution, validation, logging and evaluation.

AI Energy Consumption

How much energy an AI query really uses, how training compares with serving, and how to read the numbers without believing either the hype or the panic.

Classifier-Free Guidance (CFG)

Classifier-free guidance (CFG) is how a diffusion model follows a prompt: each step runs the model with and without the prompt and extrapolates the gap.

Context Window

Everything a model can see at once — why the advertised million tokens is not the usable number, and why you pay for the whole window on every turn.

Diffusion Language Models (DLMs)

Text models that unmask many tokens per pass instead of writing one at a time — what that actually buys in speed, and what it costs in quality.

Diffusion Model

A generative model that learns to undo a fixed noising process — what the network really predicts, why the step count is the cost, and why it displaced GANs.

Disaggregated Serving (Prefill/Decode)

Disaggregated serving runs LLM prefill and decode on separate GPU pools because the two phases hit opposite limits: compute versus memory bandwidth.

Edge AI

Edge AI runs the model on the device that captured the data instead of in the cloud — what that buys in latency, bandwidth and privacy, and what it costs.

FLOPs (Floating Point Operations)

The unit AI compute is measured in — how a training run's FLOP budget is counted, why FLOP and FLOPS are not the same, and why regulators use the number.

Grouped-Query Attention (GQA)

Grouped-query attention lets query heads share key/value heads, shrinking the KV cache that dominates long-context inference at near multi-head quality.

High Bandwidth Memory (HBM)

Stacked DRAM bonded beside the GPU die — the scarce component that gates AI accelerator supply, and why bandwidth rather than capacity is the constraint.

Inference

Running a trained model to get an answer. Training is paid once; inference is paid on every request — and it is where most AI money goes.

Inference Optimization

How a served LLM is made cheap: continuous batching, the prefill/decode split, and the throughput-versus-latency tradeoff behind every API price.

Knowledge Distillation

How a small model learns what a big one knows: the teacher's full probability distribution carries structure the hard labels do not — Hinton's dark knowledge.

KV Cache

The memory an LLM keeps so it never recomputes the past — why a long context costs money, and how many users one GPU can actually serve at once.

Memory Wall

Why LLM inference waits on memory rather than maths — arithmetic intensity, the roofline, and the one fact that explains HBM scarcity and the price of a token.

Mixture-of-Experts (MoE)

The architecture behind "671B total, 37B active": hundreds of expert sub-networks, a router that picks a few per token, and a memory bill that never shrinks.

Model Compression

Making a trained model smaller and cheaper to run — quantization, pruning, distillation and low-rank methods, and which one your problem actually needs.

Model Deployment

What has to happen between a model that trains well and users calling it: replica sizing, latency budgets, cold starts, and the rollback nobody built.

Model Routing

Model routing sends each request to the cheapest capable model — a small model handles the easy majority and hard queries escalate to a frontier model.

Model Size

Model size is a model's parameter count. What that one number predicts — memory, cost per token, capability — worked on real models from 110M to trillions.

Neural Processing Unit (NPU)

An NPU is a fixed-function matrix engine inside a phone or laptop chip that runs AI models at a few watts — and why TOPS is the wrong number to…

Parameters

The learned numbers inside a model — what '70 billion parameters' actually means, and how the count converts directly into the memory a GPU must hold.

Performance

AI performance is not one number but a frontier of quality, speed, and cost—improving one axis (via quantization or batching) usually trades off another.

Production Systems

A model serving live users under latency and uptime guarantees — judged at the 99th percentile under real load, not by the median on a laptop.

Prompt Caching

Repeated prompt prefixes bill at a fraction of the input price. How prefix matching works, what silently breaks it, and where the break-even sits.

Quantization

Running a model in fewer bits — FP16, FP8, INT4 — to cut memory and cost, what accuracy it actually costs, and why a memory-bound model also gets faster.

Reasoning Model

A model trained to think at length before answering. How it differs from a chat model, what the effort dial does, and the tasks where it earns its price.

Scalable AI

Scaling an AI system means three different things — bigger models, faster training, more concurrent users — and each axis hits a different hard limit.

Small Language Model (SLM)

A language model small enough to run on the device that uses it — roughly under 5B parameters. Trained far past compute-optimal to buy cheap inference.

Speculative Decoding

A small model drafts tokens and a large one verifies them in parallel — how LLM inference gets several times faster with mathematically identical output.

State-Space Models (Mamba)

State-space models are sequence models that scale linearly with length, not quadratically like attention. Mamba's input-dependent state made them competitive.

Structured Outputs (Constrained Decoding)

Constrained decoding masks a language model's token distribution against a JSON schema or grammar at each step, so the output is parseable by construction.

Tensor Processing Unit (TPU)

A TPU is Google's custom ASIC for the matrix multiplications in neural networks: a systolic array that reuses each value instead of refetching it.

Test-Time Compute

Spending compute at inference instead of training — how reasoning models buy accuracy with thinking time, and why it moved chip demand into serving.

Text Generation

Text generation is a language model producing text one token at a time: at each step it predicts a distribution over its vocabulary, picks a token, repeats.

Token

The unit an AI model reads, writes and bills by — roughly 1.3 tokens per English word. Input, output and cached tokens are priced up to 10x apart.