Term tag: Performance Optimization

Concurrency

Concurrency is structuring a program so independent tasks interleave. How it differs from parallelism, and why serving a model is a queueing problem.

High Bandwidth Memory (HBM)

Stacked DRAM bonded beside the GPU die — the scarce component that gates AI accelerator supply, and why bandwidth rather than capacity is the constraint.

Inference Optimization

How a served LLM is made cheap: continuous batching, the prefill/decode split, and the throughput-versus-latency tradeoff behind every API price.

Memory Wall

Why LLM inference waits on memory rather than maths — arithmetic intensity, the roofline, and the one fact that explains HBM scarcity and the price of a token.

Parallel Processing

Running many computations at the same instant across many processing units. In AI it makes matrix multiplication fast, which is why GPUs train models.

Performance

AI performance is not one number but a frontier of quality, speed, and cost—improving one axis (via quantization or batching) usually trades off another.

Speculative Decoding

A small model drafts tokens and a large one verifies them in parallel — how LLM inference gets several times faster with mathematically identical output.

Vectorization

Replacing element-by-element loops with operations on whole arrays at once, so the work runs as SIMD instructions on a CPU or in parallel on a GPU.