Term tag: AI Infrastructure

Advanced Packaging

CoWoS, chiplets and 3D stacking — how compute dies and memory become one accelerator, and the assembly step that has actually gated AI chip supply.

AI Architecture

How a production AI system is assembled: retrieval, context assembly, the model call, tool execution, validation, logging and evaluation.

AI Data Center

Why AI facilities are measured in megawatts rather than servers — power density, liquid cooling, and the grid connection that has become the real constraint.

AI Energy Consumption

How much energy an AI query really uses, how training compares with serving, and how to read the numbers without believing either the hype or the panic.

AI Infrastructure

Why AI clusters are designed around memory bandwidth and interconnect rather than FLOPs, and what that changes when you size a training run or a serving fleet.

Cloud Computing

Cloud computing is the on-demand delivery of computing power, database storage, applications, and other IT resources via the internet with pay-as-you-go.

Concurrency

Concurrency is structuring a program so independent tasks interleave. How it differs from parallelism, and why serving a model is a queueing problem.

Consensus Algorithm

A consensus algorithm makes separate machines agree on one value even when some fail. Why majorities always overlap, and why agreement costs a round trip.

CUDA

NVIDIA's software layer for GPUs — the programming model that turned graphics cards into AI hardware, and the real reason rivals with comparable silicon lose.

Disaggregated Serving (Prefill/Decode)

Disaggregated serving runs LLM prefill and decode on separate GPU pools because the two phases hit opposite limits: compute versus memory bandwidth.

Distributed Computing

Distributed computing splits work across separate machines that talk only by messages. Why partial failure, not speed, is the defining problem.

Distributed Training

How one model is trained across thousands of GPUs — data, tensor and pipeline parallelism, and why the network rather than the chip becomes the bottleneck.

EUV Lithography

The machines that print advanced chips with 13.5 nm light — how they work, why only one company makes them, and what that chokepoint means.

FLOPs (Floating Point Operations)

The unit AI compute is measured in — how a training run's FLOP budget is counted, why FLOP and FLOPS are not the same, and why regulators use the number.

Foundation Models

Not a marketing word for a big AI model: a foundation model is trained once on broad data and reused as the base for many downstream tasks.

GPU Computing

Why a graphics chip runs AI: a GPU spends its transistors on arithmetic instead of on making one thread fast — and what that trade wins and costs.

Grouped-Query Attention (GQA)

Grouped-query attention lets query heads share key/value heads, shrinking the KV cache that dominates long-context inference at near multi-head quality.

High Bandwidth Memory (HBM)

Stacked DRAM bonded beside the GPU die — the scarce component that gates AI accelerator supply, and why bandwidth rather than capacity is the constraint.

Inference

Running a trained model to get an answer. Training is paid once; inference is paid on every request — and it is where most AI money goes.

Inference Optimization

How a served LLM is made cheap: continuous batching, the prefill/decode split, and the throughput-versus-latency tradeoff behind every API price.

Interconnect (NVLink, InfiniBand)

The wiring that moves data between AI chips: on-package memory, NVLink inside a server, InfiniBand between them — and why each tier is far slower than the last.

KV Cache

The memory an LLM keeps so it never recomputes the past — why a long context costs money, and how many users one GPU can actually serve at once.

Machine Learning Operations (MLOps)

Why shipping a model is not like shipping software: the artifact has three inputs — code, data and trained weights — and only one of them lives in git.

Memory Wall

Why LLM inference waits on memory rather than maths — arithmetic intensity, the roofline, and the one fact that explains HBM scarcity and the price of a token.

Model Deployment

What has to happen between a model that trains well and users calling it: replica sizing, latency budgets, cold starts, and the rollback nobody built.

NVIDIA GPU for AI

Why AI training runs on NVIDIA rather than cheaper silicon — tensor cores, NVLink, and the software that decides how much of the datasheet you actually get.

Prompt Caching

Repeated prompt prefixes bill at a fraction of the input price. How prefix matching works, what silently breaks it, and where the break-even sits.

Quantum Computing (QC)

Quantum computing uses qubits — held in superposition and entangled — for exponential speedups on a narrow set of problems, not general computing.

Scalable AI

Scaling an AI system means three different things — bigger models, faster training, more concurrent users — and each axis hits a different hard limit.

Scaling Laws

The empirical curves that make AI capex rational: model loss falls predictably with compute, data and parameters — and what they do not promise.

Semiconductor Manufacturing

How an AI chip is actually made — wafers, process nodes, yield and the foundry model — and why supply cannot simply respond to a spike in demand.

Speculative Decoding

A small model drafts tokens and a large one verifies them in parallel — how LLM inference gets several times faster with mathematically identical output.

Test-Time Compute

Spending compute at inference instead of training — how reasoning models buy accuracy with thinking time, and why it moved chip demand into serving.

Token

The unit an AI model reads, writes and bills by — roughly 1.3 tokens per English word. Input, output and cached tokens are priced up to 10x apart.

Vector Database

A vector database stores embeddings with metadata, builds an approximate-nearest-neighbour index over them, and serves filtered similarity queries at scale.