Introduction
You type a sentence and press enter. A second later, words appear.
In between, your sentence was turned into numbers, those numbers were multiplied by a matrix a few thousand times, and somewhere a rack of processors read tens of gigabytes out of memory — for every single word that came back. Understanding that middle part explains almost everything else people argue about: why AI companies spend more on hardware than most countries spend on roads, why a long conversation costs more than a short one, why memory makers became AI winners, and why a chip's headline performance number is often the least interesting thing about it.
This is a tour of that middle part. Each stop links to a page that goes deeper.
Your Prompt Becomes Numbers
The model does not see letters. Your text is first cut into tokens — chunks of roughly a word, sometimes a fragment — by a process called tokenization. Each token is then looked up in a table and replaced by an embedding: a list of several thousand numbers that encodes what the token means in context.
From here on, nothing in the machine knows anything about language. There is only arithmetic.
The Model Is a Stack of Matrix Multiplications
A transformer is a tall stack of near-identical layers, and each layer is essentially a short sequence of matrix multiplications: the vector representing your token gets multiplied by a learned matrix, then another, then another.
The concentration is extreme. Work through one layer of a 70-billion-parameter model and you find about 1.7 billion arithmetic operations, of which more than 99.8% are matrix multiplication. Everything else — the normalisations, the activation functions, the softmax that everyone draws in diagrams — is a rounding error in the compute budget. For the whole model, one token costs roughly 139 billion operations.
This single fact is why AI chips look the way they do. NVIDIA's tensor cores and Google's TPU systolic arrays are not general-purpose accelerators that happen to be good at AI. They are hardware built around one operation, because one operation is where all the work is.
Why the GPU Spends Most of Its Time Waiting
Here is the fact that almost no coverage of the AI hardware boom mentions, and that explains most of it.
To produce one token, the machine must multiply your vector by every weight matrix in the model — which means it must read every weight out of memory. For a 70-billion-parameter model stored at one byte per weight, that is 70.6 GB, per token. An NVIDIA H100 reads memory at 3.35 TB/s, so the floor is about 21 milliseconds per token — a ceiling of roughly 47 tokens per second for a single conversation.
Now look at what the chip's arithmetic units were doing during those 21 milliseconds. The 139 billion operations they had to perform take about 0.07 milliseconds. The rest of the time they sit idle, waiting for weights to arrive.
The processor is at 0.34% utilisation while "generating fast". This is the memory wall, and once you see it, the whole industry snaps into focus. Compute has grown roughly 3× every two years for decades while memory bandwidth has grown about 1.6×, so the gap keeps widening. Nearly every optimisation described below is an attack on this one constraint.
The Cost of Remembering
A conversation is not a single token. To avoid recomputing the entire history for every new word, the model keeps a KV cache — the stored internal state of every token it has already seen.
That cache is not small. For a 70B model, it runs about 320 KiB per token, which means a single session at a 131,000-token context carries roughly 43 GB — comparable to the model's own weights. On an eight-GPU node holding a 70B model, after the weights are loaded there is room for only about eleven such long conversations at once.
This is why a long context costs money, why context limits exist at all, and why a million-token context is a memory problem long before it is an intelligence problem.
Making It Cheap
A provider serving that model has a small set of levers, and each one attacks the memory wall from a different angle.
Quantization stores each weight in fewer bits. Half the bytes means half the reading, so a memory-bound step runs roughly twice as fast — and the memory freed can hold more users.
Batching — the heart of inference optimization — serves many conversations in the same pass. The weights are read once and shared across everyone in the batch, so the cost of that expensive read is amortised. The effect is dramatic: on the same hardware, the cost of a million tokens falls from around $24 served one user at a time to about 51 cents at a batch of 384. But there is no free lunch — the individual user waits longer, and past a batch of roughly fifty, the caches of all those users start to outweigh the model itself and the returns collapse.
Speculative decoding is the most elegant of the three. A small model guesses the next several tokens; the large model checks them all in a single pass. It works only because of the memory wall: the big model was going to read all its weights anyway, so verifying five guesses costs about 1.4% more time than producing one — and the output is mathematically identical to what the big model would have said alone.
Test-time compute runs the other way. Reasoning models deliberately spend more compute per question, generating long internal chains of thought before answering. It buys accuracy on problems with checkable answers, and it is why the "training is saturating, so chip demand will fall" argument of 2024 turned out to be exactly backwards: the demand moved into serving, where it scales with usage rather than with a one-off training run.
The Machine That Made the Model
Everything above describes using a model. Making one is a different machine.
Training compute is counted in FLOPs, and the budget follows a simple rule: about six operations per parameter per token of training data. Meta's Llama 3.1 405B was trained on 15.6 trillion tokens, which works out to roughly 3.8 × 10²⁵ operations — a number now written into law, since the EU AI Act treats 10²⁵ as the threshold above which a model is presumed to carry systemic risk.
Why spend that? Because of scaling laws: the empirical finding that a model's error falls predictably as compute, data and size grow together. That set of curves — fitted to a few hundred training runs — is the entire quantitative case for the industry's capital spending. It is also more fragile than the spending implies, since the curves predict loss, not capability, and loss is not what anyone actually wants.
No single chip can hold such a run, so the work is split across tens of thousands of them — distributed training — using three kinds of parallelism at once, with the network, not the processor, usually setting the ceiling. And at that scale, hardware fails during the job: Meta reported 419 unexpected interruptions in 54 days of training, roughly one every three hours. Checkpointing is not hygiene at this scale. It is architecture.
Holding the whole thing together is CUDA, NVIDIA's software layer — and the real reason competitors with comparable silicon have struggled to take the market. The moat is not the transistors. It is two decades of libraries and the fact that every reference implementation in machine learning is written against them.
The Memory Beside the Die
Follow the chain to its end and you arrive at the physical bottleneck.
If generating tokens means reading weights, then bandwidth is what an accelerator is really selling, and bandwidth comes from high bandwidth memory: DRAM dies stacked vertically and bonded millimetres from the processor on a shared slab of silicon. It exists because of a hard limit — you cannot make a wire carry bits arbitrarily faster, so if you want more bandwidth you need more wires, and more wires means moving the memory into the package.
The clearest possible demonstration: the H100 and the H200 have the same arithmetic performance. The H200 simply has faster memory — and it generates about 43% more tokens per second. Same compute, more throughput, purely from bandwidth.
That is why HBM, not raw silicon, has been the scarce link in the chain, and why memory manufacturers found themselves at the centre of an AI boom. Bandwidth is the product.
Conclusion
The chain runs in one direction and every link constrains the next. Language becomes tokens; tokens become matrix multiplications; matrix multiplications require reading weights; reading weights requires bandwidth; bandwidth requires memory stacked next to the die; and that memory has to be manufactured by a very small number of firms with very long lead times.
Read it backwards and you have an explanation of the AI hardware market that does not depend on any forecast. Read it forwards and you have a reasonably complete answer to what happens in the second between pressing enter and seeing a word.
The pieces are all linked above. The one to read first, if you only read one, is the memory wall — it is the fact that most of the rest falls out of.
Where to go next
Four companion pieces take one link of this chain each and follow it all the way down.
What one AI query actually costs carries a single answer through every conversion — tokens to bytes to GPU-seconds to joules to cents — and then moves one dial at a time to show the answer swing by four orders of magnitude. It also settles the question of why a provider can sell you tokens for less than it would cost you to generate them yourself.
Training and inference are two economies explains why the machine that makes a model and the machine that runs it have opposite bottlenecks, and works out how much traffic it takes before serving a model costs more than training it did. The answer is the reason the "pre-training is saturating, so chip demand will fall" argument of 2024 was exactly backwards.
The AI chip supply chain is the map: who makes each link, why each has effectively one supplier, and what it would take for a second to exist. It is the page for anyone who arrived here from a stock chart and wants the mechanism rather than the ticker.
Why NVIDIA's moat is software takes the question asked every earnings season — why hasn't a rival with comparable silicon taken the market? — and answers it with the mechanism, including an honest account of what would actually have to be true for the moat to erode.
And if you want the chip comparison rather than the mechanism, the GPU, TPU and ASIC guide sits alongside this one.