---
source: 'https://howaiworks.ai/blog/how-ai-runs-on-silicon'
section: blog
title: 'From Token to Transistor: What Happens When You Send a Prompt'
description: >-
  Follow one prompt through the machine — tokens, matrix multiplications, the
  memory that everything waits on, and the chips built around a single
  operation.
date: '2026-07-13'
author: HowAIWorks Team
tags:
  - AI Infrastructure
  - AI Hardware
  - GPU
  - Inference
  - LLM
  - Memory Bandwidth
  - AI Chips
  - NVIDIA
  - Efficiency
category: AI Fundamentals
readingTime: 12 minutes
---

# From Token to Transistor: What Happens When You Send a Prompt

> Follow one prompt through the machine — tokens, matrix multiplications, the memory that everything waits on, and the chips built around a single operation.

## Introduction

You type a sentence and press enter. A second later, words appear.

In between, your sentence was turned into numbers, those numbers were multiplied by a matrix a few
thousand times, and somewhere a rack of processors read tens of gigabytes out of memory — for every
single word that came back. Understanding that middle part explains almost everything else people
argue about: why AI companies spend more on hardware than most countries spend on roads, why a long
conversation costs more than a short one, why memory makers became AI winners, and why a chip's
headline performance number is often the least interesting thing about it.

This is a tour of that middle part. Each stop links to a page that goes deeper.

## Your Prompt Becomes Numbers

The model does not see letters. Your text is first cut into [tokens](https://howaiworks.ai/glossary/token) — chunks of
roughly a word, sometimes a fragment — by a process called [tokenization](https://howaiworks.ai/glossary/tokenization).
Each token is then looked up in a table and replaced by an [embedding](https://howaiworks.ai/glossary/embedding): a list
of several thousand numbers that encodes what the token means in context.

From here on, nothing in the machine knows anything about language. There is only arithmetic.

## The Model Is a Stack of Matrix Multiplications

A [transformer](https://howaiworks.ai/glossary/transformer) is a tall stack of near-identical layers, and each layer is
essentially a short sequence of [matrix multiplications](https://howaiworks.ai/glossary/matrix-multiplication): the
vector representing your token gets multiplied by a learned matrix, then another, then another.

The concentration is extreme. Work through one layer of a 70-billion-[parameter](https://howaiworks.ai/glossary/parameters)
model and you find about **1.7 billion arithmetic operations, of which more than 99.8% are matrix
multiplication.** Everything else — the normalisations, the activation functions, the softmax that
everyone draws in diagrams — is a rounding error in the compute budget. For the whole model, one
token costs roughly **139 billion operations**.

This single fact is why AI chips look the way they do. NVIDIA's tensor cores and Google's
[TPU](https://howaiworks.ai/glossary/tensor-processing-unit) systolic arrays are not general-purpose accelerators that
happen to be good at AI. They are hardware built around **one operation**, because one operation is
where all the work is.

## Why the GPU Spends Most of Its Time Waiting

Here is the fact that almost no coverage of the AI hardware boom mentions, and that explains most of
it.

To produce one token, the machine must multiply your vector by every weight matrix in the model —
which means it must **read every weight out of memory**. For a 70-billion-parameter model stored at
one byte per weight, that is **70.6 GB, per token**. An NVIDIA H100 reads memory at 3.35 TB/s, so
the floor is about **21 milliseconds per token** — a ceiling of roughly **47 tokens per second** for
a single conversation.

Now look at what the chip's arithmetic units were doing during those 21 milliseconds. The 139
billion operations they had to perform take about **0.07 milliseconds**. The rest of the time they
sit idle, waiting for weights to arrive.

**The processor is at 0.34% utilisation while "generating fast".** This is the
[memory wall](https://howaiworks.ai/glossary/memory-wall), and once you see it, the whole industry snaps into focus.
Compute has grown roughly 3× every two years for decades while memory bandwidth has grown about
1.6×, so the gap keeps widening. Nearly every optimisation described below is an attack on this one
constraint.

## The Cost of Remembering

A conversation is not a single token. To avoid recomputing the entire history for every new word,
the model keeps a [KV cache](https://howaiworks.ai/glossary/kv-cache) — the stored internal state of every token it has
already seen.

That cache is not small. For a 70B model, it runs about **320 KiB per token**, which means a single
session at a 131,000-token context carries roughly **43 GB** — comparable to the model's own
weights. On an eight-GPU node holding a 70B model, after the weights are loaded there is room for
only about **eleven** such long conversations at once.

This is why a long context costs money, why context limits exist at all, and why a million-token
context is a *memory* problem long before it is an intelligence problem.

## Making It Cheap

A provider serving that model has a small set of levers, and each one attacks the memory wall from a
different angle.

**[Quantization](https://howaiworks.ai/glossary/quantization)** stores each weight in fewer bits. Half the bytes means
half the reading, so a memory-bound step runs roughly twice as fast — and the memory freed can hold
more users.

**Batching** — the heart of [inference optimization](https://howaiworks.ai/glossary/inference-optimization) — serves many
conversations in the same pass. The weights are read once and shared across everyone in the batch,
so the cost of that expensive read is amortised. The effect is dramatic: on the same hardware, the
cost of a million tokens falls from around **$24 served one user at a time to about 51 cents at a
batch of 384**. But there is no free lunch — the individual user waits longer, and past a batch of
roughly fifty, the caches of all those users start to outweigh the model itself and the returns
collapse.

**[Speculative decoding](https://howaiworks.ai/glossary/speculative-decoding)** is the most elegant of the three. A small
model guesses the next several tokens; the large model checks them all in a single pass. It works
*only* because of the memory wall: the big model was going to read all its weights anyway, so
verifying five guesses costs about **1.4% more time** than producing one — and the output is
mathematically identical to what the big model would have said alone.

**[Test-time compute](https://howaiworks.ai/glossary/test-time-compute)** runs the other way. Reasoning models
deliberately spend *more* compute per question, generating long internal chains of thought before
answering. It buys accuracy on problems with checkable answers, and it is why the "training is
saturating, so chip demand will fall" argument of 2024 turned out to be exactly backwards: the
demand moved into serving, where it scales with usage rather than with a one-off training run.

## The Machine That Made the Model

Everything above describes *using* a model. Making one is a different machine.

Training compute is counted in [FLOPs](https://howaiworks.ai/glossary/flops), and the budget follows a simple rule: about
six operations per parameter per token of training data. Meta's Llama 3.1 405B was trained on 15.6
trillion tokens, which works out to roughly **3.8 × 10²⁵ operations** — a number now written into
law, since the EU AI Act treats 10²⁵ as the threshold above which a model is presumed to carry
systemic risk.

Why spend that? Because of [scaling laws](https://howaiworks.ai/glossary/scaling-laws): the empirical finding that a
model's error falls predictably as compute, data and size grow together. That set of curves — fitted
to a few hundred training runs — is the entire quantitative case for the industry's capital
spending. It is also more fragile than the spending implies, since the curves predict *loss*, not
capability, and loss is not what anyone actually wants.

No single chip can hold such a run, so the work is split across tens of thousands of them —
[distributed training](https://howaiworks.ai/glossary/distributed-training) — using three kinds of parallelism at once,
with the network, not the processor, usually setting the ceiling. And at that scale, hardware fails
*during* the job: Meta reported **419 unexpected interruptions in 54 days** of training, roughly one
every three hours. Checkpointing is not hygiene at this scale. It is architecture.

Holding the whole thing together is [CUDA](https://howaiworks.ai/glossary/cuda), NVIDIA's software layer — and the real
reason competitors with comparable silicon have struggled to take the market. The moat is not the
transistors. It is two decades of libraries and the fact that every reference implementation in
machine learning is written against them.

## The Memory Beside the Die

Follow the chain to its end and you arrive at the physical bottleneck.

If generating tokens means reading weights, then bandwidth is what an accelerator is really selling,
and bandwidth comes from [high bandwidth memory](https://howaiworks.ai/glossary/high-bandwidth-memory): DRAM dies stacked
vertically and bonded millimetres from the processor on a shared slab of silicon. It exists because
of a hard limit — you cannot make a wire carry bits arbitrarily faster, so if you want more
bandwidth you need *more wires*, and more wires means moving the memory into the package.

The clearest possible demonstration: the H100 and the H200 have the **same arithmetic performance**.
The H200 simply has faster memory — and it generates about **43% more tokens per second**. Same
compute, more throughput, purely from bandwidth.

That is why HBM, not raw silicon, has been the scarce link in the chain, and why memory
manufacturers found themselves at the centre of an AI boom. Bandwidth is the product.

## Conclusion

The chain runs in one direction and every link constrains the next. Language becomes tokens; tokens
become matrix multiplications; matrix multiplications require reading weights; reading weights
requires bandwidth; bandwidth requires memory stacked next to the die; and that memory has to be
manufactured by a very small number of firms with very long lead times.

Read it backwards and you have an explanation of the AI hardware market that does not depend on any
forecast. Read it forwards and you have a reasonably complete answer to what happens in the second
between pressing enter and seeing a word.

The pieces are all linked above. The one to read first, if you only read one, is the
[memory wall](https://howaiworks.ai/glossary/memory-wall) — it is the fact that most of the rest falls out of.

## Where to go next

Four companion pieces take one link of this chain each and follow it all the way down.

**[What one AI query actually costs](https://howaiworks.ai/blog/what-one-ai-query-costs)** carries a single answer through
every conversion — tokens to bytes to GPU-seconds to joules to cents — and then moves one dial at a
time to show the answer swing by four orders of magnitude. It also settles the question of why a
provider can sell you tokens for less than it would cost you to generate them yourself.

**[Training and inference are two economies](https://howaiworks.ai/blog/training-vs-inference-two-economies)** explains
why the machine that *makes* a model and the machine that *runs* it have opposite bottlenecks, and
works out how much traffic it takes before serving a model costs more than training it did. The
answer is the reason the "pre-training is saturating, so chip demand will fall" argument of 2024 was
exactly backwards.

**[The AI chip supply chain](https://howaiworks.ai/blog/ai-chip-supply-chain-chokepoints)** is the map: who makes each
link, why each has effectively one supplier, and what it would take for a second to exist. It is the
page for anyone who arrived here from a stock chart and wants the mechanism rather than the ticker.

**[Why NVIDIA's moat is software](https://howaiworks.ai/blog/why-nvidias-moat-is-software)** takes the question asked every
earnings season — why hasn't a rival with comparable silicon taken the market? — and answers it with
the mechanism, including an honest account of what would actually have to be true for the moat to
erode.

And if you want the chip comparison rather than the mechanism, the
[GPU, TPU and ASIC guide](https://howaiworks.ai/blog/tpu-gpu-asic-ai-hardware-market-2025) sits alongside this one.

## Frequently Asked Questions

### What actually happens when you send a prompt to an AI model?

Your text is split into tokens, each token becomes a vector of numbers, and those vectors are pushed through a long chain of matrix multiplications — one pass per token generated. More than 99% of the arithmetic is matrix multiplication, which is why AI chips are built around that one operation.

### Why do AI models need such expensive chips?

Not primarily for the arithmetic. Generating a token requires reading the model's entire set of weights out of memory, so the speed of an answer is set by memory bandwidth rather than by compute. The chips are expensive because fast memory bonded next to the processor is expensive and scarce.

### Why does a longer conversation cost more?

Because the model keeps a cache of every previous token's internal state so it does not recompute the past. That cache grows with the length of the conversation and with the number of users served at once, and it competes with the model's own weights for the same memory.

### Is an AI chip's FLOPS number the thing that matters?

For training, largely yes. For generating text one token at a time, mostly no — that workload waits on memory. Two accelerators with identical FLOPS can differ by tens of percent in tokens per second purely because one has faster memory.

---

Source: https://howaiworks.ai/blog/how-ai-runs-on-silicon — HowAIWorks.ai