Definition
A Tensor Processing Unit (TPU) is a chip that Google designed to do one job as fast as physically possible: the matrix multiplications that make up almost all of the arithmetic in a neural network. It is an application-specific integrated circuit (ASIC) — silicon etched for a single task rather than a general-purpose processor — so the answer to the question people most often type about it, is a TPU an ASIC?, is simply yes.
The trick that makes it fast is a systolic array: a grid of tiny multiply-and-add circuits that data flows through, so each number is handed across the grid and reused many times instead of being fetched from memory again for every operation. In the first TPU, described in Google's 2017 ISCA paper (Jouppi et al., In-Datacenter Performance Analysis of a Tensor Processing Unit), that grid was 256×256 — 65,536 multiply-accumulate units on one chip. That single design choice is why a TPU can be far faster and more energy-efficient at matrix multiplication than a CPU or a GPU — and, as the last section shows, it is also the source of everything the TPU is bad at.
How It Works
A neural network's forward and backward passes are, underneath, a long sequence of matrix multiplications. A general-purpose core does one such multiply-add by fetching two numbers, multiplying them, adding the result to a running total, and writing that total back — and on that kind of core the fetching and the bookkeeping cost far more energy than the multiplication itself. Do this billions of times and almost all the chip's energy goes to moving operands around, not to arithmetic. That is the problem the systolic array is built to delete.
The systolic array
Picture the 256×256 grid. Each of the 65,536 cells holds one weight and does one multiply-add per clock tick. Activations enter along one edge and flow across the grid; partial sums flow down it; every cell multiplies the value passing through by its stored weight, adds the partial sum arriving from above, and passes the result to the cell below. The word systolic is the analogy to a heartbeat — data is pumped through the array in rhythmic waves.
The point of this arrangement is reuse. A weight is loaded into its cell once and then multiplies every activation that streams past it; an activation, as it crosses the row, is multiplied by all 256 weights in that row without being read from memory again. Google's paper puts it plainly: the matrix unit "uses systolic execution to save energy by reducing reads and writes" of the on-chip buffer, because reading a large memory "uses much more power than arithmetic." One control decision drives the whole array for a whole wave of data. That is the opposite of a CPU, which issues a fresh instruction for essentially every operation.
Why this beats a CPU or GPU at matmul
The concrete win is in memory traffic. Multiply a 256×256 weight matrix by a batch of 256 input rows the naive way and you re-read both operands for every one of the ~33 million multiply-adds. Stream the same computation through a systolic array and you load the 65,536 weights once and feed in the activation rows — roughly 256× fewer operand fetches for that tile. The Code Example below runs that comparison. Cutting memory traffic by that factor is exactly what buys the efficiency, and it is the flip side of the memory wall: the reason general processors stall on this workload is that they keep going back to memory, and the systolic array's whole design is not going back to memory.
The numbers, and how they fit together
Every figure here is from the 2017 ISCA paper describing that first TPU (TPUv1):
- a 256×256 array of 65,536 8-bit multiply-accumulate units,
- clocked at 700 MHz,
- with a 28 MiB software-managed on-chip buffer feeding it,
- drawing about 40 watts in typical use.
Those numbers are not independent — the headline throughput falls straight out of them. Each MAC does one multiply and one add (counted as two operations) per cycle, so peak throughput is 65,536 × 2 × 700 MHz ≈ 91.8 trillion operations per second, which the paper rounds to its stated 92 TOPS (8-bit) peak. This is the same arithmetic an NPU datasheet uses — array size times clock times two — and it is worth internalising, because it also explains the TPU's ceiling: 92 TOPS is all the chip has, and any time the array is not full, that capacity is simply lost.
Types
The name "TPU" covers two genuinely different product lines, split by where the chip lives rather than by architecture:
- Cloud TPU is the datacenter part described in the paper and its many successors — the chip you rent in Google Cloud, wired together by the thousand into "pods" to train and serve large models. This is what people mean by "TPU" in nearly every context.
- Edge TPU is the small sibling: a low-power ASIC Google sells in its Coral boards and modules for running already-trained models on-device, where a cloud round-trip is too slow or too power-hungry. Same idea — fixed hardware for matmul — at a tiny fraction of the power.
That second line is where the TPU meets the NPU, and the comparison is worth getting right because it is easy to get wrong. Both a TPU and an NPU are ASIC-class accelerators built on arrays of multiply-accumulate units, and many NPUs are themselves systolic arrays — the architecture is not what separates them. The honest distinction is scale and market: the TPU is Google's cloud and datacenter ASIC (with the Edge TPU as its on-device offshoot), while "NPU" is the low-power inference block that ships inside phone and laptop chips from Apple, Qualcomm, Intel and others.
Real-World Applications
Google's own services were the reason the TPU was built, and the 2017 paper says so directly: the workloads it benchmarks — multilayer perceptrons, convolutional networks and LSTMs — "represent 95% of NN inference" in Google's datacenters. When you run a search query, a Google Translate request or an image search, a TPU is doing the neural network inference behind it. The chip exists because CPUs could not serve that volume of inference at an acceptable cost.
Training Google's frontier models. Google trains and serves its Gemini family of foundation models on its own TPU pods rather than on third-party GPUs, and DeepMind's AlphaFold protein-structure work was likewise trained on TPU infrastructure. These are among the largest training runs publicly attributed to any accelerator.
Anthropic's Claude models. Anthropic states that it trains and runs Claude across a mix of Google TPUs, AWS Trainium and NVIDIA GPUs, and has contracted for up to one million TPUs. It does not publish which TPU generation any individual Claude model was trained on — a useful reminder that a production model is usually spread across more than one kind of silicon.
The through-line is that the TPU shows up wherever the workload is overwhelmingly matrix multiplication at scale and the operator controls its own software stack (via JAX, TensorFlow or the XLA compiler). It does not show up as a drop-in accelerator for arbitrary code, because it cannot run arbitrary code.
Challenges
Every weakness below traces back to the same fact: a systolic array is rigid. It is a grid built to do one shape of computation, and it pays for its efficiency by being helpless at everything else.
A half-empty array is wasted silicon. The 92 TOPS peak is only reached when work streams through every cell every cycle. The moment the workload cannot keep the grid full, the missing capacity is gone — you do not get it back. Google's own paper measures this: on one convolutional benchmark, CNN1, the chip runs at just 14.1 TOPS while a differently-shaped convolutional net, CNN0, hits 86 TOPS on the exact same hardware. A 6× spread on one chip, driven entirely by whether the layer shapes happen to tile cleanly onto a 256×256 grid.
Memory-bound layers leave the array idle. The systolic array only helps when there is enough arithmetic per byte of weights fetched to keep it busy. The paper's roofline analysis puts that break-even ("ridge point") far to the right, at 1,350 operations per byte of weight memory. Workloads below that line — the paper finds its MLPs and LSTMs are memory-bound — are limited by how fast weights arrive, not by the array, so more MACs would buy nothing. This is the memory wall again, and it is why single-user text generation, which reads every weight per token for very little arithmetic, is a poor fit for a big matmul engine.
Sparse and irregular work fights the grid. A systolic array multiplies dense blocks. Skip-heavy computation — pruned or sparse models, mixture-of-experts routing, control-heavy or branchy code, anything with data-dependent structure — either has to be padded back into dense blocks (wasting the array on multiplying by zero) or falls off the accelerator entirely. The hardware bets that the workload is dense matmul; when that bet is wrong, the efficiency evaporates.
It only runs what the compiler can map. A TPU is not programmed instruction-by-instruction; your model is compiled (through XLA) into operations the array can execute. An operation the compiler cannot lower well runs slowly or not at all, and the tooling is narrower and less forgiving than the mature CPU/GPU ecosystem. Combined with the fact that Cloud TPUs are rented from one vendor rather than bought, this is real lock-in: code and performance tuned for a TPU do not simply move to other hardware.
Code Example
The two claims the page rests on — that the TPU's peak throughput is just its array size times its clock, and that the array's value is reusing operands instead of refetching them — are both one short calculation. Running it makes the systolic array's advantage concrete, and shows the peak matching the paper's 92 TOPS.
def systolic_array(dim, clock_hz):
"""Peak throughput of a square systolic array.
A dim x dim grid holds dim*dim multiply-accumulate cells. Each cell does
one multiply-add per cycle, counted as 2 operations, so peak ops/s is
just the cell count times the clock times two -- the formula an NPU
datasheet also uses.
"""
macs = dim * dim
peak_tops = macs * 2 * clock_hz / 1e12
return macs, peak_tops
for dim, clk in [(256, 700e6), (128, 700e6), (512, 700e6)]:
macs, tops = systolic_array(dim, clk)
print(f"{dim:>3}x{dim:<3} array = {macs:>6,} MACs "
f"peak = {tops:5.1f} TOPS at {clk / 1e6:.0f} MHz")
# Operand fetches for one 256x256 weight matrix times a 256-row input batch.
# Naive: re-read both operands for every multiply-add.
# Systolic: load the weights once, then stream the activation rows through.
B, dim = 256, 256
naive_fetches = 2 * B * dim * dim
reuse_fetches = dim * dim + B * dim
print(f"\noperand fetches, B={B}: naive {naive_fetches:,} "
f"systolic {reuse_fetches:,} -> {naive_fetches / reuse_fetches:.0f}x fewer")
Output:
256x256 array = 65,536 MACs peak = 91.8 TOPS at 700 MHz
128x128 array = 16,384 MACs peak = 22.9 TOPS at 700 MHz
512x512 array = 262,144 MACs peak = 367.0 TOPS at 700 MHz
operand fetches, B=256: naive 33,554,432 systolic 131,072 -> 256x fewer
The first line reproduces TPUv1's 92 TOPS from nothing but its 256×256 array and 700 MHz clock. The last line is the whole argument for the architecture: for one tile, the systolic schedule touches memory 256× less than the naive one. That factor is where the speed and the energy savings come from — and, read the other way, it is why a workload that cannot be arranged into dense tiles gets none of it.