Definition
A Neural Processing Unit (NPU) is a block of fixed-function silicon, sitting on the same chip as your CPU and GPU, that does one job: multiply matrices at very low power. It cannot run your operating system, render a game or open a spreadsheet. It exists because the arithmetic inside a neural network is repetitive enough that you can delete almost all the machinery that makes a processor programmable and hard-wire the multiplications instead — and what you get back for giving up that generality is operations per watt.
That is the whole trade, and it is the answer to the question most people arrive with: my new laptop says it has an NPU — what is it for? It is for the AI features that have to run continuously without flattening your battery. Background blur on a video call, wake-word detection, live captions, noise suppression, face unlock, the photo pipeline in your camera app. None of those are impressive feats of computation. All of them would be intolerable if they cost you three hours of runtime, and that is the problem the NPU solves.
The number on the sticker — "40+ TOPS", "38 trillion operations per second" — is the least useful thing about it, and a large part of this page is about why. TOPS is a peak arithmetic rate under assumptions the vendor chooses and rarely prints. The figures that decide what your device can actually do are operations per watt and memory bandwidth, and hardly any consumer spec sheet quotes either.
How It Works
What is actually inside one
Strip the marketing off and an NPU is a grid of multiply-accumulate units — MACs — surrounded by memory. A MAC does a × b + c, which is the only operation a matrix multiplication is made of. Everything else on the die exists to keep that grid fed.
The grids come in two broad shapes, and the distinction is the real answer to the npu architecture query. In a systolic array, the design Google's TPU made famous, weights and activations are pumped through a fixed lattice of MACs in step, each cell handing its partial sum to its neighbour; one control decision drives the entire array, and data is reused inside the grid rather than fetched again from memory. In a SIMD vector engine, closer to how many mobile NPUs and DSP-derived designs work, a wide instruction applies the same operation across many lanes at once, which is less efficient per operation but far more tolerant of layer shapes the hardware designer did not anticipate. Most shipping NPUs are somewhere between the two, with additional hard-wired units bolted on for the things that surround a matrix multiply: convolution windowing, activation functions, pooling, and the rescaling that integer arithmetic requires.
The other axis worth knowing is integrated versus discrete, and on consumer devices it is effectively settled: the NPU is a block on the main system-on-chip, sharing the same memory controller as the CPU and GPU. It has no memory of its own. This turns out to matter more than anything about the MAC array, for reasons that arrive three sections down.
Why does deleting programmability buy so much? Because on a general-purpose core, fetching and decoding the instruction that orders an arithmetic operation costs far more energy than the arithmetic. The ASIC page works that figure through in detail; the short version is that a systolic array amortises one control decision across thousands of multiplications, and that ratio is where essentially all of the efficiency comes from. Not better transistors — fewer decisions.
What a TOPS figure actually counts
TOPS is trillions of operations per second, and for an NPU it is almost always this arithmetic and nothing more:
MAC units × 2 operations per MAC × clock frequency
The factor of two is because a multiply-accumulate is counted as two operations, a multiply and an add. You can verify the formula against a vendor's own datasheet. Arm publishes that the Ethos-U85 "scales from 128 to 2048 MACs, providing up to 4 TOPs of performance at 1 GHz", and quotes the bottom of that range as 256 GOP/s (Arm). Check both ends: 128 × 2 × 10⁹ = 256 billion operations per second, and 2,048 × 2 × 10⁹ = 4.1 trillion. The published numbers are the array size times the clock, exactly.
Run the same arithmetic backwards on Microsoft's Copilot+ PC requirement — an NPU "with the ability to run at 40+ TOPS" (Microsoft Learn) — and 40 TOPS at 1 GHz means roughly 20,000 MAC units. That is the entire content of the specification. It says how much silicon is dedicated to multiplication. It says nothing about whether any of it will be busy.
Now the part that makes cross-vendor comparison unsafe. A TOPS figure is quoted at some numeric precision, and narrower numbers mean more operations from the same transistors. NVIDIA's published H100 table is the cleanest demonstration — the H100 is a 2022 data-centre GPU, used here only as a worked example because the effect is a property of the number formats and holds for any chip — and every row describes one physical chip in one second (specs current as of July 2026, NVIDIA):
- FP16 Tensor Core: 1,979 teraFLOPS. The table's asterisk reads "with sparsity" — the figure assumes half the weights are zero and skipped, so the dense rate is about 990.
- FP8 and INT8 Tensor Core: 3,958. Halving the width of the numbers doubles the headline. Nothing about the silicon changed.
So the same H100, in the same second, performs either about 990 trillion operations or 3,958 trillion — a 4× spread — depending purely on which row you quote. Vendors of consumer NPUs face the identical choice and mostly do not print which one they made. Apple's M4 announcement says the Neural Engine is "capable of 38 trillion operations per second" and never states the precision (Apple). When two chips advertise 40 and 45 TOPS, you cannot conclude the second is 12% faster; you cannot conclude anything at all without knowing the precision and the sparsity assumption behind each.
And even a like-for-like number is peak. Achieved utilisation on real layers is set by the compiler, the operator coverage and the memory system, and on the workload people care about most it is catastrophically low — which is the next section.
The number that matters is TOPS per watt
Here is the comparison that explains why an NPU exists at all rather than a small GPU.
An H100 SXM — NVIDIA's 2022 Hopper data-centre part, again just a stand-in for the class — is specified at up to 700 W and 3.35 TB/s of memory bandwidth (NVIDIA, as of July 2026); its Blackwell-generation successors draw more still, so the gap below is a floor, not a ceiling. A MacBook Air 13-inch carries a 53.8 watt-hour battery and is rated for up to 15 hours of wireless web use (Apple). Divide: to reach that rating the entire machine — screen, radios, CPU, everything — must average 53.8 ÷ 15 ≈ 3.6 watts. Point a 700 W accelerator at that battery and it empties in 53.8 Wh ÷ 700 W = 0.077 hours, about four and a half minutes.
That is not a gap you close with a better cooling design. It is a factor of roughly 200 in sustained power, and it is why the on-device story is a different kind of hardware rather than a smaller version of the same hardware. The relevant question for a laptop or a phone is never "how many TOPS" but "how many TOPS can I get inside a budget of a few watts, sustained, with no fan" — and a part that answers that will always lose the headline-number contest to a data centre GPU while being the only one of the two that can do the job.
The consequence for a user is concrete. A feature that runs on the NPU at, say, 2 W costs you 2 watt-hours per hour of use out of a 53.8 Wh budget. The same feature pushed onto the integrated GPU at 15 W costs 15, and your battery life for that session drops by most of its length. This is precisely what Microsoft means when it describes Windows Studio Effects as delivering "battery-friendly AI effects that reduce the burden" on the rest of the chip (Microsoft Learn). The NPU is not producing an effect the GPU could not produce. It is producing it at a power level that lets it stay on for the whole meeting.
Memory bandwidth is the real ceiling
Now the thing almost every NPU spec sheet omits, and the reason a bigger TOPS number will not give you a faster on-device chatbot.
Generating text with a language model reads every weight in the model, once, for every single token. The arithmetic done with those weights is roughly two operations per parameter. That ratio — two operations per byte, at 8-bit weights — is so far below what any modern chip can sustain that the processor spends essentially all of its time waiting for memory. The memory wall page owns this argument in general; here is what it does to a laptop.
Take a 7-billion-parameter model quantized to 4 bits, which is a realistic on-device configuration. The weights occupy 7 × 10⁹ × 0.5 bytes = 3.5 GB. Apple publishes M5's unified memory bandwidth as 153 GB/s, up nearly 30% from M4 (Apple). The ceiling on generation speed follows in one division:
153 GB/s ÷ 3.5 GB per token ≈ 43.7 tokens per second
Nothing about the NPU appears in that calculation. Now look at what the matrix engine was doing during it: 2 × 7 × 10⁹ = 14 GFLOP per token, times 43.7 tokens per second, is about 0.61 TOPS of actual arithmetic — against the 40 TOPS the Copilot+ bar demands. The engine is 1.5% busy. Doubling it to 80 TOPS moves the token rate by nothing at all, because the token rate was never set by arithmetic.
This is also why on-device LLM work has drifted away from the NPU on some platforms. Apple's M5 press release, describing what accelerates local model runners like LM Studio, credits the new Neural Accelerators inside each GPU core and the higher memory bandwidth — while the Neural Engine is described in terms of the always-on intelligence features, and Apple quotes no TOPS number for it whatsoever. A vendor quoting GB/s and declining to quote TOPS is telling you which number binds.
Real-World Applications
Windows Copilot+ PCs are where the NPU is most visible as a product decision, because Microsoft made it a hardware gate: the on-device Windows AI features "require an NPU with the ability to run at 40+ TOPS", which is satisfied by Qualcomm Snapdragon X Elite, AMD Ryzen AI 300 series and Intel Core Ultra 200V series machines (Microsoft Learn). What actually runs there is Windows Studio Effects — background blur, eye contact, auto framing and voice focus applied at the camera and microphone level, so any application gets them without knowing they exist. Note the second requirement in Microsoft's own documentation: the hardware manufacturer must have shipped the Studio Effects driver. Silicon without the driver is silicon that does nothing.
Apple's Neural Engine has shipped in every iPhone since the A11 Bionic and in every Apple silicon Mac since the M1, and it is the clearest case of an NPU as an always-on part rather than a performance part. Apple's M5 pairs a 16-core Neural Engine with 153 GB/s of unified memory in the base configuration, rising to 614 GB/s on M5 Max (Apple). It is what runs Face ID, the computational photography pipeline, on-device dictation and the Apple Intelligence features that need to respond without a round trip to a server.
Arm's Ethos-U85 is the same idea two orders of magnitude further down. At 128 to 2,048 MACs it targets microcontroller-class devices — a doorbell camera, an industrial sensor, a hearing aid — where the entire power budget is milliwatts and "the cloud" is not a fallback because there is no radio worth the energy. Arm added native support for transformer-based networks to this generation, which tells you where even the smallest edge AI silicon expects the workload to go.
Phone camera pipelines are the largest deployment by volume and the one nobody thinks of as AI. Every frame in a modern viewfinder passes through several small networks — segmentation, denoise, tone mapping, face and scene detection — at 30 or 60 frames per second, continuously, while the phone is in your hand and getting warm. That workload is the shape NPUs were designed for: small models, fixed shapes, run forever, must not melt the phone.
Open edge NPU designs are starting to appear as well; Google's Coral NPU, announced as an open-source platform for always-on edge AI, is aimed squarely at the wearable and sensor end of the range where licensing a proprietary block is the main obstacle.
Key Concepts
A MAC array is the multiply-accumulate grid at the centre of the design; its size times the clock times two is the TOPS figure, and nothing else about the chip is in that number. Operator coverage is the set of neural network layer types the hardware and its compiler can actually execute — the single most important compatibility question and the one no spec sheet answers. Quantization is the conversion of a model to low-precision integers, which for most NPUs is mandatory rather than optional: Microsoft states plainly that many NPU devices "only support integer math in lower bit format, such as INT8", so "AI models need to be converted (or 'quantized') to run on the NPU".
The concept that ties the page together is arithmetic intensity — operations performed per byte moved. High-intensity work (a convolution over an image, processing a long prompt in one pass) keeps a MAC array busy and rewards more TOPS. Low-intensity work (generating one token at a time for one user) is bounded by memory bandwidth and is indifferent to TOPS. Almost every disappointment with an NPU is an intensity mismatch: the buyer purchased throughput for a workload that needed bandwidth.
Worth naming honestly: "NPU" is a marketing umbrella, not a technical category. Apple calls its block a Neural Engine, Qualcomm calls its Hexagon, Arm sells Ethos, Intel and AMD both ship things called NPUs, and the underlying designs differ in array topology, supported precisions and programming model. There is no NPU instruction set, no NPU standard and no portable NPU binary. Two chips advertising similar TOPS may have almost nothing in common.
Challenges
The TOPS you bought is not the TOPS you can reach. Peak throughput assumes the compiler maps your layers cleanly onto the array with the memory system keeping up. Real achieved utilisation on real models is a fraction of peak, and it varies by model, by layer shape and by vendor toolchain. There is no way to read it off a spec sheet — the only honest method is to run your model on the device and time it.
Unsupported operators fall back to the CPU, silently. This is the failure that surprises people most. When a graph contains a layer the NPU cannot execute, the runtime partitions it and runs the remainder elsewhere; ONNX Runtime's Qualcomm backend even exposes a disable_cpu_ep_fallback option specifically so developers can force an error instead of accepting the quiet downgrade (ONNX Runtime docs). Without it, a model can appear to work while running mostly on the CPU, burning the battery you bought the NPU to save, and the only symptom is that it is slower than you expected.
Quantization is not free, and it is compulsory. Because most NPUs execute only low-precision integers, a model has to be converted before it can run at all — and conversion can cost accuracy, particularly on layers with large outlier activations. That is engineering work with a real failure mode, not a build flag.
An NPU will not train anything. These are inference parts. They implement the forward pass in low-precision integer arithmetic and generally cannot represent gradients or hold optimiser state. Some platforms support limited on-device personalisation, but if a plan involves training on the NPU, the plan is wrong. Training still happens on GPUs and cloud accelerators.
Portability is poor and the toolchains are fragmented. A model optimised for one vendor's NPU is not a model that runs on another's; Qualcomm parts go through QNN, Intel parts through OpenVINO, Apple through Core ML. Runtimes like Windows ML now select the right backend automatically, which hides the fragmentation from the user but not from whoever has to validate the model on every target.
The memory system is shared, and the NPU is not the priority. Because it sits on the SoC with no dedicated memory, an NPU competes for bandwidth with the CPU, the GPU and the display. A workload that measures well in isolation can degrade badly when something else on the chip is busy — which is exactly the situation an always-on background feature lives in.
Future Trends
Bandwidth becomes the advertised number. The TOPS race is running into the wall described above, and vendors are already shifting emphasis: Apple's M5 announcement leads on a 153 GB/s memory figure and never states a TOPS figure for the Neural Engine at all. Expect on-device AI marketing to migrate from peak arithmetic toward memory bandwidth and model capacity, because those are what actually bound the workloads people now want.
Matrix engines are migrating into the GPU. Apple's M5 puts a Neural Accelerator inside each of its ten GPU cores and points local LLM runners at them, which blurs the line the word "NPU" was drawing. The likely settlement on consumer silicon is a division of labour rather than a winner: a small, extremely efficient fixed-function block for always-on perception, and matrix units inside the GPU for the bursty, bandwidth-hungry generative work.
Precision keeps narrowing below INT8. Four-bit and mixed 4/8-bit formats halve the bytes per weight, which — per the arithmetic above — roughly doubles the achievable token rate on a bandwidth-bound model. That is a far larger practical win than more MACs, and it is a safe thing to hard-wire because a number format changes much more slowly than a model architecture.
Transformer support is being designed in at the bottom of the range. NPU blocks specified during the convolutional era handle attention badly. Arm's Ethos-U85 explicitly adds native transformer support at microcontroller scale, and that retrofit is happening across the industry — a reminder that fixed-function silicon carries a bet about the workload that has to be re-placed every few years.
Runtimes are consolidating faster than the hardware. Windows ML choosing between the Qualcomm and Intel backends on the developer's behalf is the pattern: a single API on top of vendor execution providers. The hardware will stay fragmented; the intent is that the fragmentation stops being the application developer's problem.
Code Example
The argument that a bigger TOPS number does not buy a faster on-device chatbot is one division, so it is worth being able to run it for your own device. Substitute your machine's memory bandwidth and the model you want to run:
def on_device_ceiling(params_b, bits, bandwidth_gbs, npu_tops):
"""Token rate and NPU utilisation for single-user decoding.
Every weight is read once per token, so bandwidth sets the ceiling.
The arithmetic is ~2 FLOPs per parameter per token.
"""
bytes_per_token = params_b * 1e9 * bits / 8
tokens_per_s = bandwidth_gbs * 1e9 / bytes_per_token
ops_per_token = 2 * params_b * 1e9
used_tops = ops_per_token * tokens_per_s / 1e12
return bytes_per_token / 1e9, tokens_per_s, used_tops, used_tops / npu_tops
for bits in (16, 8, 4):
gb, tps, used, frac = on_device_ceiling(7, bits, bandwidth_gbs=153, npu_tops=40)
print(f"7B at {bits:>2}-bit weights {gb:5.1f} GB ceiling {tps:5.1f} tok/s"
f" arithmetic used {used:.2f} TOPS ({frac * 100:.1f}% of 40)")
Output:
7B at 16-bit weights 14.0 GB ceiling 10.9 tok/s arithmetic used 0.15 TOPS (0.4% of 40)
7B at 8-bit weights 7.0 GB ceiling 21.9 tok/s arithmetic used 0.31 TOPS (0.8% of 40)
7B at 4-bit weights 3.5 GB ceiling 43.7 tok/s arithmetic used 0.61 TOPS (1.5% of 40)
Three things fall out of it. Quantizing from 16-bit to 4-bit quadruples the ceiling, because it quarters the bytes read per token. The matrix engine never rises above 1.5% utilisation, so the TOPS rating is irrelevant across the whole table. And these are ceilings, not measurements — a real runtime lands below them once the KV cache, attention and scheduling overhead are counted. If your device is slower than this calculation says, the fix is a smaller or more aggressively quantized model, not a chip with a bigger number on the box.