Definition
Edge AI means running the model on the device that produced the data — a phone, a security camera, a car, a factory sensor — instead of sending that data to a server and waiting for the answer to come back. You accept a hard ceiling on model size, compute and power that a data centre does not have, and in exchange you get four things the network cannot sell you at any price: a reaction fast enough to be useful, freedom from a permanent uplink bill, raw data that never leaves the device, and a system that keeps working when the link is down.
The reaction-time argument, worked. A camera running at 30 frames per second delivers a frame every 1/30 of a second — 33.3 ms. If something is supposed to act on each frame (stop a conveyor, brake, reject a part), the whole decision has to fit inside that window, or frames arrive faster than they are cleared. Now price the round trip: Ookla's 2025 Speedtest Intelligence data put median multi-server latency on US mobile networks at 50.5 ms, which is why it reported the US as missing its own 50 ms threshold for text-based AI workloads. That is 1.5 frame periods, spent before a byte of the frame has been uploaded, before the model has run, and before the answer has come back. The budget is gone before the work starts. Crucially this is not the speed-of-light problem that cloud deployment has — distance is a minor term here and the radio access network dominates — so moving the data centre closer does not fix it. Running on the device gives the same decision the full 33.3 ms.
The bandwidth argument, worked. A 1080p camera at 30 fps encoded in H.264 typically runs about 4 Mbit/s (vendor sizing guides put the usual range at 3–5 Mbit/s). A hundred of them streaming continuously is 400 Mbit/s of sustained uplink, and across a 30-day month — 400 × 10⁶ bit/s × 2,592,000 s ÷ 8 — that is about 130 TB. Run detection on site instead and upload only what matters: at, say, 50 events per camera per day, each a 200 KB still with metadata, the month costs 150,000 × 200 KB ≈ 30 GB. Same cameras, same coverage, roughly 4,300× less traffic. This is why edge AI arrives in retail, logistics and utilities long before anyone there cares about latency — the uplink was the budget line.
The other two reasons need no arithmetic. Data that never leaves the device cannot be intercepted in transit or retained by a party you did not mean to trust, which is why face unlock, dictation and health sensing run locally — although privacy is something you still have to design, not something the architecture grants. And a device that must reach a server is offline whenever the link is: for a vehicle in a tunnel, a sensor on a remote pipeline, or a machine on a factory floor during a switch failure, that is a failure rather than degraded service.
How It Works
The chain of constraints runs memory → compute → power, and memory binds first: the weights must fit in the device's RAM before throughput or watts are even a question. That is why nearly every edge project begins by making the model smaller rather than the hardware bigger.
The standard pipeline is train big, then shrink. A model is trained normally on server hardware, then compressed for the target through quantization (storing weights at 8 or 4 bits instead of 32), pruning, and knowledge distillation into a smaller student. The result is converted to a runtime the device can execute — LiteRT (Google's successor to TensorFlow Lite), ONNX Runtime, or ExecuTorch for PyTorch — and dispatched to whatever accelerator is present: a CPU, a GPU, or increasingly a dedicated NPU.
Two things surprise people arriving from server inference. The first is that memory bandwidth, not arithmetic, is usually the ceiling: at batch size 1 the device spends most of its time moving weights rather than multiplying them, which is exactly why halving the bit width buys latency and not merely footprint. The second is that peak throughput is not sustained throughput. A phone or a fanless board holds its top clock for a few seconds and then throttles to stay inside a thermal envelope it has no way to cool out of, so the number that predicts field behaviour is measured on a device that has already been running for ten minutes — never on the first run after a cold boot.
Almost all of this is inference only. Training needs several times the memory of inference to hold gradients and optimizer state, so it stays on servers. The significant exception is federated learning, introduced in the FedAvg paper: each device computes an update on its own data, only the update is uploaded, and a server averages the updates into a new model — so a fleet improves without any raw data leaving it.
Types
"Edge device" spans roughly five orders of magnitude of memory, and because that one number decides what class of model can run at all, it is the axis worth sorting by.
At the bottom is the microcontroller tier, the domain called TinyML. A Cortex-M class chip with about 256 KB of SRAM runs models measured in tens to hundreds of kilobytes: wake-word spotting, vibration anomaly detection, gesture recognition. The MCUNet paper opens by noting that microcontroller memory is two to three orders of magnitude smaller than a phone's, and still reaches above 70% ImageNet top-1 accuracy on an off-the-shelf commercial microcontroller. Power here is measured in milliwatts, so these parts run for months on a battery.
In the middle is the phone and single-board tier, where a few gigabytes of RAM and an NPU run real vision models and, increasingly, small language models. The arithmetic that decides membership: a 7-billion-parameter model at 4-bit precision needs 7 × 10⁹ × 0.5 bytes ≈ 3.5 GB for weights alone, before activations and the context window. That is comfortable on a recent flagship — and about 13,000× more memory than the entire microcontroller tier has.
At the top sit edge servers and automotive compute modules, with tens of gigabytes, running full multi-camera perception pipelines. These are "edge" only in the sense of being physically near the data rather than in a cloud region; their constraints resemble a workstation's far more than a sensor's, and most of the compression pressure above simply does not apply to them.
Real-World Applications
Wake-word detection is the cleanest case: "Hey Siri" and "OK Google" run continuously on a low-power always-on core, which is precisely why they work with the screen off and no signal. Streaming a microphone to a server around the clock would be untenable for both battery and privacy. The same handset runs computational photography, face detection and keyboard dictation on its NPU — Apple's Neural Engine and its Android equivalents — so raw camera and audio data never leaves it.
Driving is where the frame-budget arithmetic above stops being a nicety: Tesla's FSD computer and NVIDIA's DRIVE and Jetson modules run perception on board because a round trip is not survivable at road speed. Industrial visual inspection is the same argument at lower stakes — smart cameras from vendors such as Cognex and Keyence classify defects inside the work cell at line rate, keeping pace with a conveyor that will not wait for a server. And on the wrist, the Apple Watch's ECG and irregular-rhythm features analyse sensor data locally and emit an alert, instead of uploading a continuous record of someone's cardiac signal.
Challenges
Hardware fragmentation is the most underestimated cost. There is no single edge target. Every NPU vendor ships its own runtime, its own supported-operator list and its own quantization quirks, so a model that runs beautifully on one chip can fall back to the CPU on another because one layer is unsupported — and that fallback is often the difference between fitting the frame budget and missing it. Supporting a product range means maintaining a matrix of conversions and re-validating each one, and this, not the modelling, is where edge projects actually spend their time.
Accuracy loss has to be measured on the device, not in the notebook. Quantization does not degrade a model uniformly; it degrades particular inputs, often the rare ones you care about most. A quantized model with unchanged average accuracy can be meaningfully worse on the tail, and only evaluation against the real input distribution on the real runtime will show it.
The fleet is a deployment problem the cloud does not have. Shipping an improved model to devices you do not physically control means staged rollouts, version skew across hardware generations, and a rollback path for a model that turns out worse in the field — model deployment work that a cloud endpoint settles with one redeploy. Physical access adds a threat the cloud does not face at all: an attacker holding the device can extract the model, which for some companies is the intellectual property.
Privacy is not automatic. Keeping raw data local removes one risk and leaves several standing: a stolen device, an extractable model, and uploaded results that are themselves sensitive. An alert saying "person detected at this door at 03:14" carries much of the information the video did.
Future Trends
The clearest near-term shift is language models being sized to the phone tier rather than the server tier. Liquid AI's LFM2.5 reasoning model fits under 1 GB and Alibaba's Qwen 3.5 compact series starts at 0.8B parameters, which puts an offline assistant that never uploads a prompt comfortably inside the 3.5 GB budget above. Alongside that, NPUs are spreading from phones into laptops and PCs, which turns "does this device have an accelerator?" from a question into an assumption.
Two structural changes are worth watching. Hybrid routing — answering most requests on-device and escalating only the hard ones to the cloud — trades a little latency on the tail for privacy and cost on the bulk, and looks likely to become the default architecture rather than a special case. And the fragmentation problem has attracted an open-silicon answer: RISC-V-based designs such as Google's Coral NPU are an attempt to give the edge one compilation target instead of a vendor matrix.
Code Example
The core edge workflow is quantize, then run on a portable runtime. This uses ONNX Runtime, which targets CPUs, mobile and many accelerators from the same model file:
from onnxruntime.quantization import quantize_dynamic, QuantType
# Shrink FP32 weights to int8 — roughly 4x smaller, and less to move per inference
quantize_dynamic(
model_input="model.onnx",
model_output="model.int8.onnx",
weight_type=QuantType.QInt8,
)
import numpy as np
import onnxruntime as ort
opts = ort.SessionOptions()
opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
session = ort.InferenceSession(
"model.int8.onnx",
opts,
providers=["CPUExecutionProvider"], # swap for the device's accelerator EP
)
input_name = session.get_inputs()[0].name
frame = np.random.rand(1, 3, 224, 224).astype(np.float32) # one camera frame
outputs = session.run(None, {input_name: frame})
print(outputs[0].shape)
# Measure latency on a device that has already been warm for ten minutes, and
# compare it against the frame budget — 33.3 ms at 30 fps. A cold first run
# tells you nothing about sustained field performance.
Academic Sources
- "MCUNet: Tiny Deep Learning on IoT Devices" — Lin et al. (2020) — ImageNet-class accuracy on a commercial microcontroller; the reference point for TinyML.
- "Communication-Efficient Learning of Deep Networks from Decentralized Data" — McMahan et al. (2016) — introduces federated learning and FedAvg.
- "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications" — Howard et al. (2017) — the architecture family designed for the mobile tier.
- "Distilling the Knowledge in a Neural Network" — Hinton et al. (2015) — training a small deployable student from a large teacher.