Definition
Federated learning is a way to train one shared machine learning model across many separate devices or organizations without ever gathering their data into one place. Instead of copying everyone's data to a central server and training there, the server sends the current model out to the participants; each one trains on the data it already holds, and sends back only the changes to the model. The server averages those changes into an improved model and repeats. The raw data — your photos, a hospital's scans, a bank's transactions — never moves.
That inversion is the entire idea, and it exists because some data cannot be centralized at all. It may be too private (the words you type), too regulated (patient records under HIPAA or GDPR), too large to ship continuously (every phone's camera roll), or owned by rival institutions that will collaborate on a model but never hand over their customers. In all of these the ordinary recipe — pool the data, then train — is off the table before you start. Federated learning is what training looks like when the data is a fixed point and the model is the thing that travels.
The catch, and the reason the technique needed a paper rather than being obvious, is that this trades a data-movement problem for a communication-and-statistics problem. You now train over a network of unreliable participants whose data is nothing like a random sample of the whole, and every improvement to the model has to be shipped back and forth over that network. Most of the engineering in federated learning is about making that loop cheap enough and stable enough to converge.
How It Works
One pass through the loop is called a communication round, and a single round has four steps. The server picks a subset of available clients — you never wait for all of them — and sends each the current global model. Each selected client trains that model on its own local data for a while, producing a slightly different model. Each client sends its update back. The server aggregates the updates into a new global model, and the next round begins. Training is not one long process on one machine; it is hundreds or thousands of these short synchronize-train-report cycles.
The algorithm that made this practical is FederatedAveraging (FedAvg), introduced by McMahan and colleagues at Google in 2016. Its aggregation step is a weighted average: the new global model is the average of the returned client models, each weighted by the number of examples that client trained on, so a device with 10,000 samples counts ten times as much as one with 1,000. FedAvg is governed by three knobs — the fraction C of clients selected per round, the number of local epochs E each client trains before reporting, and the local minibatch size B.
The single most important of these is E, and understanding why explains the whole technique. The naive approach, sometimes called FedSGD, is to have each client compute one gradient on one minibatch and send it back — that is just ordinary distributed SGD with the data pinned in place, and it needs a network round trip for every single gradient step. FedAvg instead lets each client take many steps locally (several epochs over its own data) before it says anything. You do more computation between each expensive communication, so you need far fewer of them. In the original paper this cut the number of communication rounds to reach a target accuracy by 10–100× compared to synchronized SGD, across five model architectures and four datasets.
Why the round count is the whole cost, worked
The bill for federated training is almost entirely bytes on the wire, and it has a simple shape:
total traffic ≈ (rounds) × (clients per round) × 2 × (model size)
The factor of 2 is because every participating client both downloads the model and uploads an update of the same size.
Put numbers on it. Take a modest model of 1 million parameters stored at 4 bytes each — that is 4 MB. A client that joins a round downloads 4 MB and uploads 4 MB, so it costs 8 MB per round. Sample 100 clients per round and one round moves 100 × 8 MB = 800 MB.
Now the round count decides everything. Suppose reaching the target accuracy takes 600 rounds the FedSGD way (one gradient per round) but 20 rounds with FedAvg (many local epochs per round) — a 30× reduction, squarely inside the paper's 10–100× range. The wire cost is:
- FedSGD: 600 × 800 MB = 480 GB
- FedAvg: 20 × 800 MB = 16 GB
Same model, same 100 clients, same 800 MB per round — the only thing that changed is how much work each client did before opening its mouth, and it moved the total by 30×. This is why local computation is treated as free and communication as the scarce resource: doubling the local epochs is a rounding error on a phone's battery, while every extra round is another 800 MB across a metered, flaky mobile network.
It is worth being honest about what this arithmetic does not show. Federated learning is not automatically cheaper on bytes than centralizing — if you have only a handful of clients with little data each, shipping their raw data once can be smaller than 16 GB of repeated model exchange. The bytes are a cost federated learning pays, not a prize it wins. What it wins is that the raw data stays put; the communication budget is the price of that guarantee, and FedAvg's job is to keep the price down.
Types
Federated learning splits into two genuinely different regimes, a distinction drawn sharply in the 2021 survey Advances and Open Problems in Federated Learning (Kairouz et al.). They share the loop above but almost nothing else about their engineering.
Cross-device federated learning spans a huge, shifting population of small, unreliable clients — think millions of phones. Any given device holds a tiny slice of data, is available only when it is idle, charging and on Wi-Fi, and may drop out mid-round. You can never assume the same devices return, so the system samples whoever is available and is built to tolerate massive dropout. This is the setting FedAvg was designed for.
Cross-silo federated learning spans a small number of large, reliable participants — a consortium of hospitals or banks, perhaps two to a hundred organizations. Each silo holds a substantial, well-curated dataset and stays online for the whole training run, so stragglers and availability matter far less. Here the hard problems are governance and trust between institutions that are often competitors, and the statistical gap between one hospital's patient population and another's, rather than raw device flakiness.
Real-World Applications
Federated learning is not a lab curiosity; it runs in production on devices you may be holding. Google introduced it in 2017 for Gboard, its mobile keyboard, to improve next-word prediction, query suggestions and emoji ranking — the model learns from what people actually type without any of those keystrokes being uploaded. Apple uses federated learning with differential privacy for similar on-device features such as QuickType and Siri personalization. These are the canonical cross-device deployments, and they are why the technique was invented: keyboard text is exactly the kind of data that is both enormously useful for training and completely unshippable.
In healthcare, the cross-silo case, the standout example is the EXAM model reported in Nature Medicine in 2021 (Dayan et al.), which predicted the oxygen needs of COVID-19 patients by training across 20 hospitals worldwide using NVIDIA's federated framework. No institution's patient data left its own firewall, yet the collaboratively trained model generalized better than any model trained on a single hospital's data — a direct demonstration of the argument for federated learning, since patient records are precisely the data that regulation forbids pooling. Comparable cross-silo efforts run in finance, where the open-source FATE framework (started by WeBank) lets banks jointly train fraud- and risk-detection models without exchanging their customers' transactions. Dates on these deployments matter: they are stated as of 2026 and the specific products and frameworks will change, but the class of problem — useful data that legally or commercially cannot move — is permanent.
Key Concepts
Non-IID data and client drift. In a data center you shuffle the training set so every batch is a representative sample. In federated learning you cannot: each client's data is not independent and identically distributed with the population — your phone's photos are mostly of your life, one hospital skews toward its local demographics. Local training therefore pulls each client's model toward its own idiosyncratic optimum, and averaging those divergent models can zig-zag or stall, an effect called client drift. FedAvg is surprisingly robust to it but slows down under it, and algorithms such as FedProx and SCAFFOLD add correction terms that keep local updates from wandering too far from the shared model.
Updates leak information — the loop is not private on its own. It is tempting to assume that because the raw data stays home, the system is private. It is not. A model update is a function of the data that produced it, and that function can be inverted: the 2019 Deep Leakage from Gradients work reconstructed pixel-level training images from nothing but the shared gradients. So a curious or compromised server can learn about a client's data from its update alone.
Two defenses close that gap, and production systems use both. Secure aggregation (Bonawitz et al., 2017) is a cryptographic protocol that lets the server compute only the sum of many clients' updates while learning no individual update — participants mask their updates with random values that cancel out once enough are added together. Differential privacy goes further by adding calibrated noise so that the trained model provably cannot depend too much on any single participant's data; the differentially private variant of the training loop is often called DP-FedAvg. A HowAIWorks glossary page for differential privacy does not yet exist, but the idea is the mathematical guarantee that one person's presence or absence in the data changes the outcome by at most a bounded amount. Neither defense is free — secure aggregation adds coordination overhead and differential privacy costs accuracy — which is why they are engineering choices, not defaults.
Challenges
Communication is the bottleneck, and it fights back. As the worked example showed, every round is model-sized traffic times the client count, over networks that are metered, asymmetric (slow uploads) and intermittent. Shrinking the per-round cost with update compression or quantization helps, but compressing updates too aggressively degrades the very signal you are trying to aggregate, so there is a real floor.
Non-IID data hurts convergence in ways you cannot shuffle away. This is the deepest problem, because the fix used everywhere else — randomize the data — is exactly what federated learning forbids. Skewed local distributions slow training and, in the worst cases, produce a global model that serves the majority of clients well and a minority badly, which is a fairness problem the central-training world rarely has to think about.
Systems heterogeneity and stragglers. In cross-device settings the clients are wildly unequal — a flagship phone and a five-year-old budget device, on Wi-Fi or on a weak cellular link. A round can only advance as fast as the participants it waits for, so slow or dropped clients (stragglers) stall progress, and dropout has to be tolerated rather than prevented.
Verifying updates you cannot see. The privacy that protects honest clients also shields malicious ones. A participant can send a poisoned update to corrupt the shared model — a federated analogue of data poisoning — and secure aggregation, by hiding individual contributions, makes such an attack harder to detect. Robust aggregation and anomaly checks help, but there is a genuine tension between the privacy guarantees and the ability to police what enters the model.
Future Trends
The direction with the most momentum is federated fine-tuning of large models: rather than train a network from scratch across devices, adapt a pretrained model to on-device data using a parameter-efficient method so that only a small set of adapter weights — kilobytes or a few megabytes rather than the whole model — has to be communicated each round. That directly attacks the communication cost that dominates everything above, and it fits the reality that most useful models now start from a large pretrained base. Alongside it, the pressure from data-protection regulation continues to push cross-silo federated learning into regulated industries, where "the data cannot leave our premises" is a hard legal fact rather than a preference — which is the exact condition federated learning was built for.
Code Example
The one piece worth seeing in code is the aggregation step, because it is where "federated" actually happens — everything else is ordinary local training. This is the FedAvg server update: a weighted average of the client models, each weighted by how many examples that client trained on.
def federated_average(client_updates):
"""
client_updates: list of (weights, num_examples) pairs, one per client.
`weights` is a list of parameter arrays (the client's trained model).
Returns the new global model, sample-weighted per FedAvg.
"""
total_examples = sum(n for _, n in client_updates)
# Start an accumulator shaped like the model, filled with zeros.
first_weights = client_updates[0][0]
global_weights = [[0.0 for _ in layer] for layer in first_weights]
for weights, n in client_updates:
share = n / total_examples # this client's data weight
for layer_idx, layer in enumerate(weights):
for i, value in enumerate(layer):
global_weights[layer_idx][i] += value * share
return global_weights
The important detail is share = n / total_examples: a client that trained on 10,000 examples moves the global model ten times as much as one that trained on 1,000, which is what "weighted by data" means. Note also what the server never touches — it sees only each client's weights and its example count n, never a single training example. In a real deployment this same sum is what secure aggregation hides behind masking, so the server computes exactly this weighted total while learning no individual client's contribution.