Privacy

Definition

In AI, privacy is the problem that a trained model can memorise and later reveal the specific people in its training data — or the users who query it — together with the mostly mathematical techniques that limit how much any one person's data can leak. It is not the generic goal of "keeping data safe." A large language model can be coaxed into reciting a phone number or an email address it saw once during training, so privacy here has a concrete, testable meaning: how much would the model's behaviour change if your record were quietly removed from the training set? If the answer is "not measurably," your data is private in the sense that matters.

That distinction is why this is a genuine technical topic and not a slogan. The failure it guards against is memorisation. Neural networks do not only learn general patterns; they also store fragments of individual examples, and the larger the model, the more it stores. Ship a chatbot fine-tuned on your support inbox and a curious user can prompt it into quoting a customer's message back verbatim, complete with the account number that was in it. Privacy in AI is the study of how that happens and how to make it provably rare.

How It Works

The starting point is the leak itself. Carlini and colleagues demonstrated it directly in 2021: using only query access to GPT-2 (the public 1.5-billion-parameter model), they generated 1,800 candidate sequences, filtered them for signs of memorisation, and confirmed 604 that were verbatim copies of the training data. Among them were 46 examples containing real individuals' names and 16 containing private individuals' contact details. Crucially, memorisation scaled with size — in one setting the 1.5B model reproduced over 18× as much content as the 124-million-parameter version. Bigger models are not just more capable; they are leakier by default. Two related attacks make the risk sharper: membership inference, where an adversary decides whether one person's record was in the training set (models tend to be more confident on data they have seen), and model inversion, where features of the training data are reconstructed from the model's outputs.

The main defence with a mathematical guarantee is differential privacy (DP). A randomised procedure is ε-differentially private if, for any two datasets that differ in a single person's record and any possible output, the probability of that output changes by at most a factor of exp(ε):

Pr[M(D) ∈ S] ≤ exp(ε) · Pr[M(D′) ∈ S]

The whole idea lives in that one exponential. ε is a privacy budget: it caps how loudly any single individual can influence the result. Make ε small and the bound is tight — at ε = 0.1, exp(0.1) ≈ 1.1, so no output can become more than about 10% likelier because you joined the dataset, and an attacker learns almost nothing about whether you are in it. Make ε large and the guarantee dissolves — at ε = 10, exp(10) ≈ 22,000, a bound so loose it protects nobody. You buy that small ε with noise: DP works by adding random perturbation calibrated to how much one record can shift the answer, so a smaller ε means more noise, blurrier statistics, and a less accurate model. There is no free lunch; privacy and utility trade against each other, and ε is the dial.

The second major technique attacks the problem from a different direction. Federated learning keeps the raw data where it is. Instead of shipping everyone's emails or medical records to a central server, the model is sent to the data: each phone or hospital trains on its own local records and returns only the resulting gradient update, which the server averages across participants. No raw example ever leaves the device. This shrinks the attack surface, but it is not private on its own — a gradient is a function of the data and can leak it back, and the averaged model can still memorise. In practice federated learning is paired with differential privacy (noise added to the updates) and sometimes secure aggregation, so the server sees only the sum of many masked contributions and never any individual's update in the clear.

Real-World Applications

Differential privacy is not a lab curiosity; it protects statistics that hundreds of millions of people appear in. The 2020 United States Census was published under a formal differential-privacy system: the Census Bureau injected calibrated noise into the tabulations specifically because linkage attacks had shown that "anonymised" tables from 2010 could be reconstructed down to individuals. Apple has used local differential privacy to gather usage statistics (emoji and typing data) from iOS devices while adding noise on the phone before anything is sent, and Google's RAPPOR did the same for Chrome telemetry years earlier.

Federated learning shows up wherever the data is too sensitive or too voluminous to centralise. Google's Gboard keyboard improves next-word prediction by training across millions of phones without the typed text ever leaving them. Hospital consortia use federated setups (for example on NVIDIA's FLARE framework) to train diagnostic imaging models across institutions that are legally barred from pooling patient scans, so the model learns from everyone's data while each hospital's records stay behind its own firewall. In all of these cases the point is the same: extract the statistical signal, leave the individual behind.

Key Concepts

• AI Safety: A model that regurgitates a user's private data is a safety failure, not only a compliance one — privacy leakage is one of the concrete harms a safety programme has to test for.
• AI Governance (AIG): Regulations such as the GDPR encode a "right to be forgotten," which forces a hard technical question governance cannot answer alone — how do you remove one person from a model that has already trained on them?
• Data Poisoning: The mirror image of privacy. Poisoning is an attacker writing into the training data to change the model; privacy is about the model reading out the training data to an attacker. Both exploit the same fact — that models absorb their training set more literally than people assume.

Challenges

The privacy–utility trade-off is unavoidable and often uncomfortable. Because a strong (small-ε) guarantee demands more noise, differentially private training routinely costs accuracy, and the loss falls hardest on the rare cases — under-represented groups and outliers are exactly the records DP is meant to hide and also the ones a model most needs signal from. Choosing ε is therefore a policy decision dressed as a hyperparameter, and there is no consensus on what value is "safe": deployed systems have used everything from ε below 1 to ε in the double digits.

Accounting for privacy over time is its own difficulty. Privacy budgets compose — every query or training epoch spends more of the budget, and the guarantees add up, so a model queried indefinitely eventually exhausts any fixed ε. Then there is deletion. When a user invokes a right to erasure, deleting their row from the database does nothing about the copy of it already baked into the trained weights; genuinely removing that influence is the open problem of machine unlearning, and the honest fallback — retraining from scratch — is often too expensive to do per request. Finally, federated learning and encryption-based methods carry real systems costs: extra communication rounds, slower training, and complex secure-aggregation infrastructure that can double or triple the engineering effort of an ordinary training pipeline.

Code Example

Differential privacy is easiest to feel as arithmetic. The snippet below implements the Laplace mechanism, the simplest DP building block: to release a count privately, add noise drawn from a Laplace distribution whose scale is the sensitivity (how much one person can change the count — here, 1) divided by ε. Watch how shrinking ε from 1.0 to 0.1 widens the noise and pushes the released answers further from the true count of 1000.

import random
import math

def laplace_noise(scale, rng):
    # Inverse-CDF sampling of a Laplace(0, scale) variable
    u = rng.random() - 0.5
    return -scale * math.copysign(1, u) * math.log(1 - 2 * abs(u))

def private_count(true_count, sensitivity, epsilon, rng):
    scale = sensitivity / epsilon          # noise grows as epsilon shrinks
    return true_count + laplace_noise(scale, rng)

true_count = 1000        # e.g. people in a dataset with some trait
sensitivity = 1          # one person joining/leaving changes the count by 1

for epsilon in (1.0, 0.1):
    rng = random.Random(0)
    scale = sensitivity / epsilon
    sample = [round(private_count(true_count, sensitivity, epsilon, rng), 1)
              for _ in range(5)]
    print(f"epsilon={epsilon}: noise scale b={scale:g}, "
          f"five noisy answers -> {sample}")

Running it prints:

epsilon=1.0: noise scale b=1, five noisy answers -> [1001.2, 1000.7, 999.8, 999.3, 1000.0]
epsilon=0.1: noise scale b=10, five noisy answers -> [1011.7, 1007.3, 998.3, 993.4, 1000.2]

At ε = 1 the released numbers sit within a point or two of 1000; at ε = 0.1 they scatter across a range of nearly twenty. That spread is the privacy: an attacker seeing 1007 cannot tell whether the true count is 1000, 1005, or 1010, and therefore cannot tell whether any particular person was counted. More noise, less certainty, more privacy — the same trade-off the ε in the formula above describes.