Bias in AI (Algorithmic Bias)

Definition

AI systems are biased because they learn from records of a world that was already unequal, and because the quantity they are trained to predict is almost never the quantity you actually care about — an arrest is not a crime, a medical bill is not an illness, and a past hire is not a good employee. What can be done about it is narrower and more concrete than the phrase "fix the bias" suggests: repair the label so it measures the thing you mean, repair the sample so the affected group is actually in it, and then choose which definition of fairness the system will satisfy — because you cannot have all of them at once. That last constraint is a proved mathematical result, not an engineering shortfall, and it is the single most useful thing to know about this subject.

Bias in this sense — algorithmic or social bias — means a systematic, unjustified difference in how a model treats groups of people: a CV ranker that scores women lower, a clinical risk score that under-refers Black patients at the same level of sickness, a face classifier whose error rate is forty times higher on one group than another. It is not random noise. A biased model is wrong in a consistent direction for a consistent set of people, which is why more data or a bigger model does not dissolve it.

Two other things are also called "bias" in machine learning, and neither is this page. Statistical bias is the systematic-error term in the bias-variance decomposition — how far a model's expected prediction sits from the truth because of the assumptions it makes; that lives in High Bias and Low Variance. The bias parameter is the learnable constant b in a neuron's wx + b, which shifts an activation function so it need not pass through the origin; that lives in Weights and Neurons. The three senses share a word and nothing else.

How It Works

Bias enters through the ordinary machinery of supervised learning, which is what makes it hard to see. A model is given examples and a target column, and it minimises error against that column. If the column encodes a historical decision — who was hired, who was arrested, who received care — its job is to reproduce that decision, and it does so faithfully. Nothing malfunctions. The bias is in the specification, not the bug list.

The proxy problem is the sharpest version of this. You rarely have a column for the thing you care about, so you use one that correlates with it. Hospitals do not record "how sick is this patient"; they record what was spent. Courts do not record "did this person commit a crime"; they record arrests. Every gap between the proxy and the target is inherited by the model at full strength, and if the size of that gap differs by group, the model's errors differ by group.

Deleting the protected attribute does not help, because the remaining features reconstruct it. This is called redundant encoding, and it is not subtle: postcode carries race in most segregated housing markets, purchase history carries pregnancy, sport played and college attended carry gender. A model with a few hundred correlated features will recover a dropped binary attribute to high accuracy. Removing the column mainly removes your ability to measure whether the model is discriminating.

The impossibility result

Here is the fact that reframes the whole subject. Three demands that all sound like the bare minimum for a fair classifier are mutually incompatible:

• Calibration within groups — a score of "high risk" means the same probability of the outcome whichever group you belong to.
• Equal false-positive rates — among people to whom the bad outcome never happened, the same share were wrongly flagged in each group.
• Equal false-negative rates — among people to whom it did happen, the same share were missed.

Kleinberg, Mullainathan and Raghavan (2016) and Chouldechova (2017) proved these cannot hold simultaneously when the base rate of the outcome differs between groups. The only exceptions are degenerate: a perfect predictor, or identical base rates.

Watch it happen on 2,000 people. Group A has a 30% base rate, group B a 60% base rate. The classifier is calibrated identically — 60% of everyone it flags does go on to have the outcome, in both groups — and its false-negative rate is identical too, missing half of the true cases in each group. Nothing here is tuned to disadvantage anyone:

An innocent member of group B is 3.5 times more likely to be wrongly flagged than an innocent member of group A, from a model that is equally calibrated and equally sensitive for both. The arithmetic is forced: with only 400 true negatives in group B against 700 in group A, the same 60% precision spreads a larger absolute number of false positives across a much smaller pool. Equalise the false-positive rate instead and you must break calibration, the false-negative rate, or both.

The consequence is practical rather than philosophical. "Make the model fair" is not a well-posed instruction. Someone — a regulator, a clinician, a product owner, not the model — has to decide which error the system is allowed to make unevenly, and own that decision in writing. This is why AI governance and accountability are not soft add-ons to the technical work; they are where the undecidable part gets decided.

Types

The useful taxonomy is not of biases but of the points in the lifecycle where one enters, because each entry point has a different repair. The five below follow the standard framework of harms across the machine learning lifecycle set out by Suresh and Guttag.

Historical bias is present even when the data is a perfect record. If 5% of a profession's practitioners were women during the sampling period, a model trained on that record will treat "woman" as evidence against membership — the data is accurate and the world it describes was unequal. No amount of extra sampling fixes this, because more data means more of the same world.

Representation bias comes from a sample that under-covers a group, so the model has too few examples to learn them well. This is the failure most amenable to a technical fix — resampling, targeted collection, data augmentation — and consequently the one teams reach for even when it is not the problem they have.

Measurement bias is the proxy problem above: the observed variable stands in for an unobserved one, and the substitution is worse for some groups. Spend as a proxy for need fails when a group faces barriers to access. Arrest as a proxy for offence fails where policing intensity differs.

Aggregation bias is fitting one model where the groups genuinely differ. The canonical example is clinical: the same HbA1c reading does not correspond to the same underlying blood-glucose control across ethnic groups, so a single diabetes model is miscalibrated for some patients even with a flawless dataset. The repair is separate models or group-aware features, not more data.

Deployment bias is a correct model used in a way it was never validated for — a risk score built to prioritise a waiting list, then read in court as a statement about a person. The monitoring question is not "is the model still accurate?" but "is it still being used for the thing it was measured on?"

Real-World Applications

Amazon's recruiting tool. Reuters reported in October 2018 that Amazon had abandoned an experimental system, in development since 2014, that scored candidates from one to five stars. It was trained on CVs submitted over the previous ten years, most of them from men, so it learned that male-associated patterns predicted being hired — penalising CVs containing the word "women's" and downgrading graduates of two all-women colleges. The system was scrapped because engineers could not guarantee it would not find some other route to the same conclusion. It is the clearest documented case that removing an attribute does not remove the bias.

The Optum healthcare risk algorithm. Obermeyer and colleagues, publishing in Science in 2019, dissected a commercial risk score used to select patients for extra care management, applied to roughly 200 million people a year in the United States. It predicted future healthcare cost as a stand-in for future health need. Because less is historically spent on Black patients at the same level of illness, the algorithm concluded they were healthier and referred fewer of them. Replacing the target with active chronic conditions raised the share of Black patients among those automatically referred from 17.7% to 46.5% — a textbook measurement bias, fixed at the label rather than the model.

COMPAS, and why both sides were right. In 2016 ProPublica reported that among defendants who were not rearrested within two years, Black defendants were about 1.9 times as likely as white defendants to have been labelled high risk. Northpointe, the vendor, replied that a given score carried the same probability of rearrest regardless of race. Neither party was lying. ProPublica was measuring error-rate balance, Northpointe was measuring calibration, and the base rates in the data differed — so by the theorem above, both claims could be true at once and no fix could satisfy both. The dispute is the impossibility result playing out in public, and it is the reason a fairness audit that does not name its criterion is not an audit.

Compliance work. The EU AI Act classifies employment, credit scoring and access to essential services as high-risk under Annex III — exactly the discrimination cases above — and will require providers to examine training data for possible biases, turning the choice of fairness criterion from a research question into a documented artefact someone signs, which is what the impossibility result implies it has to be. That duty is forthcoming rather than current: the Digital Omnibus agreed in May 2026 postponed the Annex III high-risk obligations to 2 December 2027, from an earlier 2 August 2026. (The Act's separate Article 50 transparency rules were not delayed and still apply from 2 August 2026 — a different layer from these high-risk duties.)

Challenges

You need the protected attribute to measure fairness, and often cannot collect it. Every fairness metric is computed per group, so it requires knowing who is in which group — yet race and ethnicity are special-category data under GDPR and are simply absent from most commercial datasets. Teams end up inferring the attribute in order to audit for it, which introduces its own errors.

Intersectional gaps hide inside per-attribute reports. Buolamwini and Gebru's 2018 Gender Shades audit of three commercial gender classifiers found a maximum error rate of 0.8% on lighter-skinned men and up to 34.7% on darker-skinned women — a 43-fold gap that a by-gender report and a by-skin-tone report would each have partly averaged away. Fairness checked one attribute at a time can pass while the intersection fails.

The fairness-accuracy trade-off is real but not universal. Where the target label is sound, constraining a model does cost accuracy. Where the label is a bad proxy, as at Optum, fixing it improves fairness and predictive quality together. Assuming the trade-off exists before checking the label is how teams talk themselves out of the cheapest available fix.

Feedback loops make the bias self-confirming. A model that directs attention — patrols, audits, inspections — changes where future data is collected. Send more officers to the areas with the most recorded arrests and you record more arrests there, which the next training run reads as confirmation. The model's error rate on its own logged outcomes looks excellent.

Generative models moved the target and kept the problem. For a classifier there is a confusion matrix to compute rates from. For a large language model writing a performance review or a loan rationale, there is no ground-truth column, so the group-conditional rates the theorems are stated over do not exist. Evaluation falls back on prompt suites and human raters, which are themselves sampled from somewhere. The mathematics of fairness is far better developed than the measurement that would let you apply it — see explainable AI and transparency for the tooling that partially fills the gap, and ethics in AI for the wider frame.