---
source: 'https://howaiworks.ai/glossary/bias'
section: glossary
title: Bias in AI (Algorithmic Bias)
description: >-
  Why AI systems produce unfair outcomes: how bias enters through data and proxy
  labels, and why no model can satisfy every definition of fairness at once.
tags:
  - fairness
  - AI Ethics
  - AI governance
  - accountability
  - Machine Learning
category: AI Ethics & Safety
datePublished: '2025-07-16'
lastUpdated: '2026-07-24'
---

# Bias in AI (Algorithmic Bias)

> Why AI systems produce unfair outcomes: how bias enters through data and proxy labels, and why no model can satisfy every definition of fairness at once.

## Definition

AI systems are biased because they learn from records of a world that was already unequal, and
because the quantity they are trained to predict is almost never the quantity you actually care
about — an arrest is not a crime, a medical bill is not an illness, and a past hire is not a good
employee. What can be done about it is narrower and more concrete than the phrase "fix the bias"
suggests: repair the label so it measures the thing you mean, repair the sample so the affected
group is actually in it, and then choose which definition of fairness the system will satisfy —
because you cannot have all of them at once. That last constraint is a proved mathematical result,
not an engineering shortfall, and it is the single most useful thing to know about this subject.

Bias in this sense — algorithmic or social bias — means a systematic, unjustified difference in
how a model treats groups of people: a CV ranker that scores women lower, a clinical risk score
that under-refers Black patients at the same level of sickness, a face classifier whose error rate
is forty times higher on one group than another. It is not random noise. A biased model is wrong
in a consistent direction for a consistent set of people, which is why more data or a bigger model
does not dissolve it.

Two other things are also called "bias" in machine learning, and neither is this page. **Statistical
bias** is the systematic-error term in the bias-variance decomposition — how far a model's expected
prediction sits from the truth because of the assumptions it makes; that lives in
[High Bias](https://howaiworks.ai/glossary/high-bias) and [Low Variance](https://howaiworks.ai/glossary/low-variance). The **bias parameter**
is the learnable constant `b` in a neuron's `wx + b`, which shifts an activation function so it need
not pass through the origin; that lives in [Weights](https://howaiworks.ai/glossary/weights) and
[Neurons](https://howaiworks.ai/glossary/neurons). The three senses share a word and nothing else.

## How It Works

Bias enters through the ordinary machinery of [supervised learning](https://howaiworks.ai/glossary/supervised-learning),
which is what makes it hard to see. A model is given examples and a target column, and it minimises
error against that column. If the column encodes a historical decision — who was hired, who was
arrested, who received care — its job is to reproduce that decision, and it does so faithfully.
Nothing malfunctions. The bias is in the specification, not the bug list.

The proxy problem is the sharpest version of this. You rarely have a column for the thing you care
about, so you use one that correlates with it. Hospitals do not record "how sick is this patient";
they record what was spent. Courts do not record "did this person commit a crime"; they record
arrests. Every gap between the proxy and the target is inherited by the model at full strength, and
if the size of that gap differs by group, the model's errors differ by group.

Deleting the protected attribute does not help, because the remaining features reconstruct it. This
is called redundant encoding, and it is not subtle: postcode carries race in most segregated
housing markets, purchase history carries pregnancy, sport played and college attended carry gender.
A model with a few hundred correlated features will recover a dropped binary attribute to high
accuracy. Removing the column mainly removes your ability to *measure* whether the model is
discriminating.

### The impossibility result

Here is the fact that reframes the whole subject. Three demands that all sound like the bare minimum
for a fair classifier are mutually incompatible:

- **Calibration within groups** — a score of "high risk" means the same probability of the outcome
  whichever group you belong to.
- **Equal false-positive rates** — among people to whom the bad outcome never happened, the same
  share were wrongly flagged in each group.
- **Equal false-negative rates** — among people to whom it did happen, the same share were missed.

Kleinberg, Mullainathan and Raghavan (2016) and Chouldechova (2017) proved these cannot hold
simultaneously when the base rate of the outcome differs between groups. The only exceptions are
degenerate: a perfect predictor, or identical base rates.

Watch it happen on 2,000 people. Group A has a 30% base rate, group B a 60% base rate. The
classifier is calibrated identically — 60% of everyone it flags does go on to have the outcome, in
both groups — and its false-negative rate is identical too, missing half of the true cases in each
group. Nothing here is tuned to disadvantage anyone:

| | Group A | Group B |
|---|---|---|
| People | 1,000 | 1,000 |
| Actually positive | 300 (30%) | 600 (60%) |
| Flagged high-risk | 250 | 500 |
| True positives | 150 | 300 |
| False positives | 100 | 200 |
| Precision (calibration) | 150/250 = 60% | 300/500 = 60% |
| False-negative rate | 150/300 = 50% | 300/600 = 50% |
| **False-positive rate** | **100/700 = 14.3%** | **200/400 = 50%** |

An innocent member of group B is 3.5 times more likely to be wrongly flagged than an innocent member
of group A, from a model that is equally calibrated and equally sensitive for both. The arithmetic
is forced: with only 400 true negatives in group B against 700 in group A, the same 60% precision
spreads a larger absolute number of false positives across a much smaller pool. Equalise the
false-positive rate instead and you must break calibration, the false-negative rate, or both.

The consequence is practical rather than philosophical. "Make the model fair" is not a well-posed
instruction. Someone — a regulator, a clinician, a product owner, not the model — has to decide
which error the system is allowed to make unevenly, and own that decision in writing. This is why
[AI governance](https://howaiworks.ai/glossary/ai-governance) and [accountability](https://howaiworks.ai/glossary/accountability) are not
soft add-ons to the technical work; they are where the undecidable part gets decided.

## Types

The useful taxonomy is not of biases but of the points in the lifecycle where one enters, because
each entry point has a different repair. The five below follow the standard framework of harms
across the machine learning lifecycle set out by Suresh and Guttag.

**Historical bias** is present even when the data is a perfect record. If 5% of a profession's
practitioners were women during the sampling period, a model trained on that record will treat
"woman" as evidence against membership — the data is accurate and the world it describes was
unequal. No amount of extra sampling fixes this, because more data means more of the same world.

**Representation bias** comes from a sample that under-covers a group, so the model has too few
examples to learn them well. This is the failure most amenable to a technical fix — resampling,
targeted collection, [data augmentation](https://howaiworks.ai/glossary/data-augmentation) — and consequently the one
teams reach for even when it is not the problem they have.

**Measurement bias** is the proxy problem above: the observed variable stands in for an unobserved
one, and the substitution is worse for some groups. Spend as a proxy for need fails when a group
faces barriers to access. Arrest as a proxy for offence fails where policing intensity differs.

**Aggregation bias** is fitting one model where the groups genuinely differ. The canonical example
is clinical: the same HbA1c reading does not correspond to the same underlying blood-glucose
control across ethnic groups, so a single diabetes model is miscalibrated for some patients even
with a flawless dataset. The repair is separate models or group-aware features, not more data.

**Deployment bias** is a correct model used in a way it was never validated for — a risk score
built to prioritise a waiting list, then read in court as a statement about a person. The
[monitoring](https://howaiworks.ai/glossary/monitoring) question is not "is the model still accurate?" but "is it still
being used for the thing it was measured on?"

## Real-World Applications

**Amazon's recruiting tool.** Reuters reported in October 2018 that Amazon had abandoned an
experimental system, in development since 2014, that scored candidates from one to five stars. It
was trained on CVs submitted over the previous ten years, most of them from men, so it learned that
male-associated patterns predicted being hired — penalising CVs containing the word "women's" and
downgrading graduates of two all-women colleges. The system was scrapped because engineers could not
guarantee it would not find some other route to the same conclusion. It is the clearest documented
case that removing an attribute does not remove the bias.

**The Optum healthcare risk algorithm.** Obermeyer and colleagues, publishing in *Science* in 2019,
dissected a commercial risk score used to select patients for extra care management, applied to
roughly 200 million people a year in the United States. It predicted future healthcare *cost* as a
stand-in for future health *need*. Because less is historically spent on Black patients at the same
level of illness, the algorithm concluded they were healthier and referred fewer of them. Replacing
the target with active chronic conditions raised the share of Black patients among those
automatically referred from 17.7% to 46.5% — a textbook measurement bias, fixed at the label rather
than the model.

**COMPAS, and why both sides were right.** In 2016 ProPublica reported that among defendants who
were not rearrested within two years, Black defendants were about 1.9 times as likely as white
defendants to have been labelled high risk. Northpointe, the vendor, replied that a given score
carried the same probability of rearrest regardless of race. Neither party was lying. ProPublica was
measuring error-rate balance, Northpointe was measuring calibration, and the base rates in the data
differed — so by the theorem above, both claims could be true at once and no fix could satisfy both.
The dispute is the impossibility result playing out in public, and it is the reason a fairness audit
that does not name its criterion is not an audit.

**Compliance work.** The EU AI Act classifies employment, credit scoring and access to essential
services as high-risk under Annex III — exactly the discrimination cases above — and will require
providers to examine training data for possible biases, turning the choice of fairness criterion
from a research question into a documented artefact someone signs, which is what the impossibility
result implies it has to be. That duty is forthcoming rather than current: the Digital Omnibus
agreed in May 2026 postponed the Annex III high-risk obligations to 2 December 2027, from an earlier
2 August 2026. (The Act's separate Article 50 transparency rules were not delayed and still apply
from 2 August 2026 — a different layer from these high-risk duties.)

## Challenges

**You need the protected attribute to measure fairness, and often cannot collect it.** Every
fairness metric is computed per group, so it requires knowing who is in which group — yet race and
ethnicity are special-category data under GDPR and are simply absent from most commercial datasets.
Teams end up inferring the attribute in order to audit for it, which introduces its own errors.

**Intersectional gaps hide inside per-attribute reports.** Buolamwini and Gebru's 2018 *Gender
Shades* audit of three commercial gender classifiers found a maximum error rate of 0.8% on
lighter-skinned men and up to 34.7% on darker-skinned women — a 43-fold gap that a by-gender report
and a by-skin-tone report would each have partly averaged away. Fairness checked one attribute at a
time can pass while the intersection fails.

**The fairness-accuracy trade-off is real but not universal.** Where the target label is sound,
constraining a model does cost accuracy. Where the label is a bad proxy, as at Optum, fixing it
improves fairness and predictive quality together. Assuming the trade-off exists before checking
the label is how teams talk themselves out of the cheapest available fix.

**Feedback loops make the bias self-confirming.** A model that directs attention — patrols, audits,
inspections — changes where future data is collected. Send more officers to the areas with the most
recorded arrests and you record more arrests there, which the next training run reads as
confirmation. The model's error rate on its own logged outcomes looks excellent.

**Generative models moved the target and kept the problem.** For a
[classifier](https://howaiworks.ai/glossary/classification) there is a confusion matrix to compute rates from. For a
[large language model](https://howaiworks.ai/glossary/large-language-model) writing a performance review or a loan
rationale, there is no ground-truth column, so the group-conditional rates the theorems are stated
over do not exist. Evaluation falls back on prompt suites and human raters, which are themselves
sampled from somewhere. The mathematics of fairness is far better developed than the measurement
that would let you apply it — see [explainable AI](https://howaiworks.ai/glossary/explainable-ai) and
[transparency](https://howaiworks.ai/glossary/transparency) for the tooling that partially fills the gap, and
[ethics in AI](https://howaiworks.ai/glossary/ethics-in-ai) for the wider frame.

## Frequently Asked Questions

### Why is AI biased?

Because a model learns from records produced by an unequal world, and because the quantity it is trained to predict is usually a proxy for what you actually care about. An arrest is not a crime, a medical bill is not an illness, and a past promotion is not merit — the model faithfully reproduces the gap between the proxy and the thing itself.

### Can you fix AI bias by removing race and gender from the data?

No. Other features reconstruct the protected attribute: postcode, school, purchase history and word choice all carry it. Amazon's recruiting tool had no gender field and still learned to downgrade CVs containing the word 'women's'. Removing the attribute mostly removes your ability to measure the bias.

### Can a model be made completely fair?

Not in the sense most people mean. Kleinberg, Mullainathan and Raghavan (2016) and Chouldechova (2017) proved that calibration within groups, equal false-positive rates and equal false-negative rates cannot all hold at once when base rates differ between groups, except when prediction is perfect. 'Make it fair' is not a well-posed instruction until someone says which fairness criterion wins.

### Does reducing bias always cost accuracy?

Not always. When the bias comes from a bad proxy label, fixing the label improves both. Obermeyer et al. (Science, 2019) found a healthcare risk algorithm predicting cost instead of illness; predicting active conditions instead was both fairer and a better prediction of health need.

### Is this the same as the 'bias' in bias-variance trade-off?

No. That is statistical bias — systematic error from a model's assumptions — and it is covered under high bias and low variance. A third meaning, the b in wx + b, is the learnable offset inside a neuron, covered under weights and neurons.

## Related

### Related terms

- [Ethics in AI](https://howaiworks.ai/glossary/ethics-in-ai)
- [AI Governance (AIG)](https://howaiworks.ai/glossary/ai-governance)
- [Explainable AI (XAI)](https://howaiworks.ai/glossary/explainable-ai)
- [Accountability](https://howaiworks.ai/glossary/accountability)
- [Transparency](https://howaiworks.ai/glossary/transparency)
- [High Bias](https://howaiworks.ai/glossary/high-bias)

---

Source: https://howaiworks.ai/glossary/bias — HowAIWorks.ai