Definition
Ethics in AI is the work of deciding which good thing to give up, because in an AI system the good things are frequently incompatible and at least one of those incompatibilities is a proved mathematical result rather than an engineering shortfall. A risk classifier cannot be equally well-calibrated for two groups and wrong at equal rates for them when the underlying rates differ. A dataset cannot be released with a strong formal privacy guarantee and full accuracy. An explanation detailed enough to contest is also detailed enough to game. In each case the choice is forced, and no amount of extra data, compute or good intention dissolves it.
This is why the standard framing of the subject — a list of virtues — is not an instruction. Anna Jobin, Marcello Ienca and Effy Vayena surveyed 84 AI ethics guideline documents from governments, companies and institutes for Nature Machine Intelligence in 2019 and extracted 11 principles. The most common, transparency, appeared in 73 of the 84 documents, about 87%; not one principle appeared in all of them, and the authors' central finding was that the apparent convergence dissolved on inspection, with the same words meaning different things across documents. Five virtues that cannot be simultaneously maximised are five constraints, and a list of constraints does not tell you what to build.
What breaks if a team skips this: the trade-off gets made anyway, silently, by whoever picks the default threshold in the config file. Someone chooses the decision cut-off, the noise parameter, the retention window. Those numbers are the ethical position of the system, whether or not anybody wrote a document about it. The practical question is never "are we being ethical" but "which of these two things did we just decide to lose, and can we say so out loud".
How It Works
An ethical question in AI becomes tractable at the moment it turns into a number or a rule that someone signs. The three conflicts below are the ones with the clearest arithmetic, and each has a documented case where a real organisation had to pick a value.
Fairness against fairness. Calibration within groups, equal false-positive rates and equal false-negative rates are all reasonable-sounding demands, and Kleinberg, Mullainathan and Raghavan (2016) and Chouldechova (2017) proved they cannot hold together when the base rate of the outcome differs between groups. The exceptions are degenerate: a perfect predictor, or identical base rates. The arithmetic is worked through on a 2,000-person example in Bias in AI, along with the COMPAS dispute in which ProPublica and the vendor were each correct about a different metric. The consequence for this page is the one that generalises: "be fair" is under-specified, so an ethics process that does not end by naming a criterion has not finished.
Privacy against accuracy. Differential privacy makes this trade quantitative. A mechanism is
ε-differentially private if adding or removing any one person's record changes the probability of
any published result by at most a factor of e^ε. At ε = 0 the output is independent of you and
also independent of the data, which is perfect privacy and useless statistics. As ε rises the
guarantee weakens exponentially and the numbers get truer. Nothing in the mathematics says what ε
should be — that is the ethical part, and it is a single scalar.
The US Census Bureau had to pick one in public. Its early demonstration products for the 2020 census used a global privacy-loss budget of 4.5 (4.0 for the persons file, 0.5 for housing units). State redistricting officials and researchers examined the released demonstration data and objected that small-geography counts were too distorted to use. A revised April 2021 demonstration raised the budget to 12.2, and on 9 June 2021 the Bureau set production values of ε = 19.61 — 17.14 for persons and 2.47 for housing units — saying the settings struck the best balance between usable statistics and its statutory duty to protect respondents.
Because ε sits in an exponent, that move is much larger than "4.5 to 19.61" suggests. The worst-case
bound on how much any individual's presence can shift a published figure went from e^4.5 ≈ 90 to
e^19.61 ≈ 330 million — roughly 3.6 million times weaker. A factor of 4.4 on the dial is a factor
of millions on the guarantee. Both endpoints are truthfully described as "differential privacy",
which is exactly why the parameter, not the label, is the ethical claim.
Whose values. The Moral Machine study (Awad and colleagues, Nature, 2018) collected roughly 40 million decisions about unavoidable-crash dilemmas from participants in 233 countries and territories — the largest survey of moral intuitions ever run. It found broad agreement on sparing humans over animals, more lives over fewer, and the young over the old, but it also found three cultural clusters, labelled Western, Eastern and Southern, that diverged sharply: the preference for sparing younger characters is much weaker in the Eastern cluster and much stronger in the Southern one.
Germany's Ethics Commission on Automated and Connected Driving reached the opposite conclusion from the same problem. Its report, delivered on 20 June 2017 with 20 numbered rules, states in Rule 9 that in unavoidable accident situations any distinction based on personal features — age, gender, physical or mental constitution — is strictly prohibited. Forty million votes said people prefer to spare the young; a national ethics body said a machine may not act on that preference. Neither is a measurement error. One is a description of what people want, the other a prescription about what is permitted, and the whole discipline lives in the space between them. This is also why value learning is a harder problem than it sounds: there is no consistent target function to learn.
The pattern across all three is the test worth applying. Ethics work has actually happened when it produces a specified artefact — a criterion, an ε, a prohibited feature — rather than an adjective.
Real-World Applications
The 2020 US Census. The privacy-loss budget above is the cleanest case of an ethical trade-off resolved as a published number by a named official. Every downstream user — redistricting authorities, epidemiologists allocating vaccines, school districts sizing catchments — inherited that single choice of 19.61, and could argue about it precisely because it was a number. Compare a guideline promising to "respect privacy while enabling data-driven decisions", which nobody can disagree with and nobody can audit. See Privacy for the mechanisms themselves.
Apple's client-side CSAM scanning. In August 2021 Apple announced on-device matching of photos against known child sexual abuse material, with two numbers doing the ethical work: an account was flagged only after about 30 matches, and Apple claimed a false-flag rate below one in one trillion accounts per year. The threshold is a trade in plain sight — raise it and more offenders go undetected, lower it and more innocent users are wrongly escalated. Apple postponed the feature on 3 September 2021 after security researchers and civil-liberties groups argued the scanning infrastructure could be repurposed, and abandoned it in December 2022, stating it could not be implemented without imperilling the security and privacy of all users. Deciding not to ship is a legitimate resolution, and it is one the principles list has no vocabulary for.
Germany's driving rules. The 2017 commission's 20 rules are the clearest example of a body deliberately overriding measured public preference on normative grounds, and they influenced how Germany subsequently legislated for higher-autonomy vehicles. For the engineering side of the same domain, see autonomous vehicle safety.
Model specifications. OpenAI's Model Spec, first published on 8 May 2024, is an attempt to move value choices out of implicit training decisions and into a document: it enumerates behavioural rules and, critically, a priority order for resolving conflicts between instructions from the platform, developers and users. Whether or not the ordering is right, publishing one makes the trade-off contestable — which is the same move the Census made with ε.
Key Concepts
Descriptive versus normative. What people believe is right and what is right are different questions, and survey data answers only the first. Moral Machine against Rule 9 is the cleanest illustration; the mistake it guards against is treating a large enough poll as an answer.
Specification is the deliverable. The output of ethics work is a criterion, a parameter or a prohibition that a third party can check, not an adjective. This is the boundary with AI governance, which supplies the process that makes a specification binding, and with accountability, which supplies the person who answers for it when it is wrong.
Contestability over comprehension. Transparency and explainable AI matter here for a narrower reason than "understanding": an affected person cannot appeal a decision whose basis they cannot see, and a trade-off nobody can inspect cannot be argued with.
Not the same as safety. AI safety asks whether a system does what it was meant to do, including under adversarial conditions and at capability levels above today's. Ethics asks whether what it was meant to do was the right thing. A perfectly safe system executing a badly-chosen trade-off is an ethics failure with no safety incident attached.
Challenges
The measurable side of a conflict wins by default. A false-positive-rate gap is a number; an ε is a number; the dignity of being asked before your data is used is not. When one side of a trade-off has a metric and the other has an adjective, review processes optimise the metric and record the adjective as "considered". This is the most reliable failure mode in the field, and it is structural rather than a matter of anyone's bad faith.
Publishing principles has not been shown to change decisions. McNamara, Smith and Murphy-Hill tested this directly in 2018: 63 software engineering students and 105 professional developers responded to 11 ethical scenarios drawn from real incidents, and explicitly instructing them to consider the ACM code of ethics had no observed effect relative to a control group. One survey is not the last word, but it is the direct evidence available, and it points at the gap between the 84 guideline documents above and what teams actually ship.
Ethics review has no denominator. The discipline is judged on the harms that occurred. The systems never built, the features cut, the datasets not collected leave no record, so there is no way to measure how often the process worked — and no way to distinguish careful restraint from having faced no hard cases.
One model, many normative regimes. A single deployed model serves users living under incompatible laws and incompatible moral consensus, as the three Moral Machine clusters show directly. Configuring behaviour per jurisdiction is itself a contested position — it treats moral claims as local preferences — while a single global behaviour imposes one region's settlement on everyone. There is no configuration that avoids taking a side.
Generative systems removed the place the metrics attached to. Fairness criteria are defined over group-conditional error rates, which require a ground-truth column. A model writing a performance review or a rejection letter has no confusion matrix, so the quantities the impossibility results are stated over do not exist for it. The best-developed part of the mathematics applies to the narrowing share of systems that are classifiers.
Future Trends
The direction worth watching is the shift from principle documents to published parameters: an ε, a named fairness criterion, a refusal policy, a detection threshold. A parameter can be wrong in a way an adjective cannot, which is precisely what makes it auditable — and it is what regulators increasingly ask for, since a documented choice is something a supervisor can review and a court can examine.
Second, empirical ethics — measuring what practitioners actually do rather than what their organisations publish — is the only route out of the McNamara result. If instructing people to consult a code changes nothing, the open question is what does: incentives, review gates, liability, or the news-story effect that same study noticed, in which participants who knew of real incidents were more inclined to reject the practices involved.
Third, these incompatibilities are being pushed into contract and statute rather than left to engineering judgement. That changes who carries the decision, not the fact that a decision is forced, and it moves the argument from whether a trade-off exists to who is entitled to set it — which is where this term hands off to AI governance and trust.