Definition
Trust in AI is a person's willingness to rely on a system's output and act on it — to treat a model's answer as good enough to use without independently checking it. What makes trust worth understanding, and what saves the word from being a vague synonym for "confidence," is that the goal is not to maximise it: the useful target is calibration, trusting a system exactly as much as its real reliability on the task in front of you deserves, and no more.
A perfectly calibrated user would defer to the system on the tasks it handles well, override it on the tasks it handles badly, and — the hard part — be able to tell those two kinds of task apart before seeing the outcome. Real users miss in both directions, and the two misses are the substance of the topic. Over-trust, relying on a system that is wrong, and under-trust, discarding a system that is right, are symmetric failures on either side of the same calibrated middle. A page about trust that only warns against having "too little" of it has told you half the story, and usually the less interesting half.
How It Works
Trust is not a single dial you turn up. It is a moment-by-moment judgement about whether this output, on this task, is good enough to act on. Getting that judgement right is calibration; getting it wrong has two distinct shapes, each with a research literature and each with a modern AI version.
Over-trust and automation bias
The first failure is deferring to a system past the point its accuracy justifies. Raja Parasuraman and Victor Riley named this precisely in their 1997 paper Humans and Automation: Use, Misuse, Disuse, Abuse (Human Factors, 39(2), 230–253): misuse is "overreliance on automation, which can result in failures of monitoring or decision biases." A crew that stops cross-checking the autopilot because it has been right for a hundred hours is not being lazy; it is being badly calibrated, treating a system that is usually right as if it were always right.
The everyday version is "death by GPS" — drivers who follow satellite navigation onto closed roads, into deserts, and off boat ramps into lakes, because a confident instruction inside its normal competence looks the same as a confident instruction outside it. The interface gives no cue that the map is now wrong, so the user's trust stays high while the system's reliability has quietly collapsed.
The version that matters now is the large language model, and it is the sharpest case in this whole entry. In a fluent model, the cues a human uses to gauge competence — confident tone, well-structured prose, specific detail — are decoupled from whether the content is correct. A hallucination arrives in exactly the same authoritative register as a fact. That is a calibration trap by construction: fluency reads as competence, so users over-trust precisely the outputs they should be scrutinising hardest. It is why a New York attorney was sanctioned in 2023 in Mata v. Avianca after filing a brief whose case citations ChatGPT had invented — the fabrications were formatted flawlessly, which is exactly what made them convincing.
Under-trust and algorithm aversion
The opposite failure is throwing away a system that is actually better than you. Berkeley Dietvorst, Joseph Simmons and Cade Massey documented it in Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err (Journal of Experimental Psychology: General, 2015, vol. 144, pp. 114–126). Across five studies, participants who watched a statistical model forecast — and therefore watched it make its inevitable small mistakes — became less likely to bet on it than participants who never saw it work, "even among those who saw the [model] outperform the human." Parasuraman and Riley's word for this is disuse: the neglect or underutilisation of a capable system.
The reason this is a mistake and not just caution is arithmetic. The models people were rejecting were the better forecasters. Robyn Dawes's tradition and, most systematically, Grove and colleagues' 2000 meta-analysis (Clinical Versus Mechanical Prediction) found that mechanical prediction was "about 10% more accurate than clinical predictions" on average, and substantially outperformed expert human judgement in 33%–47% of the study conditions examined while being substantially beaten by it in only 6%–16%. So a user who watches an algorithm miss once and switches back to their own judgement is, on the base rates, trading a better instrument for a worse one. The distortion has a telling asymmetry: people lose confidence faster in an algorithm than in a human after seeing the two make the identical error. We forgive a colleague a mistake we will not forgive a machine.
What actually moves calibration
If the goal is matching trust to reliability, then the things worth building are the ones that give the user a truer read on how reliable the system is right now. A demonstrated reliability track record over time is the base. Transparency about how the system works, and explainability for individual decisions, let a user check why an answer was produced rather than judging it on tone alone. And the most direct lever is honest uncertainty communication: a model that can say "I am not sure" hands the user exactly the signal calibration requires, while a model that is confidently wrong strips that signal out. Calibrated trust is not something a system earns once; it is a property of the fit between what the system can do and what the user believes it can do, and either side can drift.
Real-World Applications
The over- and under-trust failures are not abstractions; each shows up in named, costly ways.
- Cockpit automation is the founding case study of automation bias: human-factors research since the 1990s has documented crews accepting a wrong flight-management state because the automation was usually right, the "misuse" Parasuraman and Riley described. It is also why aviation invests so heavily in robustness and mandatory cross-checking rather than trusting monitoring alone.
- Navigation produces the "death by GPS" pattern — drivers routed into hazards outside the system's competence, an over-trust failure that persists precisely because the interface signals no drop in reliability.
- Legal practice has already generated sanctioned filings, most publicly Mata v. Avianca (2023), where fabricated citations from a fluent model were trusted because they were well formatted — the hallucination-as-authority trap in the wild.
- Clinical and managerial forecasting is the canonical under-trust setting: the algorithm-aversion studies show experts overriding statistically superior risk models after a single salient miss, discarding the more accurate tool for the more familiar one.
- Coding assistants show a newer over-trust mode: developers installing AI-suggested package names that do not exist, a gap now actively exploited by attackers registering those hallucinated names ("slopsquatting").
Key Concepts
Trust sits next to four ideas it is constantly confused with. The distinction in each case is the same: trust is the human response, and the neighbour is a property of the system or a mechanism that feeds it.
- Trust vs transparency: transparency is how openly a system's workings can be inspected. It is an input to calibration, not the same thing — a fully transparent system can still be under-trusted, and an opaque one over-trusted.
- Trust vs explainable AI: explanation is a technique that helps a user judge an individual output. It is one of the strongest calibration tools, but the trust is the outcome; the explanation is only useful if it actually shifts reliance toward the cases the model gets right.
- Trust vs AI safety: safety is about the system not causing harm. Trust is about human reliance. The two come apart cleanly — a genuinely safe system that users refuse to adopt has an under-trust problem, and a confidently unsafe system that users lean on has an over-trust problem.
- Trust vs accountability: accountability assigns responsibility for what a system does. It is a precondition for warranted trust — you can reasonably rely on a system whose failures someone is answerable for — but it is a governance property, not the reliance itself.
The unifying concept underneath all four is calibration: the alignment between how much a user relies on a system and how reliable that system actually is. Over-trust and under-trust are just the two ways that alignment fails.
Challenges
The hard problems in trust are specific to the fact that it is a moving relationship, not a fixed system property.
- Fluency has decoupled from competence. Older automation failed visibly — an alarm, a wrong number, a jerk of the controls. A language model fails smoothly, in prose indistinguishable from its correct output, which removes the very surface cues humans evolved to calibrate on. Restoring a reliable "this one is shaky" signal is an open research problem, not a solved feature.
- Calibration is hard to measure. Reliability can be quantified; trust is a belief, and the honest proxy is behaviour — does the user rely on the system on the cases it gets right and override it on the cases it gets wrong ("appropriate reliance")? That is expensive to observe and easy to confuse with raw adoption, which counts over-trust as success.
- Automation bias resists training. Telling people a system is imperfect does not reliably stop them deferring to it, and the algorithm-aversion asymmetry shows the reverse is just as sticky: one visible error can cost an accurate model a user's confidence out of all proportion to its actual reliability.
Future Trends
The most concrete direction is turning uncertainty communication from a nicety into a measured capability: evaluating whether a model's stated confidence tracks its actual accuracy, so that "I am not sure" means something a user can rely on. The related shift is in how deployed systems are judged — from raw adoption or accuracy toward appropriate reliance, whether users trust the system on the cases it gets right and catch it on the cases it gets wrong. Both aim at the same target this entry began with: not more trust, but better-calibrated trust.