Definition
Human-AI collaboration is the arrangement in which a person and a model each do part of the same task, with someone having decided which part each handles and who gets the last word. It deserves a definition rather than a slogan because the arrangement frequently loses: across 370 effect sizes drawn from 106 experiments, human-AI systems performed worse than the better of human alone or AI alone, with a pooled effect of Hedges' g = −0.23 (95% CI −0.39 to −0.07) — Vaccaro, Almaatouq and Malone, When combinations of humans and AI are useful, Nature Human Behaviour, 2024.
The same 106 experiments also found that the pairing beat the human working alone by g = +0.64 (95% CI 0.53 to 0.74), a medium-to-large gain. Both results are correct and they answer different questions. The paper calls the first synergy and the second human augmentation, and the gap between them is where almost every disappointing deployment lives: a pilot that shows assisted staff beating unassisted staff has demonstrated augmentation, while the thing being purchased is synergy.
That is what breaks when this concept is treated as a slogan. A trial with two arms — people with the tool, people without — cannot distinguish a system that improves on both parties from one that drags a strong model down to a mediocre committee, because it never records what the model would have scored on its own. Ship on that evidence and you may have paid for an interface, a review queue and a training programme to obtain something worse than turning the model on and leaving the person out, or worse than routing the case to the specialist you built the queue to avoid.
The gain is not spread evenly across kinds of work. In the same meta-analysis, decision tasks — where the participant picks between a finite set of options — came in at g = −0.27 (95% CI −0.44 to −0.10, p = 0.002), while creation tasks, which produce open-ended content, came in at g = +0.19 (95% CI −0.09 to 0.48). The creation estimate is not itself distinguishable from zero on 34 effect sizes, but the difference between the two task types is statistically significant. Drafting together is a defensible default. Choosing together is the case that needs evidence.
How It Works
All of the value in a pairing comes from cases where the two parties disagree. Where both are right, the second opinion changes nothing; where both are wrong, it cannot help. So the whole design question is: how often do they disagree, who is usually right when they do, and can whoever holds the final say tell which situation they are in?
Put numbers on it. Take a binary task — flag or clear, approve or decline — where a person is right 80% of the time and a model is right 90%. If their mistakes are statistically independent, the four outcomes fall out directly:
| Outcome | Probability |
|---|---|
| Both right | 0.80 × 0.90 = 72% |
| Person right, model wrong | 0.80 × 0.10 = 8% |
| Model right, person wrong | 0.20 × 0.90 = 18% |
| Both wrong | 0.20 × 0.10 = 2% |
They disagree on 8% + 18% = 26% of cases. The best a team could conceivably do is be right whenever either party is right, which is 100% − 2% = 98%. That is the ceiling, and it is a genuinely large prize: it cuts the error rate from the model's 10% to 2%, a fivefold reduction. Everything worth having in this arrangement is inside that 26% of disagreements.
Now the floors. A person who simply defers to the model every time scores 72% + 18% = 90% — exactly the model alone, which is the arithmetic definition of a rubber stamp. A person who resolves disagreements no better than a coin scores 72% + (0.5 × 26%) = 85%: better than working alone, worse than the model, and a textbook instance of augmentation without synergy. To merely match the model, they must resolve disagreements correctly 18 ÷ 26 = 69.2% of the time — which is precisely the rate at which the model is the right party in a disagreement. Below that threshold, every hour of review is destroying accuracy.
The break-even can be written more usefully as a ratio. Suppose the reviewer correctly overrides a fraction a of the 8% of cases where they are right and the model is wrong, and mistakenly overrides a fraction b of the 18% where the model is right. Team accuracy is 90% + 8a − 18b, so the pairing only beats the model alone when a/b exceeds 18/8 = 2.25. In plain terms: one wrong override cancels 2.25 correct ones.
That ratio is the whole story, and it moves fast. Hold the person at 80% and improve the model to 95% and the break-even ratio becomes (0.20 × 0.95) ÷ (0.80 × 0.05) = 4.75 — the reviewer now needs to be nearly five times better at catching model errors than at second-guessing correct answers. Reverse it, with the person at 90% and the model at 80%, and the ratio falls to 0.44, meaning even sloppy overriding pays. This is exactly what the meta-analysis observed empirically: when the human alone outperformed the AI alone, synergy was g = +0.46 (95% CI 0.28 to 0.66); when the AI alone outperformed the human, it was g = −0.54 (95% CI −0.71 to −0.37).
Two caveats keep the arithmetic honest. First, independence is a generous assumption. Hard cases tend to be hard for both parties, and a model trained on human-labelled data inherits human mistakes by construction, so the errors correlate. In the limiting case where the model's 10% of errors sit entirely inside the person's 20%, there is never a case where the person is right and the model is wrong, the 8% cell empties, and the ceiling collapses from 98% to 90% — the model alone. Real systems sit somewhere between, which is why measuring the disagreement rate on your own data is worth more than any argument about complementary strengths.
Second, where the person sits relative to the loop changes only how many decisions they make, not the arithmetic of each one. A human in the loop rules on every case; a human on the loop samples and intervenes on exceptions; and the automation levels and handover problems that come with pushing the person further out belong to autonomous systems. Moving from in-the-loop to on-the-loop reduces both the 8% caught and the 18% wrongly overturned in the same proportion — it buys throughput, not accuracy.
Creation tasks escape this structure, which is why they score differently. There is no forced choice between two candidate answers, so there is no override to get wrong: the person keeps a veto over an artefact they can edit rather than a verdict they must pick. A large share of the work — assembling boilerplate, producing variants, filling in routine structure — has no correct answer to be wrong about at all.
Real-World Applications
Three field studies measured all three numbers — person alone, system alone, and the two together — on the same task set. They are the ones worth knowing.
Screening mammography with computer-aided detection. Povyakalo, Alberdi, Strigini and Ayton reanalysed a study in which 50 readers interpreted 180 mammograms both with and without CAD support (Medical Decision Making, 2013). The original study found no significant average effect. Splitting readers by discriminating ability and cases by difficulty found two effects of opposite sign hiding inside that null: CAD raised sensitivity by 0.016 (95% CI 0.003 to 0.028) for the 44 least discriminating radiologists on 45 relatively easy, mostly CAD-flagged cancers, and lowered it by 0.145 (95% CI 0.034 to 0.257) for the 6 most discriminating radiologists on the 15 difficult ones. The harm to the best readers was roughly nine times the size of the help to the rest, and an average-effect analysis reported neither.
Management consulting. Dell'Acqua and colleagues gave GPT-4 to 758 Boston Consulting Group consultants in a pre-registered experiment (Harvard Business School working paper 24-013, 2023; later in Organization Science). On 18 tasks inside the model's competence, consultants using it completed 12.2% more tasks, 25.1% more quickly, at materially higher rated quality. On one managerial task deliberately designed to sit just outside that competence, the control group answered correctly 84.5% of the time against 70% for the group given the raw model and 60% for the group given the model plus a prompt-engineering briefing — an average drop of about 19 percentage points, and the briefing made things worse, not better. The authors named the underlying phenomenon the jagged technological frontier: capability that is uneven across tasks a person would judge equally hard. They also observed two working styles among the successful users — "centaurs", who split the task and hand whole subtasks to the model, and "cyborgs", who interleave with it continuously.
Customer support. Brynjolfsson, Li and Raymond studied a staged rollout of a conversational assistant across 5,172 agents (Quarterly Journal of Economics, 140(2), 889–942, 2025). Issues resolved per hour rose 15% on average, but the published version is specific about the distribution: less experienced and lower-skilled workers improved on both speed and quality, while "the most experienced and highest-skilled workers see small gains in speed and small declines in quality." That is the mammography result again, in a different industry — assistance raises the floor and, for the strongest performers, can lower the ceiling. The longest-running version of this measurement is in educational AI, where four decades of tutoring-system trials land between 0.3 and 0.8 standard deviations against the two standard deviations Bloom reported in 1984 for one-to-one human tutoring. The labour-market reading of this study belongs to AI and employment; what it contributes here is the shape of the effect, not its aggregate.
Where a specific division of labour has been worked out in detail, it looks like the guides on this site rather than like a principle: ambient clinical documentation drafts the note and the clinician signs it, described in AI medical scribes; a reviewer bot comments and a human approves the merge, in AI for code review; a classifier routes a ticket above a confidence threshold and escalates below it, in AI for support ticket triage. In each case the interesting content is the threshold and the escalation path, not the partnership.
Key Concepts
- Two claims that carry opposite signs: beating both parties is what the literature calls synergy; beating only the person is augmentation. Nearly every vendor claim and internal pilot measures the second, the decision to deploy depends on the first, and in the same 106 experiments they came out at −0.23 and +0.64.
- The two named biases are one quantity: b and 1 − a in the override arithmetic above. Automation bias is failing to reject a wrong output; algorithm aversion is rejecting a right one. Miscalibration in either direction produces the same visible symptom — a team that scores below the better party working alone.
- The jagged frontier makes allocation a measurement, not a judgement: the boundary does not follow perceived difficulty, so which side of it a task sits on cannot be reasoned out from the task description. It has to be tested, per task, on the model version you are actually running.
- Complementarity requires uncorrelated errors, not different strengths: "humans bring judgement, AI brings scale" is a description, not a prediction. What determines the ceiling is whether the two parties fail on different cases, which is an empirical property of your data and can be measured in an afternoon.
Challenges
The baseline problem is the expensive one. Of 5,126 papers screened for the Vaccaro meta-analysis, only 74 met the inclusion criteria, of which reporting all three performance figures was the binding constraint. If academic work designed to study human-AI systems reports the model-alone number that rarely, a procurement pilot almost certainly does not. Without it, no result can be interpreted: a 20% improvement over unassisted staff is excellent if the model scores below them and a failure if it scores above.
Confidence signals are the input the reviewer does not have. The override arithmetic assumes the person can tell which cases to challenge, but language models are not reliably calibrated — hallucinations arrive in the same fluent register as correct answers, and a confidently-wrong output is precisely the one that fails to trigger review. This is the practical argument for explainable AI in a decision loop: not transparency for its own sake, but raising a and lowering b.
Training the reviewer can backfire. In the BCG experiment, the group given a prompt-engineering overview did worse outside the frontier than the group given the bare tool — a 24 percentage point drop against 13 in the regression estimates. Instruction that raises confidence in the tool without improving discrimination moves the override ratio the wrong way. Prompt engineering makes the model's output better; it does not make the person a better judge of when that output is wrong.
Regulation mandates the arrangement that the evidence says is hardest to get right. Article 14 of the EU AI Act (Regulation (EU) 2024/1689) requires high-risk systems to be designed so they can be effectively overseen by natural persons, that overseers "remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias)", and that they be able "to decide, in any particular situation, not to use the high-risk AI system or to otherwise disregard, override or reverse the output". That is a legal duty to run exactly the decision-task configuration that scored g = −0.27. The duty is not misguided — accountability has to attach to a person, and it cannot attach to a model — but compliance and accuracy are separate objectives here, and an oversight step installed for the first will not automatically deliver the second.
Every allocation has an expiry date. The BCG results describe GPT-4 as it stood in April 2023. A task that sat outside the frontier then may sit inside it now, which means a division of labour is a dated measurement rather than a policy, and the benchmark that justified it needs re-running when the model version changes.
Future Trends
The clearest shift is in what gets reviewed. When an AI agent executes a multi-step agentic workflow, the person is no longer approving one output against one alternative; they are approving a trajectory in which an early error propagates through every later step. The override arithmetic still holds for each decision point, but the decision points are fewer, later and more consequential, and the reviewer's information is worse because they did not watch the intermediate steps. Expect the useful oversight artefact to become the audit trail rather than the approval click.
Second, the measurement gap has become the differentiator. Publishing an assisted-versus-unassisted number is easy and increasingly uninformative; publishing the model-alone number alongside it is what distinguishes an evaluated system from an advertised one. Organisations that run three-arm evaluations as routine practice will discover which of their review steps are load-bearing and which are ceremony, and the answer will not be uniform across tasks.
Third, as models improve, the break-even override ratio rises mechanically — 2.25 at 90% accuracy, 4.75 at 95%, and higher still beyond that. The economically sensible response is not to remove the person but to narrow their remit: fewer cases reviewed, chosen because they are the ones where human and model errors are known to decouple, with routine agreement passing through untouched. That reallocates attention rather than eliminating it, which is the same conclusion the task-level evidence on AI and employment keeps producing, and it puts a real premium on trust being calibrated to the case rather than to the system.