41 · Automating
Automation Bias
People tend to go along with what a machine tells them, and stop checking it.
Evidence: Holds upWhat we do with it
- 01
Show how sure the system is and where its answer came from, so people know when to look twice.
- 02
Explanations alone don't fix it. In one study they made people accept the AI's advice more often, right or wrong.
- 03
On decisions that matter, slow the yes down, for instance by asking for the person's own call first. Designs like that cut over-reliance in testing, though people liked them less.
- 04
Make a person answer for the result. Pilots who felt accountable for their decisions double-checked the automation more and made fewer errors.
Examples
Getting it wrong, getting it right.
Don't
An AI summary of a contract shown as a finished answer, with an Approve button and nothing to check it against.
Do
The same summary with each point linked to the clause it came from, and the points the system is least sure of marked for a person to read.
The evidence
What the research says.
Linda Skitka, Kathleen Mosier and Mark Burdick tested it in 1999 with a simulated flight task. When the automated aid worked, people who had it made fewer errors. When it failed to flag a problem, they caught 59% of those problems, against 97% for people with no aid, and they followed its wrong directions about two times in three, even when other indicators showed it was wrong. Raja Parasuraman and Dietrich Manzey's 2010 review found it in novices and experts alike, and found that training and instructions didn't prevent it. A 2012 review of 74 studies, focused on health care, found it in clinical decision tools, and found that stressing the user's accountability, showing how confident the system is, and offering information instead of a verdict all helped. It carries over to AI: in Gagan Bansal and colleagues' 2021 study, explanations made people more likely to accept the AI's recommendation whether it was right or not, and Zana Buçinca's team showed the same year that designs which made people stop and think cut over-reliance, though people rated those designs lowest.
How our view changed
We treat a machine's answer as a draft to check, and build the check into the screen.
Sources
- Mosier, K. L., Skitka, L. J., Heers, S., & Burdick, M. (1998). Automation bias: Decision making and performance in high-tech cockpits. The International Journal of Aviation Psychology, 8(1).
- Skitka, L. J., Mosier, K. L., & Burdick, M. (1999). Does automation bias decision-making? International Journal of Human-Computer Studies, 51(5).
- Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3).
- Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1).
- Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021). Does the whole exceed its parts? The effect of AI explanations on complementary team performance. CHI '21.
- Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1).
Last reviewed November 9, 2026. How we keep this current
See it differently? If you've read research that points another way, or watched this go differently in your own work, tell us at hello@synergyprod.com. We'll read it and write back.