Warnings and Endorsements: Improving Human-AI Collaboration Under Covariate Shift

Problem definition: While artificial intelligence (AI) algorithms may perform well on data that are representative of the training set (inliers), they may err when extrapolating on non-representative data (outliers). These outliers often originate from covariate shift, where the joint distribution of input features changes from the training set to deployment. How can humans and algorithms work together to make better decisions when faced with outliers and inliers? Methodology/results: We study a human-AI collaboration on prediction tasks using an anchor-and-adjust framework, and hypothesize that humans are biased towards na¨ıve adjustment behavior: making adjustments to algorithmic predictions that are too similar across inliers and outliers, when ideally adjustments should be larger on outliers than inliers. In an online lab experiment, we demonstrate that participants are indeed unable to sufficiently differentiate absolute adjustments to an AI algorithm when faced with both inliers and outliers, leading to a 143-176% increase in their absolute deviation from the optimal prediction compared to participants who only face either all inliers or all outliers. We design a ‘warning’ that alerts participants when feature values constitute outliers and, in a second experiment, we show that this warning helps participants differentiate adjustments, ultimately reducing their absolute deviation from the optimal prediction by an average of 31% on outliers and 35% on inliers. We demonstrate that an additional intervention — ‘endorsements’ that alert participants when feature values constitute inliers — reduces participants’ absolute deviation from the optimal prediction on inliers by an additional 34% on average. Managerial implications: Our work uncovers a behavioral bias towards na¨ıve adjustment behavior, and identifies a simple, educational intervention that mitigates this bias. Ultimately, we hope that this work will help managers best equip their employees with the knowledge they need to succeed in a human-AI collaboration.