Human Misunderstanding of Stated AI Accuracy: Exploring Over- and Under-Reliance Through the Lens of Prospect Theory

To enhance decision-making, humans often use AI, but over- or under-reliance can negatively impact performance. Although AI accuracy is presented to users for reliance calibration, prior work shows that stated AI accuracy often does not lead users to rely on AI in proportion to it. To explore this, we examined how users interpret stated AI accuracy as a cue. Experiment 1 explored decision shifts across differing accuracy levels (N = 91); Experiment 2 modeled the subjective weight attached to each level (N = 20). The results align with prospect theory, indicating that participants tend to undervalue high accuracy and overvalue low accuracy. At the same time, we found that compared with not providing accuracy information, presenting reliable and well-calibrated AI accuracy can potentially reduce their misperception of that and thus may offer some benefits for calibrating reliance. Based on our findings, we discuss the design implications for fostering appropriate reliance.