The Use and Usefulness of Attention Checks in Behavioral Research

Abstract Attention checks have been touted as a means of excluding inattentive and careless responses from survey data. As a result, numerous approaches have been proposed, but relatively little attention has been paid to which ones are most helpful. The present article demonstrates that this blind spot has obscured vast differences in attention check efficacy. An initial survey of consumer-behavior and decision-making researchers revealed six commonly used attention check approaches (memory checks, questions with obvious answers, English comprehension, self-reported attentiveness, straight-lining, and variations on the instructional manipulation check). These were then paired with six effects, drawn from the Many Labs series of papers in social psychology (N > 22,000, all preregistered). The different checks varied substantially in reducing noise and increasing effect size. Follow-up studies suggest that participant attentiveness, as currently measured, may reflect a given participant’s general approach to completing studies, rather than momentary fatigue or boredom varying across time or within studies. We recommend that behavioral researchers focus their efforts on going beyond mere “pulse-checkers” and consider favoring the use of memory checks and obvious questions as attention checks in their research.