Can People Identify “Deceptive” or “Misleading” Items that Tend to Produce Mostly Wrong Answers?

In many domains, two-alternative forced-choice questions produce more correct responses than wrong responses across participants. However, some items, dubbed “deceptive” or “misleading”, produce mostly wrong answers. These items yield poor calibration and poor resolution because the dominant, erroneous response tends to be endorsed with great confidence, even greater than that of the correct response. In addition, for deceptive items, group discussion amplifies rather than mitigates error while enhancing confidence in the erroneous response. Can participants identify deceptive items when they are warned about their existence? It is argued that people's ability to discriminate between deceptive and non-deceptive items is poor when the erroneous responses are based on the same process assumed to underlie correct responses. Indeed, participants failed to discriminate between deceptive and non-deceptive perceptual items when they were warned that some of the items (Experiment 1) or exactly half of the items (Experiment 2) were deceptive. A similar failure was observed for general-knowledge questions (Experiment 3) except when participants were informed about the correct answer (Experiment 4). Possibly, for these tasks, people cannot escape the dangers lurking in deceptive items. In contrast, the results suggest that participants can identify deceptive problems for which the wrong answer stems from reliance on a fast, intuitive process that differs from the analytic mode that is likely to yield correct answers (Experiment 5). The practical and theoretical implications of the results were discussed.