Tristan Lemke, Alexander W Marka, Philipp Prucker, Alexander Komenda, Dominik Weller, Christian J Mertens, Su Hwan Kim, Jacqueline Lammert, Thomas Huber, Markus M Graf, Sebastian Ziegelmayer, Keno K Bressem, Marcus R Makowski, Lisa C Adams, Fabian Lohöfer, Felix Busch
Commercial AI tools did not improve diagnostic accuracy and, for some findings, reduced it through increased false positives, while shortening reading time, increasing confidence, and lowering escalation. These results suggest that to avoid automation bias, maintain accuracy, and achieve efficiency gains, AI deployment requires careful local adaptation. Additionally, clinical trials are necessary to address patient outcomes.
RATIONALE AND OBJECTIVES: To evaluate the impact of four commercially available AI solutions for chest radiography on diagnostic performance, workflow efficiency, and clinical decision-making in a real-world setting.
MATERIALS AND METHODS: In this prospective, monocentric, crossover reader study, five readers (one to six years of experience) assessed 1200 consecutive patients undergoing chest radiography (1861 total radiographs) for pulmonary infiltrates, pleural effusions, mediastinal masses, pneumothorax, and pulmonary nodules under five conditions: without AI and with each of four AI algorithms. A 14-day washout period separated sessions, and case order was randomized. The reference standard was the finalized clinical report, supplemented by confirmatory CT when available. The primary outcome was diagnostic performance. Secondary outcomes included interpretation time, diagnostic confidence (5-point Likert scale), escalation to senior review, and CT recommendations.
RESULTS: AI assistance did not improve diagnostic accuracy. For pleural effusions and pulmonary nodules, accuracy decreased in several reader-tool pairings due to increased false positives. However, workflow benefits were observed: three of five residents reported shorter interpretation times, with median reductions of 6 to 17 s per case (p≤0.031). Four of five readers reported higher diagnostic confidence. AI support reduced senior consultations in selected pairings and, in one case, lowered CT escalation.
CONCLUSION: Commercial AI tools did not improve diagnostic accuracy and, for some findings, reduced it through increased false positives, while shortening reading time, increasing confidence, and lowering escalation. These results suggest that to avoid automation bias, maintain accuracy, and achieve efficiency gains, AI deployment requires careful local adaptation. Additionally, clinical trials are necessary to address patient outcomes.