Gregory L. Watson, Grace L. Staples, Robin Carver, Akhil Bhargava, Carlos G. Lopez-Espina, Lee Schmalz, Farhan Ali, Peter S. Antkowiak, Saleem Azad, Ramona Berghea, Lavneet Chawla, M.J. Crisp, Alon Dagan, Francisco R. Davila, Hugo Dávila, Carmen DeMarco, Amanda Doodlesack, Aimee Espinosa, Neil R. Evans, Clinton Ezekiel, Andrew Friederich, Falgun Gosai, Alexandra Halalau, Karthik Iyer, Max S Kravitz, Niko Kurtzman, J. H. Lee, Nicholas Maddens, Roneil G. Malkani, Stockton Mayer, Vikram Oke, Ashok Palagiri, Roshni Patel, Lekshminarayan Raghavakurup, Samuel Raouf, Eric Reseland, Farid Sadaka, Deesha Sarma, Scott Smith, Tatyana Shvilkina, Matthew Sims, Sahib Singh, B. Stenson, Anwaruddin Syed, Muleta Tafa, Kurian Thomas, Sihai Dave Zhao, Ruoqing Zhu, Rashid Bashir, Bobby Reddy, Nathan I Shapiro
Objectives: To assess the interpretability and acceptance of Shapley values for making artificial intelligence/machine learning (AI/ML) tools more transparent, interpretable, and useful to clinicians. Materials and Methods: Structured assessments were conducted with 30 clinicians (15 providers; 15 nurses; 8 assessments per clinician) to evaluate their ability to understand interventional Shapley Additive exPlanations (SHAP) values, a type of Shapley value that provides individualized variable importance scores and ascertain their perspective on SHAP value utility for the use of an AI/ML sepsis diagnostic. Participants were shown the diagnostic interface for real clinical scenarios with de-identified patient data with and without SHAP values. The primary outcomes were clinician ability to correctly interpret SHAP values and clinician self-reported improvement in their understanding of how the AI/ML algorithm produced its result. Results: Participants correctly interpreted SHAP values in 235 of 240 assessments (98%; CI, 95%-99%) and reported SHAP values improved their understanding of how the algorithm produced its result in every case (240/240; 100%; CI, 99%-100%). Participants were unanimous (30/30) in preferring the interface with SHAP values over the interface without. Discussion: Clinician participants strongly preferred the device interface with SHAP values, were unanimous in reporting SHAP values improved their understanding of the AI/ML diagnostic, and scored nearly perfectly when asked to interpret SHAP values. Conclusion: These results suggest health care providers value transparency into AI/ML algorithms designed for clinical use, and that Shapley values are a useful approach to providing that transparency, which in turn may improve tool adoption and clinical utility.