The user experiences of AI-based clinical decision support systems and implications for usage over time: A scoping review

Objective: Artificial intelligence (AI)-based clinical decision support systems (AI-CDSS) have the potential to improve many facets of care, whether aimed toward unlocking new analysis methods, improving efficiency, or increasing patient safety. As AI-CDSS begins to see further real-world usage, there is an urgent need to understand how user experiences develop temporally, more so given these systems dynamic iterative nature. To explore this gap, we conducted a scoping review aiming to map user experiences with AI-CDSS, barriers, and facilitators and synthesize an overview of experiences temporally. Method: Following the scoping review methodology of Arksey and O'Malley, three databases were searched with 257 records retrieved, 16 of which met the inclusion criteria. After identifying reported experiences, we carried out a reflexive thematic analysis and "best-fit" synthesis to explore reported user experiences over time. Results: Nine overall user experience themes spanning two domains emerged from our analysis with 23 sub-themes. Themes include clinical context, clinical users, learnability, usability, trust and usefulness. Temporal mapping of experiences highlights their dynamic, interconnected nature and how these develop over time. These findings enrich the current understanding of how experiences with AI-CDSS develop over time, presenting implications for design. We additionally discuss gaps in current knowledge and opportunities for future work. Conclusions: Long-term experiences, particularly those occurring as a result of changes in model performance or as a result of iterative updates to AI-CDSS are sparsely described. Future work in this area should explore in more detail how experiences unfold over time and alongside AI-CDSS as they evolve.