Prospective validation and real-time implementation of an automated machine learning postoperative mortality prediction model

BACKGROUND: Machine learning prediction models require prospective validation to ensure implementation fidelity and feasibility. Our primary objective was to prospectively validate a previously reported postoperative mortality prediction model in inpatients undergoing surgery. Our secondary objective was to evaluate feasibility of a pilot clinical decision support tool. METHODS: We prospectively validated and implemented a random forest machine learning model trained to predict in-hospital mortality using data from a single academic medical centre. A reduced 32-feature model was implemented into the electronic health record (EHR) using a real-time data mart at the same institution. To assess model performance, the area under the receiver operating characteristic curve (AUROC), area under the curve precision-recall (AUCPR), and other performance measures were calculated. To assess feasibility, implementation workflow metrics were evaluated and a survey was administered to anaesthesiologists trained to use the pilot clinical decision support tool. RESULTS: The AUROC for the prospectively implemented model was 0.874 (95% confidence interval [CI] 0.860-0.887), and the AUCPR was 0.111. By comparison, the AUROC for the 58-feature model was 0.925 (95% CI 0.900-0.947), and for ASA physical status the AUROC was 0.814 (95% CI 0.802-0.827) and the AUCPR was 0.103. The implementation demonstrated feasibility through real-time data updates, automated transfer of model outputs to the EHR, and provider survey entries. CONCLUSIONS: This prospective validation and EHR implementation of a previously published random forest machine learning model predicting postoperative in-hospital mortality demonstrated acceptable real-world performance of the implemented model and feasibility of integrating such a system into clinical practice.