Identifying Acuity and Patient Prioritization: Development of a Checklist for Student Assessment.

INTRODUCTION: Identifying acuity and patient prioritization are essential but difficult skills to teach to medical students. This study aimed to develop and assess the initial validity evidence of a checklist to be used in simulation scenarios for fourth-year medical students addressing these skills. METHODS: We applied the Kane validation framework to gather evidence supporting the use of this checklist in simulation-based assessment, focusing on the Scoring and Generalization domains. A modified Delphi method was used to create an ideal performance checklist to be used in simulation scenarios of patients requiring urgent or emergent care. Following development, a Mastery Angoff approach was used to establish a Minimum Passing Standard (MPS). Fourth-year medical students were recruited to participate in simulations, and performance was assessed using the created checklists. During the simulations, students received patient handoffs and were asked to prioritize acuity and manage evolving clinical situations across 2 simulated encounters. Simulations were scored by both real-time and video-observed raters and inter-rater reliability (IRR) was calculated. RESULTS: Twenty-five students participated in the 2023 to 2024 academic year. The expert panel finalized a 10-item checklist and the MPS was determined to be 85%. Overall, IRR was excellent, with a Cohen kappa coefficient of 0.872. Cohen kappa coefficient for individualized checklist domains ranged from 0.65 to 1. CONCLUSION: This study describes the development of a performance checklist assessing students' abilities in identifying acuity and patient prioritization, with initial validity evidence supporting Kane Scoring and Generalization inferences.