Shelby Willis, Phillip D Jenkins, Steven Bedrick, Jonathan Jesneck, Julie W Doberne, Ruchi M Thanawala
The comment component of EPA assessments offers essential context to facilitate learner progress and faculty guidance of learners. Yet comments are inconsistently provided, and when they are provided, 3/4s lack suggestions for improvement and a connection to the behavior described. Future directions include faculty development initiatives regarding high quality comment utility and EPA platform driven prompts of essential comment components.
OBJECTIVE: Entrustable professional activities (EPAs) are core assessment tools for competency based education. EPA assessments provide a comment section to enrich the context of the assessment that is crucial for learner growth. Previous studies developed the Quality of Assessment for Learning score (QuAL), a validated rating tool for assessing the quality of narrative comments in competency-based medical education. We applied QuAL scoring to a large assessment database to assess the quality of comments.
DESIGN: Assessment comments from a nationwide digital medical education platform were evaluated by QuAL with scores ranging from 0-5 with 0 indicating very low quality, 3 an average comment and 5 a high quality comment. This is based on three components (1) sufficient evidence about performance scored from 0-3 (2) offering suggestion for improvement, scored as 0 or 1 and (3) connection of suggestion to a described behavior, scored as 0 or 1. We leveraged previous work developing a publicly available machine learning model to score comments with QuAL. Python was used to automate the extraction and processing of comment data through the machine learning model.
PARTICIPANTS: Faculty from surgical training programs across the United States submitted EPA assessment comments for surgical residents and fellows.
RESULTS: Of 12,841 EPA assessments, 7,211 (56%) contained a comment. Assessments without comment were excluded from analysis. The mean overall comment score was 2.91 representing average quality. The average evidence was 2.4/3 (± 0.86). Only 29% of EPAs provided a suggestion with 26.5% connecting that suggestion to a described behavior.
CONCLUSIONS: The comment component of EPA assessments offers essential context to facilitate learner progress and faculty guidance of learners. Yet comments are inconsistently provided, and when they are provided, 3/4s lack suggestions for improvement and a connection to the behavior described. Future directions include faculty development initiatives regarding high quality comment utility and EPA platform driven prompts of essential comment components.