Aurélie Pham, Florian Blanchard, Antoine Monsel, Manon Allaire, Alain Carrié, Jean-Michel Davaine, Maxens Decavele, Jean-Philippe Foy, Kamila Kolanska, Sohela Moussaoui, Nadia Nathan, Marie-Christine Renaud, Laure Serresse, Yanis Tamzali, Jessica Taytard, Nada Sabourdin
During live pediatric training OSCE, inter-rater reliability was good regarding clinical skills, and moderate for general skills. Inter-rater variability might be more pronounced for students with lower overall OSCE score. Our study emphasizes the importance of a careful drafting of OSCE cases with precise and clear-cut scoring checklist, especially in stations focusing on very specialized clinical situations.
BACKGROUND AND OBJECTIVE: In France, OSCE (Objective Structured Clinical Examination) have recently been incorporated as an official assessment method for the final National Ranking Examination of undergraduate medical students. Given these high stakes, it is of paramount importance to provide high-quality OSCE training in every discipline. Our pediatrics department has developed an OSCE training program covering all areas of pediatric specialties. Students are evaluated by two independent assessors for each station. As part of a quality assessment process, this study investigated inter-rater reliability in OSCE grading between the two assessors.
MATERIALS AND METHODS: This prospective observational study was conducted from March to June 2023 in the Department of Pediatrics of Sorbonne University (Paris, France). Undergraduate medical students were evaluated during OSCE training sessions. Each student had to complete two consecutive stations. Each station was evaluated by two independent assessors. The scoring checklist was divided into two parts: "clinical skills" and "general skills," assessing the student's behavior and relational aptitudes. Evaluation scales were dichotomous for all items.
RESULTS: About 206 fifth-year medical students participated in four distinct OSCE sessions. A total of 411 stations were evaluated by 20 pairs of assessors, resulting in 822 different scores. The intraclass correlation coefficients of "total score," "clinical skills," and "general skills" scores between both assessors were 0.83 [0.64-0.93], 0.85 [0.68-0.94], and 0.50 [0.14-0.75], respectively. Inter-rater reliability was influenced by overall OSCE score and varied significantly between OSCE scenarios. Variability was greater in pediatric tuberculosis and hemolytic uremic syndrome scenarios, compared to bronchiolitis.
CONCLUSIONS: During live pediatric training OSCE, inter-rater reliability was good regarding clinical skills, and moderate for general skills. Inter-rater variability might be more pronounced for students with lower overall OSCE score. Our study emphasizes the importance of a careful drafting of OSCE cases with precise and clear-cut scoring checklist, especially in stations focusing on very specialized clinical situations.