Artificial Intelligence-Augmented Standardized Patient Models for AETCOM (Attitude, Ethics, and Communication) Competency Evaluation: A Pilot Study.

INTRODUCTION: Communication skills are a core competency in the AETCOM (Attitude, Ethics, and Communication) module of the Competency-Based Medical Education curriculum. They are essential for building trust, taking accurate history, explaining interventions, obtaining informed consent, showing empathy, and managing challenging situations. Traditional assessments, such as Objective Structured Clinical Examinations (OSCEs), standardized patients, and mini clinical evaluation exercise (mini-CEX), are reliable but resource-intensive, are limited in frequency, and often provide delayed feedback. METHOD: The study was conducted at Pondicherry Institute of Medical Sciences, Puducherry, India, after obtaining approval from the Institutional Research and Ethics Committee. Following expert discussion, informed consent was selected as the AETCOM competency to assess. To achieve the study objectives, two AI chatbots were developed: a virtual standardized-patient chatbot simulating a spinal injury scenario and an AI-based evaluation chatbot incorporating a 19-item content-validated scoring rubric. Thirty-three interns interacted with the virtual patient, and anonymized transcripts were independently evaluated by the AI system and five subject experts. AI reliability was assessed using intraclass correlation coefficients (ICCs), and agreement between AI and expert scores was evaluated using Bland-Altman analysis. RESULT: Expert ratings showed moderate reliability individually (ICC (2, 1) = 0.536) but improved significantly when averaged (ICC (2, 5) = 0.852), highlighting the benefit of consensus scoring. AI scoring demonstrated excellent internal reliability (ICC (2, 1) = 0.904; ICC (2, 5) = 0.979) across repeated assessments. Bland-Altman analysis revealed a small negative bias (-0.529) with acceptable limits of agreement (-1.998 to 0.940), indicating close alignment with expert consensus. CONCLUSION: The findings show that AI is highly reproducible, consistent, and standardized. AI virtual patients provide immediate feedback, support repeated practice, and reduce variability in assessment. Combined with expert assessment, AI-based systems may offer a scalable, standardized, and learner-centred approach to communication-skills assessment that warrants further evaluation in larger multicentre studies. This approach has the potential to enhance communication-skills training and assessment by providing standardized, reproducible, and timely feedback. Further studies are required to evaluate its impact on learner performance and clinical practice.