Applying multimodal learning analytics to naturalistic recordings of clinical simulations: Towards an accurate and scalable pipeline for automated feedback generation

Educational audiovisual recordings, spanning domains from teacher training to clinical simulations, often remain underutilized due to the intensive human labor required for comprehensive analysis and timely feedback. This study addresses this challenge by developing and evaluating a multimodal learning analytics (MmLA) pipeline that integrates state-of-the-art visual, speech, and language models to automatically capture complex communication skills. 244 medical and social work students practicing breaking bad news in a team-based simulated clinical scenario. We developed and evaluated the MmLA pipeline, which integrates speech extraction, gaze tracking, and semantic analysis using a Large Language Model (LLM). We empirically evaluated the pipeline by measuring the accuracy of each feature-extraction stage, including diarized transcript generation, visual-attention pattern tracking, and LLM-based semantic evaluation of discourse, against human assessors, and by comparing the final automated predictions with human-generated scores. The developed MmLA pipeline effectively extracts and fuses features from naturalistic videos of standardized patient simulations, approaching human-level accuracy, though diarization remains challenging with a moderate 29.9 % SER (lower is better). The predictive model using behavioral, verbal, and semantic features achieved 81.82 % accuracy in assessing interaction comfort, with eye contact emerging as the most influential predictor. By leveraging the most common multimodal data sources, such as video and audio, the study provides a scalable methodological blueprint that can be adapted to diverse educational settings to foster communication skills. • First MmLA pipeline for automated feedback generation for breaking bad news simulation. • Provides methodological blueprint that can be adapted to diverse educational settings. • Machine learning model predicts instructor score with 81.82 % accuracy. • Automates identification of teachable moments in clinical communication training.