Extended reality and multimodal artificial intelligence for human performance: A scoping synthesis

We explored the current status and future outlook of the use of AI-supported multimodal extended reality for human performance with a systematic scoping review and a machine learning-based semi-automatic approach supplemented by a human pattern review. Specifically, text mining and topic modeling identified twenty-six topics as the optimal solution for classifying the included literature. These classifications are salient in the extended reality technologies used (i.e., virtual and augmented reality), the multimodal techniques involved (i.e., haptic, eye, and brain tracking), and the AI technologies leveraged (i.e., machine learning accuracy). Through a human pattern review, we distilled topical patterns on 1) Goals and Outcomes of AI-supported Multimodal Extended Reality for Human Performance; 2) Disentangling the Dynamics of User Interactions in Virtual Environments with Multimodal Strategies; 3) Synergistic Multimodality with Emerging AI Technologies Using Machine Learning, Large Language Models, and Vision Language Models; 4) Fostering Engaging, Interactive and Immersive Human Experiences through Ambient Intelligence.