Sequential sensitivity analysis of multimodal large language models for rare orbital disease detection

Delayed diagnosis of rare orbital diseases is attributed to limited clinical awareness. Building on prior evidence of multimodal large language models (MLLMs) for detecting common ocular conditions, this study aims to evaluate whether integrating multimodal clinical data can enhance the diagnostic accuracy of MLLMs for rare orbital diseases. We conducted a multinational, multiracial, retrospective study. Two datasets were analyzed: Dataset 1, containing 6,786 single-eye photographs from China, was used to fine-tune a contrastive language-image pre-training (CLIP) for preliminary classification of healthy eyes, orbital diseases, and non-orbital diseases, and to compare its performance against three traditional models and three next-generation models. Dataset 2, comprising 170 participants from China, Singapore, and Thailand, was used to evaluate a MLLM (GPT-4o-Latest). Sequential sensitivity analysis assessed the impact of adding external eye photographs, chief complaints, racial information, and diagnostic reasoning prompts. An AI agent combining the CLIP model with GPT-4o-Latest was further evaluated. The model’s ability to generate medical reports and examination recommendations was also assessed. Here we show that the CLIP model achieves 90.21% preliminary detection accuracy, surpassing all baseline models. MLLM detection accuracy improves significantly with the inclusion of multimodal inputs. When relying on external eye images, the top-5 accuracy is 25.68%. The combined agent raises top-5 accuracy to 85.29%. Generated reports and recommendations display high accuracy, readability, completeness, and low potential for harm. Our study demonstrates the potential of MLLM in improving diagnostic accuracy and supporting clinical decision-making for rare orbital diseases. Orbital diseases affect the eye socket. Limited clinical awareness often results in delayed diagnosis of rare orbital diseases. While computational models trained on large datasets, particularly large language models (LLMs), including multimodal large language models (MLLMs), have shown potential in diagnosing common eye conditions, their effectiveness in diagnosing rare orbital diseases remains unclear. We integrated various data sources to see if this could improve the accuracy of MLLMs in diagnosing rare orbital diseases. Our model provided high-quality medical reports and examination recommendations. These findings show that MLLMs could be used to improve diagnostic accuracy and support clinical decision-making for rare orbital diseases. Lei et al. present a multimodal large language model for the classification of orbital diseases. They find that their model achieves high preliminary detection accuracy, with strong readability and completeness.