Evaluating reasoning models for therapy recommendations in gastrointestinal stromal tumors: expert and LLM-based evaluations of OpenAI o1 and DeepSeek-R1

PURPOSE: This study aims to evaluate two advanced reasoning LLMs in generating treatment recommendations for real-world gastrointestinal stromal tumor (GIST) cases and assess their concordance with multidisciplinary team (MDT) decisions at a certified tertiary sarcoma center. METHODS: Sixty-five real-world GIST cases from a tertiary sarcoma center were used to compare two advanced reasoning models-OpenAI o1 and DeepSeek-R1. Recommendations were generated using a multi-expert prompting strategy with current clinical guidelines as context. Five sarcoma specialists and an independent LLM (Mistral AI) evaluated alignment with MDT decisions and guideline concordance. RESULTS: OpenAI o1 achieved higher concordance with MDT decisions than DeepSeek-R1 (76.9% vs. 53.8%, p < 0.001) and more recommendations aligned with either MDT or guidelines (80.0% vs. 63.1%, p = 0.005). Inter-rater reliability among human evaluators was excellent (ICC = 0.929). The LLM judge's evaluations showed moderate agreement with human assessments (κ = 0.647). OpenAI o1 responses were significantly longer than those of DeepSeek-R1 and MDT records. CONCLUSIONS: OpenAI o1 outperformed DeepSeek-R1 in generating clinically relevant GIST therapy recommendations. The study highlights the feasibility of using LLMs both as decision support tools and as evaluators ("LLM-as-a-judge") in oncology, while emphasizing the need for expert oversight in clinical deployment.