Darshan Thota, David Alt, Sriram Venkatesan, Jesus Caban
Failure to generate an accurate response can delay decision-making and increase cognitive burden and risk. Confidently incorrect responses are particularly dangerous. Although LLMs can generate useful clinical information, they remain vulnerable to biases and factual errors. Human oversight, training, and model tuning are necessary before utilizing military data.
OBJECTIVE: To evaluate the performance of Large Language Models (LLMs) within the Military Health System (MHS), specifically assessing for bias, hallucinations, and safety risks.
METHODS: In late 2024, 47 clinicians across 13 military treatment facilities evaluated 8 blinded LLMs. Using standardized clinical vignettes, 835 conversations were reviewed for demographic bias, hallucinations, classification errors, and safety risks.
RESULTS: Demographic bias was present in 82 conversations (9.8%), hallucinations in 85 (10.2%), classification errors in 19 (2.3%), and safety risks in 78 (9.3%). Classification errors were linked to military-specific jargon. Safety risks were most prominent in overdose, critical care, and psychiatric scenarios.
CONCLUSION: Failure to generate an accurate response can delay decision-making and increase cognitive burden and risk. Confidently incorrect responses are particularly dangerous. Although LLMs can generate useful clinical information, they remain vulnerable to biases and factual errors. Human oversight, training, and model tuning are necessary before utilizing military data.