A benchmark dataset for evaluating Syndrome Differentiation and Treatment in large language models

Abstract The emergence of Large Language Models (LLMs) within the Traditional Chinese Medicine (TCM) domain presents an urgent need to assess their clinical application capabilities. However, such evaluations are challenged by the individualized, holistic, and diverse nature of TCM’s “Syndrome Differentiation and Treatment” (SDT). Existing benchmarks are confined to knowledge-based question-answering or the accuracy of syndrome differentiation, often neglecting assessment of treatment decision-making. Here, we propose a comprehensive benchmark derived from clinical cases, classical case records, and authoritative examination questions. Designated TCM-BEST4SDT, the dataset comprises 600 questions covering four tasks: TCM Basic Knowledge, Medical Ethics, LLM Content Safety, and SDT. Data annotation adheres to a rigorous three-stage pipeline consisting of expert annotation, mutual cross-validation, and independent third-party review. The evaluation framework integrates three mechanisms, namely selected-response evaluation, judge model evaluation, and a specialized reward model employed to quantify prescription-syndrome congruence. Crucially, a controlled evaluation on the Qwen3 series demonstrated a strictly monotonic correlation between performance and model scale, verifying the benchmark’s sensitivity in discriminating model capabilities. This study establishes a standardized evaluation framework for intelligent TCM research, objectifies the personalized diagnostic logic of complementary medicine, and facilitates the optimization and application of large language models in digital medicine.