Ruilong Xu, Huan Fang
TriFuse effectively distills high-value medical training data through multi-signal progressive filtering, significantly reducing computational overhead while maintaining model performance. This framework provides a scalable, low-cost solution for efficient medical language model training in resource-constrained environments.
OBJECTIVE: Large-scale medical instruction datasets often contain substantial noise and redundancy, leading to high training costs without significant performance gains. Existing data selection and distillation methods rely on single static criteria, failing to capture the diversity and dynamic nature of medical samples during learning. This study proposes TriFuse, a three-stage data distillation framework that integrates multiple complementary learning signals for efficient training of medical question-answering models.
METHODS: The framework comprises three stages: (1) Proxy-student filtering: utilizing masked language modeling loss to remove information-deficient samples; (2) Gradient-median filtering: under the guidance of a weak teacher model, retaining samples with stable and moderate gradient contributions to suppress noise and outliers; (3) Forgettable-count filtering: tracking sample stability across training epochs to identify and remove samples exhibiting unstable learning behaviors. The framework was applied to the large-scale medical instruction dataset MedS-Ins to construct a refined subset (MedCore) comprising less than 30% of the original data.
RESULTS: Models trained on MedCore (28% of the original data) achieved comparable or superior performance across multiple tasks while substantially reducing training time and computational cost compared with full-data training. MedCore-Llama outperformed GPT-4 on French medical QA (65.6%) and HeadQA (63.2%), and achieved state-of-the-art results in Participant Extraction (82.61%) and medical concept explanation. Ablation studies confirmed the necessity of all three filtering stages.
CONCLUSION: TriFuse effectively distills high-value medical training data through multi-signal progressive filtering, significantly reducing computational overhead while maintaining model performance. This framework provides a scalable, low-cost solution for efficient medical language model training in resource-constrained environments.