A multimodal cross-temporal fusion system for complementary evaluation of neoadjuvant chemotherapy in breast cancer.

PURPOSE: Neoadjuvant chemotherapy (NAC) has been established as a standard treatment for breast cancer. We aimed to develop a deep-learning multimodal system using longitudinal cross-temporal dynamic contrast-enhanced MRI (DCE-MRI) and clinical data to provide complementary evaluation of NAC. METHODS: Here, we developed a multimodal automatic NAC assistance system (MANAS), which leverages a dual-pathway attention-based multi-scale feature fusion framework and a multicenter dataset. This dataset comprises pre- and post-NAC DCE-MRI and associated clinical information of 1,515 patients with locally advanced breast cancer. The performance of MANAS was evaluated using various metrics. Confidence-based decision criteria were employed to defer low-confidence cases to clinicians. RESULTS: MANAS consisted of three modules from breast cancer diagnosis to follow-up treatment, designed for predicting molecular subtype, NAC response, and residual tumor burden, respectively. MANAS achieved area under the curves (AUCs) of 0.8066 (95 % CI: 0.7552-0.8599), 0.7748 (95 % CI: 0.7210-0.8275) and 0.7661 (95 % CI: 0.6980-0.8252) on the internal, pooled external and I-SPY2 test sets for molecular subtype prediction; AUCs of 0.8400 (95 % CI: 0.7903-0.8896) and 0.8299 (95 % CI: 0.7886-0.8687) on the internal and pooled external test sets for NAC response prediction; and AUCs of 0.8885 (95 % CI: 0.7891-0.9657), 0.8628 (95 % CI: 0.8047-0.9132) and 0.8695 (95 % CI: 0.7856-0.9432) on the internal, pooled external and I-SPY2 test sets for residual tumor burden prediction. Confidence-based decision criteria could enhance their predictive performance. CONCLUSION: Our MANAS serves as a valuable complementary tool, providing longitudinal supportive evidence throughout the NAC process, from initial diagnosis to surgical intervention.