Trustworthy Multimodal Depression Screening via Cross-Attention Fusion and Calibrated Uncertainty

Conventional automated systems for screening for depression leverage speech/text-based features, which makes such systems sensitive to external noises, potential mistakes in automatic speech recognition (ASR), and other modality-related limitations. Besides, most current approaches fail to provide any form of uncertainty estimates, properly calibrated output probabilities, and explanation capabilities for their predictions, limiting the use of these tools within a clinical environment. In this work, we present an approach for building trustworthy depression screening systems based on a fusion of acoustic and linguistic features using a novel cross-attention-based method. Specifically, self-supervised learning techniques like wav2vec 2.0 and HuBERT models are used for extracting acoustic features from raw audio recordings. For text processing, our framework leverages DistilBERT and RoBERTa language representation models. By employing a multi-head cross-attention module, we allow our model to effectively exploit interactions between linguistic content and acoustic features. Predictive uncertainty estimates are produced by incorporating Monte Carlo dropout into the model architecture. Temperature scaling is applied for proper calibration of output probabilities. Token-level attributions are used for explaining predictions made for linguistic input, while attention coefficients for segments of audio signal correspond to explanation. Experiments conducted on a dataset of clinical interviews from the DAIC-WOZ corpus show that our method significantly outperforms audio-only, text-only, and fusion baselines, reaching an accuracy, Macro-F1, Weighted-F1, AUROC, and ECE of 0.82, 0.80, 0.81, 0.87, and 0.034 respectively. Our system also shows increased robustness against noisy audio conditions, ASR-based transcripts, and missing data.