Entropy-Guided, Optimization-Tuned Fusion for Multimodal Emotion Recognition in Stress and Depression Detection: A Gap Analysis and Proposed Framework

Multimodal emotion recognition (MER) has become a more powerful solution than unimodal approaches for stress and depression screening or other tasks, but the literature presents several common and recurring shortcomings that hinder accuracy and application. This paper provides a systematic gap analysis of the MER literature in the unimodal (facial, speech, text, EEG and physiological) and multimodal fusion research and a detailed description of a proposed AI based deep-learning framework to tackle the identified gaps directly. The results of the gap analysis, which is carried out on sixty-six publications reviewed, result in the following six identified deficiencies: (1) limited multimodal integration by using video, EEG or physiological signals in the same framework; (2) the use of fixed or heuristically chosen fusion weights, without optimization; (3) no multimodal fusion rules mentioned in the reviewed literature based on entropy; (4) limited validation on Indian-context data; (5) scarcity of the use of metaheuristic optimization to optimize multimodal fusion; and (6) limited explicit framing of multimodal emotion recognition in the context of stress and depression detection. In the proposed method, the gaps are addressed by five components: unimodal models for modality-specific speech, facial images and EEG; a novel Deep Belief Network and Deep Recurrent Neural Network for the video and EEG/physiological branches respectively; a cross-modal heterogeneity resolution mechanism using a shared latent space across the two modalities; a novel hybridization of Bat-Rider Optimization Algorithm for the fusion rule; and a validation plan covering the DEAP benchmark and an original Indian-context multimodal dataset. Finally, the paper explicitly traces each part of the proposed approach to the gap it is filling, giving a clear foundation for the design of the framework.