Integrating Multimodal Emotion Recognition with Deep Q-Learning for Adaptive Social Robot Interaction

This paper aims to enhance social interaction with robots by utilizing artificial emotional intelligence and multimodal communication systems. For this, a framework consisting of audio, video, and text channels is described as a means of expressing emotions within a common framework of emotional intelligence. Adaptive behavior is facilitated by reinforcement learning, enabling robotic behavior to be adjusted according to the level of user experience and the likelihood of task accomplishment. The experiments were conducted in various settings, including healthcare, education, and aged care. The findings obtained are significantly better than any previously reported approach in the literature, with rates for correct emotional responses of 95.6%, task success rates of 91.6%, and user satisfaction ratings of 4.8 out of 5 points in a survey. The system also exhibited an improved reaction and maintained longer, more interactive communications, which made it even more effective and efficient in the intended human-robot interactions. They also highlight the proposed system's effectiveness in addressing various problems in the field of perception, enabling robots to interact with humans. Attention focused on the integration of robotic multichip modules, ethical issues, and scaling concepts for multiple robot scenarios. This research serves as the foundation for developing interactive and socially intelligent robots that can understand the unique needs of different users and operate effectively in diverse environments.