Ao Luo, Chayanon Phucharoen, Nichapat Sangkaew
Authentic emotion recognition is vital for future tourism marketing, yet single-modality approaches overlook the disparity between performed narratives and visual reality. This study introduced a within-subject, cross-context multimodal AI framework grounded in Plutchik’s emotion theory and Goffman’s self-presentation lens to quantify emotional differences in paired tourist selfies and captions. Analyzing user-generated content, the results showed a consistent amplification of joy in captions (front-stage). In contrast, selfies revealed complex back-stage emotions, such as trust, surprise, fear, disgust, and anger. These robust disparities challenge text-only sentiment analysis and underscore the need for integrated visual – textual emotional intelligence to better capture authentic tourist experiences.