Sensory encoding of emotion conveyed by the face and visual context

Humans rapidly detect and interpret sensory signals that have emotional meaning. The posterior superior temporal sulcus (pSTS) and amygdala are known to be critical for this ability, but their precise contributions remain contentious. Here we investigate how these structures process visual emotional cues using artificial neural networks (ANNs) to model fMRI signal acquired as human participants view complex, naturalistic stimuli. Characterizing data from two archival studies (Ns = 20, 45), we evaluated whether representations from ANNs optimized to recognize emotion from facial expressions alone or the broader visual scene differ in their ability to predict brain responses. We found that abstract representations of facial expressions were more strongly encoded in pSTS than the amygdala, whereas representations of the broader emotional scene were robustly encoded in both regions. These findings demonstrate how the pSTS operates on abstract representations of facial expressions to a greater extent than the amygdala, which more strongly encodes the broader emotional significance of visual information, depending on the context.