Strength in Numbers, Power in Subjectivity: Scalable Modeling of Individual Annotators for Emotion Recognition Within and Across Corpora.

Emotion expression and perception are nuanced, complex, and highly subjective processes. As a result, when multiple annotators label emotional data, the resulting labels generally contain high degrees of variability. This variability is often addressed by averaging annotator labels into a single averaged or majority-voted ground-truth label. However, an averaged label may mask variation among the perception of individuals, which is at the core of subjective perception. This is an issue because inter-annotator variability contains valuable information about both the sample being evaluated and the annotators themselves. Previous works have proposed methods to retain this variability by predicting distributions of annotator perception, rather than a single label. However, distributions lose the information about the individual annotators. Other approaches have instead learned models of individual annotators. Yet, computational complexity limited these approaches to considering only small sets of annotators. In this paper, we present a new approach that learns to efficiently model arbitrarily large numbers of individual annotators. We demonstrate the efficacy of this approach, compared to previous state-of-the-art approaches, and the insights that can be gleaned once annotator variability is retained. For the first time, it is possible to ask when it is beneficial to model every single annotator, as an individual, vs. when it is beneficial instead to model a more limited number of annotators who have provided ample data.