Modeling Face Emotion Perception from Naturalistic Face Viewing: Insights from Fixational Events and Gaze Strategies

Face Emotion Recognition (FER) is a foundational human skill crucial for social interactions and understanding others' mental states. The use of eye tracking as a tool for investigating FER has provided valuable insights into the underlying cognitive processes. In this study, we adopted an instructionless paradigm, collecting eye movement data from 21 participants to explore two distinct FER processes: free viewing and grounded FER.

We extracted fixational, pupillary, and microsaccadic events from eye movements and demonstrated their correlation with emotion perception and individual participant performance in the grounded task. By defining regions of interest on the face, we delved into the role of eye-gaze strategies in face processing and its connection to the presented emotions and emotion perception performance. During the free viewing phase, participants revealed distinct attention patterns for the different emotions. In the grounded FER tasks, participants interpreted emotions based on words, allowing for the assessment of performance and context. Interestingly, the participants' gaze patterns during free viewing were predictive of their success in the grounded FER tasks, emphasizing the importance of initial gaze behavior in emotion perception.

We leveraged features from pre-trained deep-learning models for face recognition to analyze attention during free viewing, enhancing scalability and comparability across datasets and populations. This approach also enabled the prediction and modeling of individual emotion perception performance from limited observations. Our study contributes to a deeper understanding of the relationship between eye movements and emotion perception, with implications for psychology, human-computer interaction, and affective computing. Furthermore, it opens avenues for the development of precise emotion recognition systems.