Evaluating the Validity of Automated Emotion Detection from Spontaneous Expressions across Modalities: Insights from the PEM Dataset

Affective computing research aims to accurately detect users' emotional states to enhance human-computer interaction. While existing studies demonstrate the ability of affective computing models to accurately detect posed expressions of emotions, there is limited investigation into their validity in detecting natural or spontaneous emotional expressions. This study examines the validity of several models in accurately detecting six basic emotional states from spontaneous expressions. Data were collected from 114 participants who viewed emotionally provocative video clips, while multi-modal information on their emotional experiences was captured (questionnaires, video recordings, audio recordings, and text). Open-source software was used to detect basic emotional states from facial expressions (Py-Feat), text (National Research Lexicon), and audio (an LTSM-DNN model). The construct validity of these measurements was assessed by correlating their relationship to self-reported emotional experiences (positive convergent validity) and Big Five personality traits (both discriminant and negative convergent validity). Data analyses demonstrated limited construct validity for automatic detection of emotion through facial expression and text, and poor construct validity through audio. These results suggest that existing models are not yet capable of accurately measuring basic emotional states in real-life settings, highlighting the critical need for improving affective computing models using spontaneous emotional data to enhance their accuracy. The Personality-Emotion Mapping (PEM) dataset is provided as a resource for training and testing these models, with the aim of facilitating more effective human-computer interactions.