Spatiotemporal structure and substates underly emotional signalling in facial expressions and speech movements

From overt emotional displays to a subtle eyebrow raise during speech, facial expressions are key cues for social interaction. How these inherently dynamic signals encode emotion remains only partially understood. Moreover, despite the ubiquity of speech in face-to-face interactions, the dynamics of facial emotion signals during speech have been overlooked. In Study 1 we recorded participants’ facial movements signalling happy, sad and angry emotions in Expression-only and Emotive-speech conditions. We then employed a data-driven pipeline integrating facial motion quantification, spatiotemporal classification and clustering to investigate the structure and function of facial dynamics in signalling emotion. Results reveal that a few fundamental spatiotemporal patterns reliably differentiated emotion in non-verbal expressions and emotive speech facial signals. Furthermore, we identified transient substates – or dynamic phases-that are diagnostic of emotion and conditions. A perceptual validation with naive observers (Study 2) showed that the low-dimensional spatiotemporal structure captures meaningful cues that closely predicts human emotion categorisations. We argue that this low-dimensional spatiotemporal structure optimises both the transmission and perception of dynamic facial emotion signals. This work contributes insights into face-to-face emotion signalling and provide a versatile framework for modelling dynamic social cues and can inform the design of expressive emotive capabilities in social agents.