Beyond Intensity: Cross-Dataset Consistency of Temporal Facial Action-Unit Dynamics as Transferable Markers of Depression

Facial behavior is a widely studied objective signal for depressive-symptom analysis, yet most systems are trained and evaluated within a single dataset, leaving it unclear whether the learned representations reflect depression or dataset-specific artifacts. Prior work has also relied on aggregate intensity statistics and small samples. We reframe the problem by asking which facial action unit (AU) features transfer across datasets, rather than which maximize within-dataset discrimination. We used time-resolved AU dynamics from a Korean cohort of 2,608 participants, including 265 with PHQ-9-defined depressive symptoms, recorded under happy and unhappy emotion-elicitation conditions. From AU time-series extracted with OpenFace, we computed 568 features spanning intensity, temporal, dynamic, peak-structure, and Duchenne (genuine smile) co-activation patterns. We then tested their directional transferability on the US DAIC-WOZ dataset, which differs in race, language, task, recording length, and setting. Within the source cohort, peak-interval features gave the strongest signal (AU26 peak-interval, Cohen’s d = −0.66), yet reversed sign externally, whereas slower temporal features and AU06 (cheek raiser) preserved their direction despite modest effect sizes. Excluding peak features raised directional agreement from 56-64% to 82-90%. Within-dataset discriminative strength therefore does not predict cross-dataset transferability, supporting directional consistency as a practical criterion for selecting transferable affective-model inputs. Smile analysis further showed that depressive symptoms were marked less by reduced smiling than by eye-mouth decoupling (d = −0.41), which survived covariate adjustment and matched the voluntary–emotional facial-pathway distinction. Being low-dimensional and non-identifying, AU time series may support privacy-conscious multisite depression research without raw video.