Modeling Complex Effects and Individual Variability in Multi-Paradigm fMRI with Nonlinear Mixed Models

Functional magnetic resonance imaging (fMRI) data are inherently complex, characterized by high dimensionality, intricate inter-regional dependencies, and substantial individual variability across experimental paradigms. Traditional linear mixed models (LMMs) provide a principled framework that models population-level fixed effects while estimating variance components arising from subject-level random effects; however, they often fail to adequately capture nonlinear relationships inherent in neuroimaging data. To address these limitations, we introduce the nonlinear mixed model (NMM) approach, an innovative extension of the LMM framework that integrates neural networks to flexibly model complex fixed-effect relationships while preserving the random-effects structure to account for individual differences. NMM advances fMRI analysis by: (1) identifying robust functional connectivity (FC) patterns consistently observed across multiple paradigms; (2) leveraging SHapley Additive exPlanations (SHAP) analysis to provide post-hoc interpretability of the nonlinear fixed effects, quantifying how age, sex, and paradigm contribute to predicted FC and how these effects are distributed across large-scale brain networks; and (3) using subject-specific random effects as neural fingerprints that not only show systematic variability across attention and default mode systems but also predict standardized cognitive scores, demonstrating biological relevance. Applied to the Philadelphia Neurodevelopmental Cohort (PNC) across emotion, n-back, and resting-state paradigms, NMM achieved superior model fit relative to classical LMMs, as evidenced by lower mean squared error (MSE) in predicting FC. This framework offers a statistically rigorous and practically explainable approach for modeling large-scale FC from modest covariates while explicitly separating population-level effects from stable individual variability in functional brain organization.