A Stability-Enhanced Lasso Approach for Covariate Selection in Non-Linear Mixed Effect Model

Non-linear mixed effect models (NLMEMs) defined by ordinary differential equations (ODEs) are central to modeling complex biological systems over time, particularly in pharmacometrics, virus dynamics and immunology. However, selecting relevant covariates associated with a dynamics in high-dimensional settings remains a major challenge. This study introduces a novel model-building approach called lasso-SAMBA that integrates Lasso regression with a stability selection algorithm for robust covariate selection within ODE-based NLMEMs. The method iteratively constructs models by coupling penalized regression with mechanistic model estimation using SAEM algorithm. It extends a prior strategy (SAMBA: Stochastic Approximation for Model Building Algorithm) originally based on stepwise inclusion, by replacing this step with a penalized, stability-driven approach that reduces false discoveries and improves selection robustness. By maintaining the monotonic decrease of the information criterion through a calibrated exploration of penalization parameters, the proposed method outperforms conventional stepwise and Bayesian variable selection alternatives. Extensive simulation studies, spanning pharmacokinetic and immunological models, demonstrate the superiority of lasso-SAMBA in variable selection fidelity, FDR control, and computational efficiency. The lasso-SAMBA method is implemented in an R package. Applied to a Varicella-Zoster virus vaccination study, the method reveals robust, biologically plausible associations between parameters of the mechanistic model of the humoral immune response with early transcriptomic expressions. These results underscore the practical utility of our method for high-dimensional model building in systems vaccinology and beyond.