Benchmarking resting state fMRI connectivity pipelines for classification: Robust accuracy despite processing variability in cross-site eye state prediction

The rapid evolution of machine learning (ML) methods has yielded promising results in human brain neuroscience. However, the reproducibility of ML applications in neuroimaging remains limited, challenging the generalizability of inferences to broader populations. In addition to the inherent variability of the brain activity (both in healthy and pathological states), poor reproducibility is further enhanced by inconsistencies in data preprocessing techniques and methods for calculating functional connectivity (FC), which are used as parameters for brain state classification. To systematically assess the impact of abovementioned factors on ML applications to fMRI data, we benchmarked a comprehensive set of FC analysis pipelines for the classification task between fMRI data recorded in two fundamentally different states: eyes open and eyes closed. In contrast to studies involving heterogeneous clinical populations or using complex cognitive tasks, our controlled experimental design – based on two independent datasets of healthy participants collected in different laboratories – minimizes variability related to a task design or pathological brain states. Classification accuracy and reproducibility were compared for 192 distinct FC analysis pipelines, covering common preprocessing approaches, brain parcellation schemes, and connectivity metrics. Notably, we employed a cross-site validation strategy, training predictive models on data from one site and testing them on the other. Despite the substantial variability in pipeline configurations, we observed consistently high classification accuracy (∼80%), confirming that FC-based models can robustly discriminate between well-defined brain states (eye conditions) across different acquisition sites. Best results both in terms of classification accuracy and stability were observed using Pearson correlation and tangent correlation as FC, Brainnetome as atlas, and confound regression strategies based on the CompCor method. These findings highlight the resilience of rs-fMRI FC-derived characteristics to methodological variation and support their utility in the discovery of biomarkers, particularly in settings that involve stable and reproducible brain states.