Joint Generative Modeling of EEG and fMRI for Cognitive Task Simulation Using Adversarial Learning

Realistic simulation of human brain activity is a critical enabler for data augmentation, pri-vacy preservation, methodological benchmarking, and virtual experimentation in cognitiveneuroscience and neuroimaging-based machine learning. Most existing generative approachesfocus on unimodal signal synthesis, either electroencephalography (EEG) or functional mag-netic resonance imaging (fMRI), and therefore fail to capture the cross-modal dependenciesinherent to human cognition. In this work, we propose a novel multimodal neural activitypattern simulator based on a joint EEG-fMRI generative adversarial network (GAN) for syn-thesizing task-evoked cognitive brain activity. The proposed framework is evaluated usingtask-based recordings from the PEARL-Neuro database, using a 20-case multimodal analysiscohort for attention switching (Multi-Source Interference Task, MSIT) and working mem-ory (Sternberg task). Frequency-domain EEG band-power features and fMRI functionalconnectivity features are extracted following a robust preprocessing pipeline, where Inde-pendent Component Analysis (ICA) is applied using the frontal electrode Fp1 as an EOGproxy for artefact removal. These features are jointly modelled within a unified adversariallearning framework to preserve cross-modal structure. Evaluation using a balanced dataset(19 real plus 19 synthetic samples for attention, 18 real plus 18 synthetic samples for mem-ory) with 5-fold stratified cross-validation reports Accuracy of 0.833 and 0.909, F1-Score of0.800 and 0.889, RMSE of 0.2022 and 0.1593, and R-squared of 0.5148 and 0.7123 for theattention and memory tasks respectively. The proposed framework demonstrates that jointadversarial modelling can generate statistically plausible multimodal brain signals, support-ing privacy-preserving data augmentation and reproducible cognitive neuroscience research.These findings establish joint adversarial modelling as a powerful paradigm for multimodalbrain signal synthesis, with significant potential for adaptive human-computer interactionsystems and virtual cognitive experimentation.