Generative learning with multimodal prompts as computational model for brain responses

Exploring brain activity via high-resolution functional magnetic resonance imaging (fMRI) is important in the field of neuroscience. Traditional methodology leverages statistical analysis to associate the stimulus and brain responses, as well as identify cortical selectivity, restricted by insufficient data-driven learning. Here we address the brain activity analysis from the perspective of deep generative learning. We develop a diffusion model, NeoDiffuser, for generating and recovering fMRI data of visual cortex areas with controllable conditions. Technically, NeoDiffuser is composed of multimodal encoders, and a conditional diffusion model accommodating multimodal prompts. We demonstrate the capabilities of NeoDiffuser in simulating fMRI responses with compact guidance encoded from visual stimuli and contextual layouts, to perform stimulus-to-cortex functional association. NeoDiffuser also exhibits the ability in recovering brain signals of missing vertices, to further analyze cortex-to-cortex association. Owing to the controllable prompts, NeoDiffuser shows the feasibility of exploratory factor analysis on what impacts the neural responses. We explore the impacts of geometrical features and hemisphere-specific properties. We demonstrate that the areal cortex associations revealed by fMRI generation and recovery show high consistency with streamed visual processing. The development of NeoDiffuser displays the potential of bridging the human cognitive process and artificial neural networks.