Simulating closed-loop transcranial brain stimulation for reinforcement learning-based treatment discovery

The human brain is a complex system, where closed-loop mechanisms are vital in maintaining stability, synchronization, and adaptability of brain function. An important component of these mechanisms are neural oscillations (brain waves). Abnormal neural oscillation patterns are associated with psychiatric and neurological disorders. Brain stimulation techniques are being explored as a treatment option that can alter these abnormal oscillatory patterns. In closed-loop brain stimulation, an electric current or magnetic field stimulates brain activity, while being dynamically adjusted using feedback from ongoing brain activity captured through real-time electroencephalogram (EEG) measurements, with the aim to enhance the effectiveness of stimulation at modulating the target neural activity. Developing these treatment strategies is challenging due to the complex nonlinear dynamics of brain activity, so treatment development benefits from computer simulations. However, existing simulation tools do not integrate brain dynamics with transcranial stimulation to conduct closed-loop simulations, restricting the application of advanced methods such as reinforcement learning (RL), which refer to machine and deep learning algorithms capable of identifying complex patterns in data to discover novel decision-making strategies. Therefore, in this study, we first introduce a framework named NeuroStimEnv for simulating closed-loop brain stimulation. The tool provides the capability for researchers to integrate different models of neural circuits (exhibiting characteristics observed in Alzheimer's disease or depression), configure different electrode montage setups for stimulation and EEG measurement, and simulate treatment strategies. We have released the code as open source under the MIT license. Next, we simulate a depression microcircuit to demonstrate the feasibility of using RL to discover a transcranial alternating current stimulation treatment strategy. We demonstrate promising initial results on using RL-based algorithms for treatment selection, to successfully shift neural circuits representative of depression towards circuits representative of healthy individuals. The proposed simulation framework provides valuable insights into transcranial stimulation and enables the application of RL methods as shown in the case study.