Reinforcement learning for closed-loop optimisation of spatiotemporal stimulation in patterned neuronal networks
Curation statements for this article:-
Curated by eLife
eLife Assessment
This study presents an open-source reinforcement learning framework for the real-time, closed-loop optimization of spatiotemporal electrical stimulation in engineered neuronal networks. Using single-spike-resolution activity as continuous feedback, the authors provide solid evidence that their platform can identify stimulation patterns that drive specific activity motifs within a structurally constrained four-node circuit. While this reproducible system offers a valuable and accessible tool for interacting with biological neural networks in an adaptive manner, further validation is needed to determine how well these stimulation strategies generalize to larger, unstructured, or more conventional network architectures.
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (eLife)
Abstract
Understanding how neuronal circuits transform inputs into outputs requires systematic perturbation under controlled conditions. In vitro neuronal networks can be cultured on microelectrode arrays (MEAs) that allow stimulation and recording, and microfluidic patterning can constrain network topology to yield stable stimulation-evoked responses. Yet, the space of possible spatiotemporal stimulation patterns remains too large for exhaustive exploration. Additionally, the evoked responses depend on prior stimulation history. Here, we embedded topologically constrained biological neuronal networks in a closed-loop reinforcement learning (RL) framework that sends electrical stimuli to the MEA and evaluates the evoked responses to efficiently identify stimulation patterns that evoke specific target activity motifs. We extend inkube, a low-cost, open-source electrophysiology system from off-the-shelf components, with closed-loop electrophysiology. This enables reliable delivery of stimulation at single-sample precision with millisecond-range round-trip times. It also allows independent RL agents to control multiple networks simultaneously. We first demonstrated that stimulation-evoked responses in engineered recurrent networks were stable and separable across the action space over hours of continuous operation. We characterised the dependency of responses on prior stimulation history, finding state dependence in a subset of stimulus pairs. We then benchmarked different RL agents, multi-armed bandits (MABs) and linear contextual bandits (LCBs), on the task of identifying stimulation patterns that maximise the length of clockwise-circular firing sequences. All agents learned to improve reward during training with respect to random stimulation. Agents converged on non-trivial stimulation patterns that span the full action space rather than mirroring the target motif. LCBs exploited the identified state dependence through action switching, yielding measurable reward benefits for specific action pairs, though this did not translate into overall performance gains over MABs. All hardware designs and software are publicly available, providing an accessible platform for goal-directed functional characterisation of engineered neuronal networks at single-spike resolution.
Article activity feed
-
eLife Assessment
This study presents an open-source reinforcement learning framework for the real-time, closed-loop optimization of spatiotemporal electrical stimulation in engineered neuronal networks. Using single-spike-resolution activity as continuous feedback, the authors provide solid evidence that their platform can identify stimulation patterns that drive specific activity motifs within a structurally constrained four-node circuit. While this reproducible system offers a valuable and accessible tool for interacting with biological neural networks in an adaptive manner, further validation is needed to determine how well these stimulation strategies generalize to larger, unstructured, or more conventional network architectures.
-
Reviewer #1 (Public review):
Summary:
The manuscript presents several reinforcement learning (RL) approaches to control activity in small, ring-shaped neural networks. The paradigm is to find temporally and spatially patterned stimuli applied to axons that maximize the length of activity propagation along the network. Several RL versions were compared. Some RL-designed stimulation patterns worked better than others.
Strengths:
The work is technically and statistically very sound, and important controls were done that are missing in some earlier work along this line of research. The statistics and technical solutions seem solid, though I am no specialist in RL. The figures are well done and informative, with matching quality of the captions. The work presented is of value mainly for someone who wants to build a good closed-loop …
Reviewer #1 (Public review):
Summary:
The manuscript presents several reinforcement learning (RL) approaches to control activity in small, ring-shaped neural networks. The paradigm is to find temporally and spatially patterned stimuli applied to axons that maximize the length of activity propagation along the network. Several RL versions were compared. Some RL-designed stimulation patterns worked better than others.
Strengths:
The work is technically and statistically very sound, and important controls were done that are missing in some earlier work along this line of research. The statistics and technical solutions seem solid, though I am no specialist in RL. The figures are well done and informative, with matching quality of the captions. The work presented is of value mainly for someone who wants to build a good closed-loop stimulation system to experiment with neuronal networks in-vitro.
Weaknesses:
The manuscript appears undecided about whether it wants to be about RL control, about a technical implementation of long-term stimulation in vitro, or about the properties of neuronal networks and interaction with them. The introduction is well written, comprehensive and insightful, focusing on biological aspects. Methods are then extensively about RL algorithms, without explaining why several were used or why these in particular, but with specialist language hard to understand for neuroscientists. The results then quantitatively compare the performance of the RL but do not really explain what this teaches us about neuroscience or what we learn about the networks beyond that they can be stimulated for longer propagation patterns. Extensive supplementary material almost advertises the hardware built by the team. Some figures suggest, though I'm not 100% certain about this, that different algorithms find different optimal stimulation patterns in the same network - which I find puzzling. What then does this tell us about the stimulation patterns and the variability of the responses? There is some speculative mechanistic reasoning, but no data to support this further. The conclusions hardly address the initial motivation of the manuscript. To me, it is not clear what can be learned that was not, in one way or another, presented previously, with as well as without RL.
-
Reviewer #2 (Public review):
The authors developed a reinforcement learning framework to identify stimulation patterns that produce a predefined activity pattern in neuronal networks cultured on a microelectrode array. The main contribution of this study lies in the development and validation of the experimental framework, rather than in providing new insight into neuronal response properties or the biological mechanisms underlying network activity.
A major strength of the work is that the framework is implemented using open-source software. Electrical stimulation through microelectrode arrays is already widely used, and the proposed approach is therefore likely to be of interest to researchers seeking to automate the exploration and optimization of stimulation parameters. The authors also provide the software, hardware configuration, …
Reviewer #2 (Public review):
The authors developed a reinforcement learning framework to identify stimulation patterns that produce a predefined activity pattern in neuronal networks cultured on a microelectrode array. The main contribution of this study lies in the development and validation of the experimental framework, rather than in providing new insight into neuronal response properties or the biological mechanisms underlying network activity.
A major strength of the work is that the framework is implemented using open-source software. Electrical stimulation through microelectrode arrays is already widely used, and the proposed approach is therefore likely to be of interest to researchers seeking to automate the exploration and optimization of stimulation parameters. The authors also provide the software, hardware configuration, and experimental data, which substantially improves transparency and should facilitate reproduction and further development of the system by other laboratories.
The experiments provide a useful proof of concept showing that the framework can search for stimulation patterns associated with the predefined task in the tested neuronal network. The evidence is solid for demonstrating the feasibility of the approach within this specific experimental configuration. In particular, the closed-loop integration of stimulation, recording, evaluation of the neuronal response, and subsequent selection of stimulation patterns is clearly implemented and experimentally tested.
An important limitation is that the framework was evaluated using a relatively small and highly structured network. Four stimulation electrodes were positioned around a single network, and stimulation was delivered to microchannels in which axons were concentrated. This configuration is well suited to the initial demonstration, but it remains uncertain whether the same approach will perform similarly in conventional monolayer dissociated cultures, larger networks, or systems with different numbers and spatial arrangements of electrodes. Additional validation across a broader range of network structures and experimental configurations would therefore be needed to establish the general applicability of the framework.
Overall, this study presents a valuable and reproducible methodological framework for the closed-loop optimization of electrical stimulation in cultured neuronal networks. Its likely impact lies primarily in providing an accessible experimental and computational platform that can be adapted for studies requiring systematic exploration of stimulation patterns, although the extent to which the current findings generalize beyond the tested network configuration remains to be determined.
-
Reviewer #3 (Public review):
Summary:
This study integrates living neuronal circuits with a real-time reinforcement-learning framework to enable closed-loop optimization of directional firing sequences at single-spike resolution. By using the spatiotemporal structure of neuronal firing as feedback, the system adaptively searches for stimulation patterns that increase a predefined propagation reward. The work represents an exciting step toward systematic and adaptive control of living neuronal networks.
Strengths:
The study presents a cutting-edge closed-loop platform that combines neuronal cultures, high-temporal-resolution electrophysiology, and reinforcement learning with millisecond-scale latency. The ability to optimize directional activity propagation at single-spike resolution is particularly novel. More broadly, the work provides …
Reviewer #3 (Public review):
Summary:
This study integrates living neuronal circuits with a real-time reinforcement-learning framework to enable closed-loop optimization of directional firing sequences at single-spike resolution. By using the spatiotemporal structure of neuronal firing as feedback, the system adaptively searches for stimulation patterns that increase a predefined propagation reward. The work represents an exciting step toward systematic and adaptive control of living neuronal networks.
Strengths:
The study presents a cutting-edge closed-loop platform that combines neuronal cultures, high-temporal-resolution electrophysiology, and reinforcement learning with millisecond-scale latency. The ability to optimize directional activity propagation at single-spike resolution is particularly novel. More broadly, the work provides a compelling framework for interacting with biological neural networks in an adaptive rather than purely predefined manner.
Weaknesses:
Several aspects of the analysis and interpretation require further clarification. In particular, the definition and computation of directional propagation and reward are not always clear. The generalizability of the optimized stimulation patterns across cultures also remains unclear.
-
Author response:
eLife Assessment
This study presents an open-source reinforcement learning framework for the real-time, closedloop optimization of spatiotemporal electrical stimulation in engineered neuronal networks. Using single-spike-resolution activity as continuous feedback, the authors provide solid evidence that their platform can identify stimulation patterns that drive specific activity motifs within a structurally constrained four-node circuit. While this reproducible system offers a valuable and accessible tool for interacting with biological neural networks in an adaptive manner, further validation is needed to determine how well these stimulation strategies generalize to larger, unstructured, or more conventional network architectures.
We thank the editors and reviewers for their kind and insightful words, as well as …
Author response:
eLife Assessment
This study presents an open-source reinforcement learning framework for the real-time, closedloop optimization of spatiotemporal electrical stimulation in engineered neuronal networks. Using single-spike-resolution activity as continuous feedback, the authors provide solid evidence that their platform can identify stimulation patterns that drive specific activity motifs within a structurally constrained four-node circuit. While this reproducible system offers a valuable and accessible tool for interacting with biological neural networks in an adaptive manner, further validation is needed to determine how well these stimulation strategies generalize to larger, unstructured, or more conventional network architectures.
We thank the editors and reviewers for their kind and insightful words, as well as their constructive feedback and assessment. We agree that in its current stage, the manuscript does not convincingly argue, that specific stimulation strategies applied to one network generalise well to other network architectures. We intend to address this problem by adjusting the focus of the manuscript to be more on the platform than on any specific neuroscience claim, as we believe that this is the more useful angle for the community at large. We also intend to discuss the transferability of our results between the presented culture system and other neuroscience systems.
Based on the reviewer comments, we will further give a more approachable introduction to reinforcement learning to make the concept more accessible to a wider audience. We will also motivate the algorithm selection in more depth.
In the following, we will discuss the comments put forward by the reviewers and how we intend to adapt our manuscript to address them. In this provisional response, we will only focus on major concerns. Minor comments, where we are following the suggestions made by the reviewers directly, will not yet be discussed.
Reviewer #1 (Public Review):
The manuscript appears undecided about whether it wants to be about RL control, about a technical implementation of long-term stimulation in vitro, or about the properties of neuronal networks and interaction with them. The introduction is well written, comprehensive and insightful, focusing on biological aspects. Methods are then extensively about RL algorithms, without explaining why several were used or why these in particular, but with specialist language hard to understand for neuroscientists. The results then quantitatively compare the performance of the RL but do not really explain what this teaches us about neuroscience or what we learn about the networks beyond that they can be stimulated for longer propagation patterns. Extensive supplementary material almost advertises the hardware built by the team. Some figures suggest, though I’m not 100% certain about this, that different algorithms find different optimal stimulation patterns in the same network - which I find puzzling. What then does this tell us about the stimulation patterns and the variability of the responses?
We thank the reviewer for pointing out these issues with our manuscript. The goal of our work is the hardware framework. We plan to highlight this more in the revised version. The RL part will be revised with less specialist language and will get less emphasis in the next version of our manuscript. We will also discuss why different algorithms seem to find different solutions, which can be traced back to having different networks and agents finding different local maxima.
By clearly setting the scope of our manuscript on the hardware aspects and revising the RL part to be seen more as one possible closed-loop control paradigm, we further intend to address the concerns put forward by the reviewer regarding our focus on RL in other parts of the manuscript.
Reviewer #2 (Public Review):
An important limitation is that the framework was evaluated using a relatively small and highly structured network. Four stimulation electrodes were positioned around a single network, and stimulation was delivered to microchannels in which axons were concentrated. This configuration is well suited to the initial demonstration, but it remains uncertain whether the same approach will perform similarly in conventional monolayer dissociated cultures, larger networks, or systems with different numbers and spatial arrangements of electrodes. Additional validation across a broader range of network structures and experimental configurations would therefore be needed to establish the general applicability of the framework.
We thank the reviewer for their feedback. They are right to point out that the manuscript here focuses purely on small and highly structured networks. With the presented experiments, our manuscript does not and cannot make any biological claims about how neurons communicate. However, with this manuscript, we also do not intend to do so, as such an endeavour would be out of scope. In the next version of our manuscript, we will better highlight the focus of our manuscript, which lies on the hardware framework itself. Furthermore, we will discuss in the outlook the generalisability and scalability concerns raised by the reviewer in more detail.
Reviewer #3 (Public Review):
Several aspects of the analysis and interpretation require further clarification. In particular, the definition and computation of directional propagation and reward are not always clear. The generalizability of the optimized stimulation patterns across cultures also remains unclear.
We thank the reviewer for their insightful feedback. We agree with them and will discuss (1) the currently implemented reward based on directional propagation of activity and (2) the limitations and aspects influencing reward algorithm selection in more detail in our revised version of the manuscript. We further believe that we cannot make any claims about generalisability between cultures or when changing the experimental paradigm.
-
