Signal denoising through topographic modularity of neural circuits
Curation statements for this article:-
Curated by eLife
eLife assessment
This manuscript puts forward a new idea that topography in neural networks helps to remove noise from inputs. The authors show that there is a critical level of topography that is needed for network to denoise inputs. At present, the analysis is limited to inputs that are constant in time.
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (eLife)
- Neuroscience (eLife)
Abstract
Information from the sensory periphery is conveyed to the cortex via structured projection pathways that spatially segregate stimulus features, providing a robust and efficient encoding strategy. Beyond sensory encoding, this prominent anatomical feature extends throughout the neocortex. However, the extent to which it influences cortical processing is unclear. In this study, we combine cortical circuit modeling with network theory to demonstrate that the sharpness of topographic projections acts as a bifurcation parameter, controlling the macroscopic dynamics and representational precision across a modular network. By shifting the balance of excitation and inhibition, topographic modularity gradually increases task performance and improves the signal-to-noise ratio across the system. We demonstrate that in biologically constrained networks, such a denoising behavior is contingent on recurrent inhibition. We show that this is a robust and generic structural feature that enables a broad range of behaviorally relevant operating regimes, and provide an in-depth theoretical analysis unraveling the dynamical principles underlying the mechanism.
Article activity feed
-
-
Author Response
Reviewer #1 (Public Review):
Overview
In this work, the authors set to study the effects of topographic connectivity in a hierarchical model of neural networks. They hypothesize that the topographic connectivity, often observed in cortical networks, is essential for signal propagation and allows faithful transmission of signals. To study the effects of topographic connectivity on the dynamics, the authors consider a network composed of several layers. Each layer is a recurrent neural network with excitatory and inhibitory sub-populations. The excitatory neurons in each layer enervate a sub-population of the following layer. The receiving excitatory sub-population targets a specific group in the next layer and so on. This procedure leads to separate channels that carry the inputs through the network. The authors …
Author Response
Reviewer #1 (Public Review):
Overview
In this work, the authors set to study the effects of topographic connectivity in a hierarchical model of neural networks. They hypothesize that the topographic connectivity, often observed in cortical networks, is essential for signal propagation and allows faithful transmission of signals. To study the effects of topographic connectivity on the dynamics, the authors consider a network composed of several layers. Each layer is a recurrent neural network with excitatory and inhibitory sub-populations. The excitatory neurons in each layer enervate a sub-population of the following layer. The receiving excitatory sub-population targets a specific group in the next layer and so on. This procedure leads to separate channels that carry the inputs through the network. The authors study how the degree of specificity in each targeted projection, called ’modularity,’ affects signal propagation through the network.
The authors find that the network reduces noise above a critical level of network modularity: the deep layers show a clear separation of an active channel and inactive channels, despite the noisy input signal. They study how different dynamical and structural properties affect the signal propagation through the network layers and suggest that the dynamics can implement a winnertakes-all computation.
We thank the reviewer for the concise summary of our work.
Strengths and novelty
Topographic projections, in which sub-populations of neurons target specific cells in efferent populations, are common in the central nervous system. The dynamic and computation benefits of this organization are not fully understood. With their simple model, the authors were able to quantify the amount of topographic structure and selectivity in the network and study its impact on the network’s steady-state. In particular, a bifurcation point suggests a qualitative difference between networks with and without sufficient topographic modularity. The theoretical analysis in the paper is rigorous, and the mean-field study shows good agreement with computer simulations of the model.
We thank the reviewer for acknowledging the rigor of our work both in terms of theory and simulations.
The authors describe simulation results of networks with different dynamical properties, including rate-based networks, integrate-and-fire neurons, and more realistic conductancebased spiking neurons. All simulations exhibit similar qualitative behavior, supporting the conclusion that the behavior due to structural modularity will carry to more complex and biologically relevant neural dynamics.
Overall, the authors convince that the topographic structure of the network can lead to noise reduction, given that the input to the network is provided as distinct channels.
Weaknesses
The authors support their hypothesis and show a relation between topographic connection and noise reduction in their model. However, I find the study limited and struggle to see the impact it will have on the field. The paper is purely theoretical; it does not provide any physiological evidence that supports the conclusion. On the other hand, and this is the key issue, I do not find real theoretical insights in this work. In the following, I elaborate on why I hold this opinion.
We understand the reviewer’s point and therefore significantly extended our theoretical results and their conclusions in the revised manuscript (see below). We are confident that the revised manuscript provides the theoretical insights that the reviewer was asking for.
The hypothesis is that topographic projections in cortical areas allow faithful signal propagation. However, as the authors point out, reliable transmission can be achieved in other ways, such as by direct routing of information (lines 17-19). Furthermore, denoising can be accomplished by a simple feedforward network (e.g., ref 38) without E/I balance and with plasticity rules that do not require topographic connectivity. Thus, I find the computational model not well motivated.
The reviewer mentions an important point that has not been sufficiently addressed in the previous version, namely the distinguishing feature of our model. Direct routing is indeed a simple way to transmit signals, but without the possibility of denoising them. The reviewer is also right that the denoising solution in the work by Kadmon and Sompolinsky (ref 38) does not require any topographic connectivity. However, their model does not constrain feedforward connections between layers in any way. In particular, neurons can excite and inhibit other neurons (i.e., ignoring Dale’s law) in downstream layers so that feedforward input covers a much wider range, thereby extending the activity range of the target neurons and generating fixed points more easily. In the biologically more plausible setting that we study (excitatory and inhibitory populations, excitatory background input and excitatory feedforward connectivity), we find that recurrent inhibition is crucial to compensate the excitation from previous layers and the external input. Only if the recurrent inhibition is sufficiently strong does the topographic organization of feedforward connections enable denoising. This is addressed in a new section ”Critical modularity for denoising” of the revised manuscript, where we also study the case of no recurrent connectivity and excitatory recurrent connectivity (for further details, see answers below). We further extended our discussion on other forms of signal transmission and denoising (see lines 489-498).
The task studied here is a simple classification of static inputs: the efferent readout needs to identify the active channel. Again, this could be achieved by a single layer of simple binary neurons [Babadi and Sompolinsky 2014]. The recurrent connectivity and E/I balance suggest that dynamics should play an essential part in the model. However, the task is not well suited for understanding the role of dynamics.
We appreciate the reviewer’s comments and completely agree. The simple classification task we explored can certainly be performed by simpler network architectures, such as the one studied in Babadi and Sompolinsky. However, as discussed above, this only works if the feedforward connectivity is unconstrained. In the case of Babadi and Sompolinsky, there is an expansion of inputs into a higher dimensional space through random connectivity drawn from a centered Gaussian distribution and appropriately chosen readout weights. This scenario is not compatible with the well-established biological constraints mentioned above that our model takes into account. In the new section ”Critical modularity for denoising” of the revised manuscript we show that recurrent inhibition is necessary to enable signal transmission and denoising under these constraints. The inhibition thereby not only generates competition between input channels but it also allows the modules to track their input very rapidly (as originally demonstrated by van Vreeswijk and Sompolinsky in 1996). To demonstrate this point and emphasize the relevance of dynamics, we added a new signal reconstruction task in the new section ”Reconstruction and denoising of dynamic inputs”, where we show that our model can faithfully track and denoise spatially encoded time-varying inputs.
The authors perform a mean-field study to explain how modularity affects signal propagation. At the heart of their argument is that the E/I network exhibit bistability. However, bistability can be achieved by an excitatory population with a threshold [Renart et al., 2013]. The role of the inhibitory population does not seem crucial for the task and questions the motivations for this analysis.
We thank the reviewer for raising this important point which we address in the section ”Critical modularity for denoising” of the revised manuscript. The reviewer is correct that bistability can be obtained in a purely excitatory network, and the modular topographic connectivity in our work essentially renders the stimulated pathway excitatory. The important feature of our model, however, is that the non-stimulated pathways remain inhibitory to get a distinction between stimulated and non-stimulated populations and the denoising feature. This is only achieved by recurrent inhibition that causes competition between pathways. Our analyses show that, for networks without recurrent connections or even excitatory recurrent connections, the network lacks mechanisms to compensate the excitatory feedforward and external background input. In these cases, all populations show high (and synchronous) activity and no classification and denoising can be achieved. Therefore, the revised manuscript unambiguously demonstrates the critical role of recurrent inhibition.
Active and inactive channels are decided by the two stable states of the network: the high and the low activity regimes. However, noise fluctuations and their propagation through the network may have a prominent role in the overall dynamics. I find that noise fluctuation analysis is bluntly missing in this work.
Fig. 7b of the previous version showed the stability of theoretically predicted fixed points using numerical fluctuation analysis around the fixed points. We apologize for not having made this sufficiently clear, and have therefore updated the caption of Fig. 7 to emphasize this point and extended the subsection ”Fixed point analysis” of the Methods detailing our approach. Furthermore, we fully agree with the reviewer that fluctuation analyses are important to understand the dynamics of our system. Therefore, we performed a theoretical fluctuation analysis in the new Figure 8 and the extended Appendix B of the revised version. This extended theory shows that competition induced by recurrent inhibition stabilizes the low activity state of non-stimulated sub-populations such that fluctuations cannot build up and propagate across layers, in line with the previously presented numerical simulation results.
The main finding is a critical level of modularity, m= 0.83, above which the network shows denoising properties of silencing inactive channels and increasing the mean activity of active ones. However, the critical modularity is numerically demonstrated and is not derived theoretically. For a theoretical insight into this transition between denoising and mixing properties of the network, I would have liked to see a more rigorous discussion on the critical value. What does the critical point depend on? The authors show that the single-neuron dynamics do not affect the critical value, but what about other structural elements such as the relative efficacies of the E/I and the feedforward connectivity matrices? Do the authors suggest that m=0.83 is a universal number? I expect a more detailed analysis and discussion of this core issue in a theoretical paper.
We fully agree with the reviewer and are grateful that this point was brought up. The initial submission did not provide a sufficent or deep enough discussion on which features determine the critical modularity and it certainly is important to do so. We also apologize that our presentation was misleading and suggested a universal number for the critical modularity. Unfortunately, there is no closed form expression for the critical modularity for the non-linear activation functions shown in the previous version. We therefore added a new analysis with a fully tractable piecewise linear activation function that allows us to derive a closed-form solution for the critical modularity. The new section ”Critical modularity for denoising” and Appendix B show the results of this analysis and discuss the various parameters that affect the value of the critical modularity. In short, the reviewer was completely right that the critical modularity depends on a number of connectivity parameters as well as single-neuron properties. In particular, our theoretical results show that recurrent inhibition is crucial for denoising.
To conclude my main criticism, I believe that a theoretical paper should offer a more in-depth analysis and discussion of the core ideas presented and not rely mainly on simulations. For example, to provide theoretical insight, the authors should address central questions such as the origin of the critical modularity, the role of the recurrent balance connectivity, and how the network can facilitate computations other than winner-takes-all among channels. Alternatively, if the authors aim to describe a neural dynamics model without deep theoretical insights, I would expect to see physiological evidence supporting the suggested dynamics.
We are very grateful for the reviewer’s criticism and believe the manuscript has substantially improved as a consequence. We are confident that our revised manuscript, by addressing these issues and extending the theoretical insights, now provides a much more thorough and comprehensive understanding.
Conclusions
The model studied by the authors is novel and provides a valuable way of exploring the effects of modularity and topographic connectivity on signal propagation through hierarchical recurrent neural networks. However, the study lacks theoretical insights into cortical circuit functions in its current version. I believe that for this work to impact the field, it needs to show further analysis and not rely on a numerical study of the model with limited theoretical derivations.
Reviewer #2 (Public Review):
This manuscript puts forward a new idea that topography in neural networks helps to remove noise from inputs. The neural network consists of multiple stages. At each stage, the network is structured to be balanced in terms of the strength of inhibitory and excitatory signals. Because of topography, the networks become ”dis-balanced” and receive more recurrent excitatory signals locally for those regions that receive strong initial inputs. This leads to error correction. The main weakness in the manuscript is that the approach will only work for inputs that are constant-in-time. It is important to acknowledge this limitation in both the title and throughout the manuscript.
We thank the reviewer for the concise summary of our work and for acknowledging its novelty. Given the importance of the issue raised by the reviewer regarding the nature of the input signals, in the revised manuscript we added a new section ”Reconstruction and denoising of dynamic inputs” in which we investigate more complex, time-varying inputs and demonstrate that the model, due to the balance between excitation and inhibition, is able to quickly follow, process and denoise the external inputs. There are of course limits to the signal frequencies which can be successfully denoised, which we discuss in the Supplementary Materials (see Figure 10 - supplement 1) and elaborate on in the Discussion, but these are roughly within the ranges found in Human psychophysics studies.
-
eLife assessment
This manuscript puts forward a new idea that topography in neural networks helps to remove noise from inputs. The authors show that there is a critical level of topography that is needed for network to denoise inputs. At present, the analysis is limited to inputs that are constant in time.
-
Reviewer #1 (Public Review):
Overview:
In this work, the authors set to study the effects of topographic connectivity in a hierarchical model of neural networks. They hypothesize that the topographic connectivity, often observed in cortical networks, is essential for signal propagation and allows faithful transmission of signals.
To study the effects of topographic connectivity on the dynamics, the authors consider a network composed of several layers. Each layer is a recurrent neural network with excitatory and inhibitory subpopulations. The excitatory neurons in each layer enervate a subpopulation of the following layer. The receiving excitatory subpopulation targets a specific group in the next layer and so on. This procedure leads to separate channels that carry the inputs through the network. The authors study how the degree of …
Reviewer #1 (Public Review):
Overview:
In this work, the authors set to study the effects of topographic connectivity in a hierarchical model of neural networks. They hypothesize that the topographic connectivity, often observed in cortical networks, is essential for signal propagation and allows faithful transmission of signals.
To study the effects of topographic connectivity on the dynamics, the authors consider a network composed of several layers. Each layer is a recurrent neural network with excitatory and inhibitory subpopulations. The excitatory neurons in each layer enervate a subpopulation of the following layer. The receiving excitatory subpopulation targets a specific group in the next layer and so on. This procedure leads to separate channels that carry the inputs through the network. The authors study how the degree of specificity in each targeted projection, called 'modularity,' affects signal propagation through the network.
The authors find that the network reduces noise above a critical level of network modularity: the deep layers show a clear separation of an active channel and inactive channels, despite the noisy input signal. They study how different dynamical and structural properties affect the signal propagation through the network layers and suggest that the dynamics can implement a winner-takes-all computation.
Strengths and novelty:
- Topographic projections, in which subpopulations of neurons target specific cells in efferent populations, are common in the central nervous system. The dynamic and computation benefits of this organization are not fully understood. With their simple model, the authors were able to quantify the amount of topographic structure and selectivity in the network and study its impact on the network's steady-state. In particular, a bifurcation point suggests a qualitative difference between networks with and without sufficient topographic modularity.
- The theoretical analysis in the paper is rigorous, and the mean-field study shows good agreement with computer simulations of the model.
- The authors describe simulation results of networks with different dynamical properties, including rate-based networks, integrate-and-fire neurons, and more realistic conductance-based spiking neurons. All simulations exhibit similar qualitative behavior, supporting the conclusion that the behavior due to structural modularity will carry to more complex and biologically relevant neural dynamics.
- Overall, the authors convince that the topographic structure of the network can lead to noise reduction, given that the input to the network is provided as distinct channels.Weaknesses:
The authors support their hypothesis and show a relation between topographic connection and noise reduction in their model. However, I find the study limited and struggle to see the impact it will have on the field. The paper is purely theoretical; it does not provide any physiological evidence that supports the conclusion. On the other hand, and this is the key issue, I do not find real theoretical insights in this work. In the following, I elaborate on why I hold this opinion.
- The hypothesis is that topographic projections in cortical areas allow faithful signal propagation. However, as the authors point out, reliable transmission can be achieved in other ways, such as by direct routing of information (lines 17-19). Furthermore, denoising can be accomplished by a simple feedforward network (e.g., ref 38) without E/I balance and with plasticity rules that do not require topographic connectivity. Thus, I find the computational model not well motivated.
- The task studied here is a simple classification of static inputs: the efferent readout needs to identify the active channel. Again, this could be achieved by a single layer of simple binary neurons [Babadi and Sompolinsky 2014]. The recurrent connectivity and E/I balance suggest that dynamics should play an essential part in the model. However, the task is not well suited for understanding the role of dynamics.
- The authors perform a mean-field study to explain how modularity affects signal propagation. At the heart of their argument is that the E/I network exhibit bistability. However, bistability can be achieved by an excitatory population with a threshold [Renart et al., 2013]. The role of the inhibitory population does not seem crucial for the task and questions the motivations for this analysis.
- Active and inactive channels are decided by the two stable states of the network: the high and the low activity regimes. However, noise fluctuations and their propagation through the network may have a prominent role in the overall dynamics. I find that noise fluctuation analysis is bluntly missing in this work.
- The main finding is a critical level of modularity, m=~0.83, above which the network shows denoising properties of silencing inactive channels and increasing the mean activity of active ones. However, the critical modularity is numerically demonstrated and is not derived theoretically. For a theoretical insight into this transition between denoising and mixing properties of the network, I would have liked to see a more rigorous discussion on the critical value. What does the critical point depend on? The authors show that the single-neuron dynamics do not affect the critical value, but what about other structural elements such as the relative efficacies of the E/I and the feedforward connectivity matrices? Do the authors suggest that m=0.83 is a universal number? I expect a more detailed analysis and discussion of this core issue in a theoretical paper.To conclude my main criticism, I believe that a theoretical paper should offer a more in-depth analysis and discussion of the core ideas presented and not rely mainly on simulations. For example, to provide theoretical insight, the authors should address central questions such as the origin of the critical modularity, the role of the recurrent balance connectivity, and how the network can facilitate computations other than winner-takes-all among channels. Alternatively, if the authors aim to describe a neural dynamics model without deep theoretical insights, I would expect to see physiological evidence supporting the suggested dynamics.
Conclusions:
The model studied by the authors is novel and provides a valuable way of exploring the effects of modularity and topographic connectivity on signal propagation through hierarchical recurrent neural networks. However, the study lacks theoretical insights into cortical circuit functions in its current version. I believe that for this work to impact the field, it needs to show further analysis and not rely on a numerical study of the model with limited theoretical derivations.
-
Reviewer #2 (Public Review):
This manuscript puts forward a new idea that topography in neural networks helps to remove noise from inputs. The neural network consists of multiple stages. At each stage, the network is structured to be balanced in terms of the strength of inhibitory and excitatory signals. Because of topography, the networks become "dis-balanced" and receive more recurrent excitatory signals locally for those regions that receive strong initial inputs. This leads to error correction. The main weakness in the manuscript is that the approach will only work for inputs that are constant-in-time. It is important to acknowledge this limitation in both the title and throughout the manuscript.
-
-