A hardware system for real-time decoding of in vivo calcium imaging data

  1. Zhe Chen
  2. Garrett J Blair
  3. Changliang Guo
  4. Jim Zhou
  5. Juan-Luis Romero-Sosa
  6. Alicia Izquierdo
  7. Peyman Golshani
  8. Jason Cong
  9. Daniel Aharoni
  10. Hugh T Blair

  • Curated by eLife

    eLife logo

    eLife assessment

    This study demonstrates ultrafast real-time decoding of place fields in the hippocampus thanks to a head-mounted microscope for calcium imaging and to a novel data processing pipeline. This is a useful tool that aims at obtaining real-time capabilities that will enable closed-loop experiments that include decoding of a wide neuronal population, which could be applied in a variety of neuroscience fields. This will be of interest to anyone studying behaviors or functions that involve the hippocampus.

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

Epifluorescence miniature microscopes (‘miniscopes’) are widely used for in vivo calcium imaging of neural population activity. Imaging data are typically collected during a behavioral task and stored for later offline analysis, but emerging techniques for online imaging can support novel closed-loop experiments in which neural population activity is decoded in real time to trigger neurostimulation or sensory feedback. To achieve short feedback latencies, online imaging systems must be optimally designed to maximize computational speed and efficiency while minimizing errors in population decoding. Here we introduce DeCalciOn , an open-source device for real-time imaging and population decoding of in vivo calcium signals that is hardware compatible with all miniscopes that use the UCLA Data Acquisition (DAQ) interface. DeCalciOn performs online motion stabilization, neural enhancement, calcium trace extraction, and decoding of up to 1024 traces per frame at latencies of <50 ms after fluorescence photons arrive at the miniscope image sensor. We show that DeCalciOn can accurately decode the position of rats ( n = 12) running on a linear track from calcium fluorescence in the hippocampal CA1 layer, and can categorically classify behaviors performed by rats ( n = 2) during an instrumental task from calcium fluorescence in orbitofrontal cortex. DeCalciOn achieves high decoding accuracy at short latencies using innovations such as field-programmable gate array hardware for real-time image processing and contour-free methods to efficiently extract calcium traces from sensor images. In summary, our system offers an affordable plug-and-play solution for real-time calcium imaging experiments in behaving animals.

Article activity feed

  1. Version published to 10.7554/elife.78344 on eLife
  2. eLife

    Author Response

    Reviewer #1 (Public Review)

    This manuscript describes a new method to perform online movement correction and extraction of calcium signals from a miniscope. The efficiency of the algorithm is tested by quantifying the accuracy of animal location decoding from hippocampal place cells. The online decoding happens with virtually no delay which is promising for closed-loop methods. It seems to be superior to online decoding without motion correction, which was the state of the art.

    The strength of this technique is therefore that it achieves real-time processing.

    The weakness of the study is the lack of comparison of the decoding accuracy with what can be obtained with electrophysiological state of the art, which prevents really estimating how precise the technique is.

    In revision, we present data showing that when our system is used to decode contour-based calcium traces from N≈50 neurons, the decoder achieves a mean distance error of ~30 cm which is worse than the mean error of ~20 cm achieved using maximum likelihood decoding of single unit spike trains from electrophysiological recordings (Fig. 7E). However, when decoding of N=900 contour-free calcium traces from the same image frames in the same rats, the mean decoding error goes down to ~15 cm, which is better than the mean for electrophysiological recordings. From this we conclude that real-time decoding of position from calcium traces achieves accuracies similar to those achievable with electrophysiology.

    Although less critical, there is no demonstration of a closed-loop application.

    It is true that we have not yet demonstrated a real-time closed loop application, but by demonstrating short latency generation of TTL outputs triggered by the decoder, we demonstrate the capability for closed-loop applications.

    Real-time position decoding is technically nice, but the position can be obtained from tracking the animal so it is practically useless.

    We offer two points in reply to this comment. First, decoding position from neural activity could offer useful (though not yet demonstrated) capabilities that would not be achievable with simple position tracking; for example, the position decoder could be trained on CA1 signals obtained during waking and then used to read out position trajectories generating during REM sleep.

    Second, and more importantly, position decoding was selected as a benchmark for performance testing mainly because it allows highly precise comparisons between decoder predictions and ground truth, which is important for establishing that the fidelity of calcium signals imaged in real time is adequate for accurate decoding of behavior at short latencies.

    It is also clear that decoding position on a linear track is easier than on a 2D arena, therefore it is difficult to estimate how much the efficiency of the method can be challenged in harder settings.

    It is true that decoding in a 2D arena would be a greater challenge than a 1D linear track, but in pursuit of our goal to rapidly disseminate a system with capabilities for short latency decoding of behavior from calcium signals, optimizing system performance for one specific application (e.g,, position decoding) is not our main priority. A higher priority is to offer versatility for a wide range of experimental applications. To better demonstrate such versatility, the revised manuscript includes a new section in the Results that demonstrates categorical classification of behaviors during an instrumental touchscreen task.

    Reviewer #2 (Public Review):

    In this paper, the authors developed a new device for online decoding of position based on calcium imaging in freely moving rodents. This device could be used in the brain-computer interface to investigate neurofeedback-based therapies for neurological disorders. The technical part is properly done and gives convincing results that can be truly helpful for the scientific community using the miniscope. Nevertheless, as a methodological article, there should be more details regarding the accuracy of the decoding and of the different steps to follow if someone wants to use their methodology. Moreover, a true online real-time experiment should be performed to validate the device.

    Please find below my comments:

    • From what I read the authors did not perform a true real-time experiment. I think this step iscrucial to ensure the quality of their device.

    It is unclear from this comment where to draw the bar for a “true real-time experiment.” Some previous publications of real-time approaches (such as refs #6,#11,#26) have proposed causal algorithms without performance tests in hardware at all, whereas others (such as ref #14) have performance tested their system in hardware by carrying full experiments using closed-loop feedback (albeit with much smaller numbers of calcium trace predictors than we demonstrate here) without comparing different algorithmic approaches. Here we use an intermediate strategy of feeding raw offline video from a virtual sensor through the hardware processing pipeline (verifying that calcium trace outputs were identical for the real and virtual sensors). We adopted this intermediate approach to achieve the dual objectives of testing a true hardware implementation on real-time performance measures (e.g., microsecond processing latencies) while also benchmarking different algorithms (such as CB versus CF trace extraction as in Fig. 3, or raw calcium traces versus deconvolved spikes as in panel A of the Supplement to Fig. 3) against one another on the same datasets.

    • There should be a validation against a classical offline Bayesian decoding.

    We have presented an accuracy comparison for decoding linear track position from calcium traces with DeCalciOn versus decoding from single-unit spikes with electrophysiological recording data (Fig. 7E); decoding from single-unit spikes utilized a classical Bayesian maximum likelihood approach (see Methods), so Fig. 7E not only offers a comparison between calcium imaging versus electrophysiology, but between online linear classifier versus classical offline Bayesian approaches as well. In addition, we compared the performance of the linear classifier to a naïve Bayes decoder in panel B of the Supplement to Fig 3, showing that performance is better for the linear classifier than naïve Bayes.

    • "To mimic these steps using the virtual sensor in our performance tests, one session of imagedata was collected and stored from each of the 13 rats, yielding ~7 min (8K-9K frames) of sensor and position tracking data per rat. The linear classifier was then trained on data from the first half of each session and tested on data from the second half." This sentence is not clear enough. The authors should clearly describe the exact time needed for each experimental step. What is the time needed for instance for the experimental step 2, during which the linear classifier is trained to decode behavior from the initial dataset? This is crucial information if someone wants to use this device.

    In response to this comment, the Results section of the revised manuscript includes an extensive subsection (‘Steps of a real-time imaging session’) that describes each experimental step in detail (pages 4-6), including the time required for each step. In addition, this information is now more thoroughly summarized in the diagram of Fig. 1B.

    How the accuracy varies with the duration (or the quality) of the initial dataset? It is important that the authors provide an investigation of this to validate their device.

    This issue is now discussed in the Results near the bottom of page 5. In addition, Fig. 3G now plots how position decoding improves as a function of the size of the training dataset.

    • For instance, what is the decrease in decoding accuracy 1) with fewer place cells?

    The scatterplots in the right panels of Fig. 3D show that decoding accuracy improves as a function of the number of neurons imaged in given rat.

    What is the approximative number of place cells to obtain reliable decoding?

    This question is addressed by showing how decoding accuracy improves with the number of imaged neurons (Fig. 3D scatterplots). We also address this issue on our performance comparison of CB versus CF and CF+ traces since differing numbers of calcium trace predictors appear to be an important factor in accounting for the observed performance differences, as discussed in the main text (page 16, last paragraph).

    1. With the duration of the initial recording session. Here it seems to be of the order of 3-4 min.What if the recording session is shorter? Is there some constraint about this recording session (in terms of speed, stops, etc...) to obtain good decoding?

    The revised Fig. 3G plots how position decoding improves as a function of the size of the training dataset.

    1. Is there a link between the decoding accuracy and the number of place cells nearby?

    We did not select calcium traces that met a spatial criterion (i.e, “place cells”) to be include in the decoding analysis, Instead, all detected CA1 calcium traces provided input to the decoder, regardless of their spatial tuning properties (Fig. 3D and panels D,E of the Supplement to Fig. 3 show that many cells were indeed spatially tuned). Also note that when contour-free (CF) trace extraction methods were used, each calcium trace could detect fluorescence from multiple neurons. Under this methodology it is not straightforward to analyze how decoding accuracy at a given position varies with the “number of place cells nearby” and we are not convinced that presenting such an analysis would advance our main goal of demonstrating DeCalciOn’s capabilities to researchers.

    • The authors specified the time delay of 2.5ms for their device. Yet, it is pointless regarding thepurpose of the decoding. The important information is the precise position of the animal when the device is used to trigger a stimulation at a given location. Again, a true online experiment should be done to validate that a TTL can be triggered by the device at a precise location (with a quantification of the error made).

    We agree that this is an important issue, and it has been thoroughly addressed in the revised manuscript.

    • There is no information on the accuracy of the decoding with respect to the location in thelinear track. It is likely that the extremities of the linear track will be better identified. Figure 4C does not provide a clear description of the error made. The choice of D=2 (which seems to represent the spatial bin) is not justified. Two spatial bins seem to represent +/-40 cm which is quite large.

    Polar plots in Fig. 3F of the revised manuscript show mean accuracy in each position bin for decoders trained on offline, CB, CF,. and CB+ calcium traces.

    • The movement artefacts are not equally observed in the maze. The way they are correctedmight be captured by the linear decoder. These artefacts might have a strong influence on the decoding. Please provide a quantification of the correction made during steps 1 and 2 in relation to the position of the animal on the linear track. The authors should provide a correlation between the presence of these corrections with the decoding accuracy.

    Regardless of whether analysis is done offline or online, any calcium imaging and decoding experiment is vulnerable to two potential problems arising from motion artifact:

    PROBLEM #1. Image motion can generate noise in calcium signals that disrupts the accuracy of decoding.

    PROBLEM #2. Image motion that is correlated with behavior can convey uncontrolled information that allows the decoder to learn predictions from image motion rather than calcium signals. Very few published in-vivo calcium imaging experiments provide adequate controls for these two possible sources of artifact (again, such controls are just as necessary for offline as for online experiments). In response to the referee comments, we have provided controls for these confounds in our performance tests of DeCalciOn’s online decoding capabilities.

    Fig. 4B of the revised paper shows that without online motion correction, several rats in the linear track experiment show a significant correlation between position error and motion artifact (indicated by positive values on the y-axis); hence, motion artifact impairs decoding of position on the linear track in these rats (problem #1 above). This correlation between motion artifact and decoding error is reduced or eliminated by online motion correction (as indicated by values near zero on the x-axis), demonstrating that online motion correction helps to prevent motion artifact from impairing the accuracy of decoding.

    Fig. 6 of the revised paper shows that during an operant touchscreen experiment, motion artifact occurs preferentially during specific behaviors such as visiting the food magazine (reward retrieval, Fig. 6A) or touching the screen to make a response (correct choice, Fig. 6B). When motion correction is not used (top graphs in Figs. 6C-F), the average motion artifact is higher during frames when the decoder accurately predicts behavior than during frames when the decoder fails to predict behavior; hence, motion artifact appears to improve the accuracy of predicting these behaviors (problem #2 above). When motion correction is used, the average motion artifact no longer differs for correctly versus incorrectly decoded frames (except in one case, bottom right graph of Fig. 6E), indicating that motion correction helps to prevent the decoder from learning to predict behavior from motion artifact.

    • Besides the methodological part, I have some physiological questions. It is quite common inlinear tracks to have bi-directional and unidirectional place cells. Is it the case here? How many? It is difficult to see this in figure C. Is there an error due to the online decoding of the position in the two directions of the linear track?

    Again, since we did not select calcium traces that met a spatial criterion (i.e, “place cells”) to be include in the decoding analysis, and since CF traces could detect fluorescence from multiple neurons, we are not convinced that presenting a detailed analysis of this issue would advance our primary goal of demonstrating DeCalciOn’s capabilities to reseachers.

    Reviewer #3 (Public Review):

    DeCalciOn is an innovative contribution to the toolbox of real-time processing of calcium imaging data. It provides calcium traces from hippocampal CA1 neurons with a roughly two-millisecond latency and uses them to decode the position of rats running along a linear track - setting the stage for closed-loop experiments requiring fast interpretation of neural activity. The manuscript would be strengthened by a more systematic, empirical comparison to other, currently available alternative approaches. In addition, the decoding analysis does not fully account for the possibility of artifactual motion in the imaging video being informative of position.

    We suggest strengthening this manuscript by addressing the following four points:

    1. In the discussion of other platforms, the authors state that "Any system that lacks motionstabilization would also be vulnerable to artifactually decoding behavior from brain motion (which can be correlated with behavior) rather than neural activity." It follows that the same problem might also occur with incomplete motion correction. While the motion-corrected video shown in Supplementary Video 1 has reduced motion compared to the raw video, motion is still visible, including outside of the marked jitter. It remains possible that the linear decoders for the position in the linear track are utilizing brain motion-induced, as opposed to calcium fluorescence-induced, signal changes. A critical first step to assess this issue is to ask whether the motion in the video is related to the rat's behavior. One could test whether the 2D motion displacement traces can be used to predict rat position using linear classifiers.

    Briefly, we show that motion correction helps to prevent the decoder from learning to predict behavior from motion artifact.

    1. The manuscript would benefit from repeating the experiment in a more complex environment,such as a 2D arena. This would increase the generalizability of the findings. In addition, increasing the complexity of the environment would reduce the possibility that particular types of brain motion are closely linked with positions in the environment.

    We have diversified our performance testing by presenting results for decoding calcium activity from a different brain region (OFC rather than CA1) during a different kind of behavior (an instrumental touchscreen task rather than a linear track).

    1. The authors present an interesting comparison between "contour-free" and traditionalcontour-based source extraction. A more comprehensive discussion on the history or novelty of "contour-free" calcium imaging processing would contextualize this result.

    The revised Discussion section contains a new subsection titled “Source identification” to contextualize this issue.

    1. In the discussion, the authors compare DeCalciOn to two previous online calcium imagingalgorithms. The technical innovations of this work would be better highlighted by directly testing all three of these algorithms, ideally on similar datasets.

    Briefly, one of the two cited systems is designed for compatibility with benchtop 2P microscopes and does not interface with miniscopes; public resources are not available for the other cited online algorithm.

  3. eLife

    eLife assessment

    This study demonstrates ultrafast real-time decoding of place fields in the hippocampus thanks to a head-mounted microscope for calcium imaging and to a novel data processing pipeline. This is a useful tool that aims at obtaining real-time capabilities that will enable closed-loop experiments that include decoding of a wide neuronal population, which could be applied in a variety of neuroscience fields. This will be of interest to anyone studying behaviors or functions that involve the hippocampus.

  4. eLife

    Reviewer #1 (Public Review):

    This manuscript describes a new method to perform online movement correction and extraction of calcium signals from a miniscope. The efficiency of the algorithm is tested by quantifying the accuracy of animal location decoding from hippocampal place cells. The online decoding happens with virtually no delay which is promising for closed-loop methods. It seems to be superior to online decoding without motion correction, which was the state of the art.

    The strength of this technique is therefore that it achieves real-time processing.
    The weakness of the study is the lack of comparison of the decoding accuracy with what can be obtained with electrophysiological state of the art, which prevents really estimating how precise the technique is.

    Although less critical, there is no demonstration of a closed-loop application. Real-time position decoding is technically nice, but the position can be obtained from tracking the animal so it is practically useless. It is also clear that decoding position on a linear track is easier than on a 2D arena, therefore it is difficult to estimate how much the efficiency of the method can be challenged in harder settings.

    Thus despite its technical excellence, the impact of this method seems weak.

  5. eLife

    Reviewer #2 (Public Review):

    In this paper, the authors developed a new device for online decoding of position based on calcium imaging in freely moving rodents. This device could be used in the brain-computer interface to investigate neurofeedback-based therapies for neurological disorders. The technical part is properly done and gives convincing results that can be truly helpful for the scientific community using the miniscope. Nevertheless, as a methodological article, there should be more details regarding the accuracy of the decoding and of the different steps to follow if someone wants to use their methodology. Moreover, a true online real-time experiment should be performed to validate the device.

    Please find below my comments:

    - From what I read the authors did not perform a true real-time experiment. I think this step is crucial to ensure the quality of their device.

    - There should be a validation against a classical offline Bayesian decoding.

    - "To mimic these steps using the virtual sensor in our performance tests, one session of image data was collected and stored from each of the 13 rats, yielding ~7 min (8K-9K frames) of sensor and position tracking data per rat. The linear classifier was then trained on data from the first half of each session and tested on data from the second half." This sentence is not clear enough. The authors should clearly describe the exact time needed for each experimental step. What is the time needed for instance for the experimental step 2, during which the linear classifier is trained to decode behavior from the initial dataset? This is crucial information if someone wants to use this device. How the accuracy varies with the duration (or the quality) of the initial dataset? It is important that the authors provide an investigation of this to validate their device.

    - For instance, what is the decrease in decoding accuracy 1) with fewer place cells? What is the approximative number of place cells to obtain reliable decoding? 2) with the duration of the initial recording session. Here it seems to be of the order of 3-4 min. What if the recording session is shorter? Is there some constraint about this recording session (in terms of speed, stops, etc...) to obtain good decoding? 3) Is there a link between the decoding accuracy and the number of place cells nearby?

    - The authors specified the time delay of 2.5ms for their device. Yet, it is pointless regarding the purpose of the decoding. The important information is the precise position of the animal when the device is used to trigger a stimulation at a given location. Again, a true online experiment should be done to validate that a TTL can be triggered by the device at a precise location (with a quantification of the error made).

    - There is no information on the accuracy of the decoding with respect to the location in the linear track. It is likely that the extremities of the linear track will be better identified. Figure 4C does not provide a clear description of the error made. The choice of D=2 (which seems to represent the spatial bin) is not justified. Two spatial bins seem to represent +/-40 cm which is quite large.

    - The movement artefacts are not equally observed in the maze. The way they are corrected might be captured by the linear decoder. These artefacts might have a strong influence on the decoding. Please provide a quantification of the correction made during steps 1 and 2 in relation to the position of the animal on the linear track. The authors should provide a correlation between the presence of these corrections with the decoding accuracy.

    - Besides the methodological part, I have some physiological questions. It is quite common in linear tracks to have bi-directional and unidirectional place cells. Is it the case here? How many? It is difficult to see this in figure C. Is there an error due to the online decoding of the position in the two directions of the linear track?

  6. eLife

    Reviewer #3 (Public Review):

    DeCalciOn is an innovative contribution to the toolbox of real-time processing of calcium imaging data. It provides calcium traces from hippocampal CA1 neurons with a roughly two-millisecond latency and uses them to decode the position of rats running along a linear track - setting the stage for closed-loop experiments requiring fast interpretation of neural activity. The manuscript would be strengthened by a more systematic, empirical comparison to other, currently available alternative approaches. In addition, the decoding analysis does not fully account for the possibility of artifactual motion in the imaging video being informative of position.

    We suggest strengthening this manuscript by addressing the following four points:

    1. In the discussion of other platforms, the authors state that "Any system that lacks motion stabilization would also be vulnerable to artifactually decoding behavior from brain motion (which can be correlated with behavior) rather than neural activity." It follows that the same problem might also occur with incomplete motion correction. While the motion-corrected video shown in Supplementary Video 1 has reduced motion compared to the raw video, motion is still visible, including outside of the marked jitter. It remains possible that the linear decoders for the position in the linear track are utilizing brain motion-induced, as opposed to calcium fluorescence-induced, signal changes. A critical first step to assess this issue is to ask whether the motion in the video is related to the rat's behavior. One could test whether the 2D motion displacement traces can be used to predict rat position using linear classifiers.

    2. The manuscript would benefit from repeating the experiment in a more complex environment, such as a 2D arena. This would increase the generalizability of the findings. In addition, increasing the complexity of the environment would reduce the possibility that particular types of brain motion are closely linked with positions in the environment.

    3. The authors present an interesting comparison between "contour-free" and traditional contour-based source extraction. A more comprehensive discussion on the history or novelty of "contour-free" calcium imaging processing would contextualize this result.

    4. In the discussion, the authors compare DeCalciOn to two previous online calcium imaging algorithms. The technical innovations of this work would be better highlighted by directly testing all three of these algorithms, ideally on similar datasets.

  7. Version published to 10.1101/2022.01.31.478424v4 on bioRxiv
  8. Version published to 10.1101/2022.01.31.478424v3 on bioRxiv
  9. Version published to 10.1101/2022.01.31.478424v2 on bioRxiv
  10. Version published to 10.1101/2022.01.31.478424v1 on bioRxiv