BonVision – an open-source software to create and control visual environments
Curation statements for this article:-
Curated by eLife
Evaluation Summary:
Increasingly, neuroscience experiments require immersive virtual environments that approximate natural sensory motor loops while permitting high-bandwidth measurements of brain activity. BonVision is an open-source graphics programming library that allows experimenters to quickly implement immersive 3D visual environments across display hardware and geometry with automated calibration and integration with hundreds of different neural recording technologies, behavioral apparatuses, etc. BonVision standardizes sharing complex, closed-loop visual tasks between labs with vastly different equipment, provides a concrete and easy way to do so, and should be of interest to a wide array of visual systems neuroscientists.
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (eLife)
- Neuroscience (eLife)
Abstract
Real-time rendering of closed-loop visual environments is necessary for next-generation understanding of brain function and behaviour, but is prohibitively difficult for non-experts to implement and is limited to few laboratories worldwide. We developed BonVision as an easy-to-use open-source software for the display of virtual or augmented reality, as well as standard visual stimuli. As the architecture is based on the open-source Bonsai graphical programming language, BonVision benefits from native integration with experimental hardware. BonVision therefore enables easy implementation of closed-loop experiments, including real-time interaction with deep neural networks and communication with behavioural and physiological measurement and manipulation devices.
Article activity feed
-
Author Response:
Reviewer #1 (Public Review):
In this project, the authors set out to create an easy to use piece of software with the following properties: The software should be capable of creating immersive (closed loop) virtual environments across display hardware and display geometries. The software should permit easy distribution of formal experiment descriptions with minimal changes required to adapt a particular experimental workflow to the hardware present in any given lab while maintaining world-coordinates and physical properties (e.g. luminance levels and refresh rates) of visual stimuli. The software should provide equal or superior performance for generating complex visual cues and/or immersive visual environments in comparison with existing options. The software should be automatically integrated with many other …
Author Response:
Reviewer #1 (Public Review):
In this project, the authors set out to create an easy to use piece of software with the following properties: The software should be capable of creating immersive (closed loop) virtual environments across display hardware and display geometries. The software should permit easy distribution of formal experiment descriptions with minimal changes required to adapt a particular experimental workflow to the hardware present in any given lab while maintaining world-coordinates and physical properties (e.g. luminance levels and refresh rates) of visual stimuli. The software should provide equal or superior performance for generating complex visual cues and/or immersive visual environments in comparison with existing options. The software should be automatically integrated with many other potential data streams produced by 2-photon imaging, electrophysiology, behavioral measurements, markerless pose estimation processing, behavioral sensors, etc.
To accomplish these goals, the authors created two major software libraries. The first is a package for the Bonsai visual programming language called "Bonsai.Shaders" that brings traditionally low-level, imperative OpenGL programming into Bonsai's reactive framework. This library allows shader programs running on the GPU to seamlessly interact, using drag and drop visual programming, with the multitude of other processing and IO elements already present in numerous Bonsai packages. The creation of this library alone is quite a feat given the complexities of mapping the procedural, imperative, and stateful design of OpenGL libraries to Bonsai's event driven, reactive architecture. However, this library is not mentioned in the manuscript despite its power for tasks far beyond the creation of visual stimuli (e.g. GPU-based coprocessing) and, unlike BonVision itself, is largely undocumented. I don't think that this library should take center stage in this manuscript, but I do think its use in the creation of BonVision as well as some documentation on its operators would be very useful for understanding BonVision itself.
We have added a reference to the Shaders package at multiple points in the manuscript including lines 58-59 and in Supplementary Details. We will be adding documentation of key Shaders nodes that are important for the creation of BonVision stimuli to the documentation on the BonVision website.
Following the creation of Bonsai.Shaders, the authors used it to create BonVision which is an abstraction on top of the Shaders library that allows plug and play creation of visual stimuli and immersive visual environments that react to input from the outside world. Impressively, this library was implemented almost entirely using the Bonsai visual programming language itself, showcasing its power as a domain-specific language. However, this fact was not mentioned in the manuscript and I feel it is a worthwhile point to make.
Thank you - we have now added clarification on this in Supplementary details (section Customised nodes and new stimuli)
The design of BonVision, combined with the functional nature of Bonsai, enforces hard boundaries between the experimental design of visual stimuli and (1) the behavioral input hardware used to drive them, (2) the dimensionality of the stimuli (i.e. 2D textures via 3D objects), (3) the specific geometry of 3D displays (e.g. dual monitors, versus spherical projection, versus head mounted stereo vision hardware), and (4) automated hardware calibration routines. Because of these boundaries, experiments designed using BonVision become easy to share across labs even if they have very different experimental setups. Since Bonsai has integrated and standardized mechanisms for sharing entire workflows (via copy paste of XML descriptions or upload of workflows to publicly accessible Nuget package servers), this feature is immediately usable by labs in the real world.
After creating these pieces of software, the authors benchmarked them against other widely used alternatives. IonVisoin met or exceeded frame rate and rendering latency performance measures when compared to other single purpose libraries. BonVision is able to do this while maintaining its generality by taking advantage of advanced JIT compilation features provided by the .NET runtime and using bindings to low-level graphics libraries that were written with performance in mind. The authors go on to show the real-world utility of BonVision's performance by mapping the visual receptive fields of LFP in mouse superior colliculus and spiking in V1. The fact that they were able to obtain receptive fields indicates that visual stimuli had sufficient temporal precision. However, I do not follow the logic as to why this is because the receptive fields seem to have been created using post-hoc aligned stimulus-ephys data, that was created by measuring the physical onset times of each frame using a photodiode (line 389). Wouldn't this preclude any need for accurate stimulus timing presentation?
We thank the reviewer for this suggestion. We now include receptive field maps calculated using the BonVision timing log in Figure5 – figure supplement 1. Using the BonVision timing alone was also effective in identifying receptive fields.
Finally the authors use BonVision to perform one human psychophysical and several animal VR experiments to prove the functionality of the package in real-world scenarios. This includes an object size discrimination task with humans that relies on non-local cues to determine the efficacy of the cube map projection approach to 3D spaces (Fig 5D). Although the results seem reasonable to me (a non-expert in this domain), I feel it would be useful for the authors to compare this psychophysical discrimination curve to other comparable results. The animal experiments prove the utility of BonVision for common rodent VR tasks.
The psychometric test we performed on human subjects was primarily to test the ability of BonVision to present VR stimuli on a head-mounted display. We have edited the text to reflect this. The efficacy of the cube map approach for 3D spaces is well-established in computer graphics and gaming and is currently the industry standard, which was the reason for our choice.
In summary, the professionalism of the code base, the functional nature of Bonsai workflows, the removal of overhead via advanced JIT compilation techniques, the abstraction of shader programming to high-level drag and drop workflows, integration with a multitude of input and output hardware, integrated and standardized calibration routines, and integrated package management and workflow sharing capabilities make Bonsai/BonVision serious competitors to widely-used, closed-source visual programming tools for experiment control such as LabView and Simulink. BonVision showcases the power of the Bonsai language and package management ecosystem while providing superior design to alternatives in terms of ease of integration with data sources and facilitation of sharing standardized experiments. The authors exceeded the apparent aims of the project and I believe BonVision will become a widely used tool that has major benefits for improving experiment reproducibility across laboratories.
Reviewer #2 (Public Review):
BonVision is a package to create virtual visual environments, as well as classic visual stimuli. Running on top of Bonsai-RX it tries and succeeds in removing the complexity of the above mentioned task and creating a framework that allows non-programmers the opportunity to create complex, closed loop experiments. Including enough speed to capture receptive fields while recording different brain areas.
At the time of the review, the paper benchmarks the system using 60Hz stimuli, which is more than sufficient for the species tested, but leaves an open question on whether it could be used for other animal models that have faster visual systems, such as flies, bees etc.
Thank you for prompting us to do this - we have now added new benchmarks for a faster refresh rate (144 Hz; new Figure 4 - figure supplement 1).
The authors do show in a nice way how the system works and give examples for interested readers to start their first workflows with it. Moreover, they compare it to other existing software, making sure that readers know exactly what "they are buying" so they can make an informed decision when starting with the package.
Being written to run on top of Bonsai-RX, BonVision directly benefits from the great community effort that exists in expanding Bonsai, such as its integration with DeepLabCut and Auto-pi-lot. Showing that developing open source tools and fostering a community is a great way to bring research forward in an additive and less competitive way.
Reviewer #3 (Public Review):
Major comments:
While much of the classic literature on visual systems studies have utilized egocentrically defined ("2D") stimuli, it seems logical to project that present and future research will extend to not only 3D objects but also 3D environments where subjects can control their virtual locations and viewing perspectives. A single software package that easily supports both modalities can therefore be of particular interest to neuroscientists who wish to study brain function in 3D viewing conditions while also referencing findings to canonical 2D stimulus responses. Although other software packages exist that are specialized for each of the individual functionalities of BonVision, I think that the unifying nature of the package is appealing for reasons of reducing user training and experimental setup time costs, especially with the semi-automated calibration tools provided as part of the package. The provisions of documentation, demo experiments, and performance benchmarks are all highly welcome and one would hope that with community interest and contributions, this could make BonVision very friendly to entry by new users.
Given that one function of this manuscript is to describe the software in enough detail for users to judge whether it would be suited to their purposes, I feel that the writing should be fleshed out to be more precise and detailed about what the algorithms and functionalities are. This includes not shying away from stating limitations -- which as I see it, is just the reality of no tool being universal, but because of that is one of the most important information to be transmitted to potential users. My following comments point out various directions in which I think the manuscript can be improved.
We thank the reviewer for this suggestion. We have added a major new section, “Supplementary Details”, where we have highlighted known limitations and available workarounds. We also added new rows in the Supplementary Table that make these limitations transparent (eg. web-based deployment).
The biggest point of confusion for me was whether the 3D environment functionality of BonVision is the same as that provided by virtual spatial environment packages such as ViRMEn and gaming engines such as Unity. In the latter software, the virtual environment is specified by geometrically laying out the shape of the traversable world and locations of objects in it. The subject then essentially controls an avatar in this virtual world that can move and turn, and the software engine computes the effects of this movement (i.e. without any additional user code) then renders what the avatar should see onto a display device. I cannot figure out if this is how BonVision also works. My confusion can probably be cured by some additional description of what exactly the user has to do to specify the placement of 3D objects. From the text on cube mapping (lines 43 and onwards), I guessed that perhaps objects should be specified by their vectorial displacement from the subject, but I have very little confidence in my guess and also cannot locate this information either in the Methods or the software website. For Figure 5F it is mentioned that BonVision can be used to implement running down a virtual corridor for a mouse, so if some description can be provided of what the user has to do to implement this and what is done by the software package, that may address my confusion. If BonVision is indeed not a full 3D spatial engine, it would be important to mention these design/intent differences in the introduction as well as Supplementary Table 1.
Thank you for prompting us to do this. BonVision does indeed essentially render the view of an avatar in a virtual world (or multiple views, of multiple avatars), without any additional coding required by the user. We have now included in the new “Supplementary Details” specific pathways to the construction and rendering of 3D scenes. We have avoided the use of the terminology ‘game-engine’ as it has a particular definition that most softwares do not satisfy.
More generally, it would be useful to provide an overview of what the closed-loop rendering procedure is, perhaps including a Figure (different from Supplementary Figure 2, which seems to be regarding workflow but not the software platform structure). For example, I imagine that after the user-specified texture/object resources have been loaded, then some engine runs a continual loop where it somehow decides the current scene. As a user, I would want to know what this loop is and how I can control it. For example, can I induce changes in the presented stimuli as a function of time, whether this time-dependence has to be prespecified before runtime, or can I add some code that triggers events based on the specific history of what the subject has done in the experiment, and so forth. The ability to log experiment events, including any viewpoint changes in 3D scenes, is also critical, and most experimenters who intend to use it for neurophysiological recordings would want to know how the visual display information can be synchronized with their neurophysiological recording instrumental clocks. In sum, I would like to see a section added to the text to provide a high-level summary of how the package runs an experiment loop, explaining customizable vs. non-customizable (without directly editing the open source code) parts, and guide the user through the available experiment control and data logging options.
We have now added a brief paragraph regarding the basic structure of a BonVision program, and how to ‘close the loop’ in the new “Supplementary Details”.
Having some experience myself with the tedium (and human-dependent quality) of having to adjust either the experimental hardware or write custom software to calibrate display devices, I found the semi-automated calibration capabilities of BonVision to be a strong selling point. However I did not manage to really understand what these procedures are from the text and Figure 2C-F. In particular, I'm not sure what I have to do as a user to provide the information required by the calibration software (surely it is not the pieces of paper in Fig. 2C and 2E..?). If for example, the subject is a mouse head-fixed on a ball as in Figure 1E, do I have to somehow take a photo from the vantage of the mouse's head to provide to the system? What about the augmented reality rig where the subject is free to move? How can the calibration tool work with a single 2D snapshot of the rig when e.g. projection surfaces can be arbitrarily curved (e.g. toroidal and not spherical, or conical, or even more distorted for whatever reasons)? Do head-mounted displays require calibration, and if so how is this done? If the authors feel all this to be too technical to include in the main text, then the information can be provided in the Methods. I would however vote for this as being a major and important aspect of the software that should be given air time.
We have a dedicated webpage going through the step-by-step protocol for the automated screen calibration. We now explicitly point to this page in the new Supplementary Details section.
As the hardware-limited speed of BonVision is also an important feature, I wonder if the same ~2 frame latency holds also for the augmented reality rendering where the software has to run both pose tracking (DeepLabCut) as well as compute whole-scene changes before the next render. It would be beneficial to provide more information about which directions BonVision can be stressed before frame-dropping, which may perhaps be different for the different types of display options (2D vs. 3D, and the various display device types). Does the software maintain as strictly as possible the user-specified timing of events by dropping frames, or can it run into a situation where lags can accumulate? This type of technical information would seem critical to some experiments where timings of stimuli have to be carefully controlled, and regardless one would usually want to have the actual display times logged as previously mentioned. Some discussion of how a user might keep track of actual lags in their own setups would be appreciated.
We now provide this as part of the Supplementary Details, specifically animation and timing lags.
On the augmented reality mode, I am a little puzzled by the layout of Figure 3 and the attendant video, and I wonder if this is the best way to showcase this functionality. In particular, I'm not entirely sure what the main scene display is although it looks like some kind of software rendering — perhaps of what things might look like inside an actual rig looking in from the top? One way to make this Figure and Movie easier to grasp is to have the scene display be the different panels that would actually be rendered on each physical panel of the experiment box. The inset image of the rig should then have the projection turned on, so that the reader can judge what an actual experiment looks like. Right now it seems for some reason that the walls of the rig in the inset of the movie remain blank except for some lighting shadows. I don't know if this is intentional.
Because we have had limited experimental capacity in this period, we only simulated a real-time augmented reality environment off-line, using pre-existing videos of animal behaviour. We think that the comment above reflects a misunderstanding of what the Figure and associated Supplementary Movie represents, and we realise that their legends were not clear enough. We have now made sure that these legends make clear that these are based on simulations (new legends for Figure 3 and Figure 3 - video supplement 1).
-
Reviewer #3 (Public Review):
Summary:
This is a tools paper that describes an open source software package, BonVision, which aims to provide a non-programmer-friendly interface for configuring and presenting 2D as well as 3D visual stimuli to experimental subjects. A major design emphasis of the software is to allow users to define visual stimuli at a high level independent of the actual rendering physical devices, which can range from monitors to curved projection surfaces, binocular displays, and also augmented reality setups where the position of the subject relative to the display surfaces can vary and needs to be adjusted for. The package provides a number of semi-automated software calibration tools to significantly simplify the experimental job of setting up different rigs to faithfully present the intended stimuli, and is …
Reviewer #3 (Public Review):
Summary:
This is a tools paper that describes an open source software package, BonVision, which aims to provide a non-programmer-friendly interface for configuring and presenting 2D as well as 3D visual stimuli to experimental subjects. A major design emphasis of the software is to allow users to define visual stimuli at a high level independent of the actual rendering physical devices, which can range from monitors to curved projection surfaces, binocular displays, and also augmented reality setups where the position of the subject relative to the display surfaces can vary and needs to be adjusted for. The package provides a number of semi-automated software calibration tools to significantly simplify the experimental job of setting up different rigs to faithfully present the intended stimuli, and is capable of running at hardware-limited speeds comparable to and in some conditions better than existing packages such as Psychtoolbox and PsychoPy.
Major comments:
While much of the classic literature on visual systems studies have utilized egocentrically defined ("2D") stimuli, it seems logical to project that present and future research will extend to not only 3D objects but also 3D environments where subjects can control their virtual locations and viewing perspectives. A single software package that easily supports both modalities can therefore be of particular interest to neuroscientists who wish to study brain function in 3D viewing conditions while also referencing findings to canonical 2D stimulus responses. Although other software packages exist that are specialized for each of the individual functionalities of BonVision, I think that the unifying nature of the package is appealing for reasons of reducing user training and experimental setup time costs, especially with the semi-automated calibration tools provided as part of the package. The provisions of documentation, demo experiments, and performance benchmarks are all highly welcome and one would hope that with community interest and contributions, this could make BonVision very friendly to entry by new users.
Given that one function of this manuscript is to describe the software in enough detail for users to judge whether it would be suited to their purposes, I feel that the writing should be fleshed out to be more precise and detailed about what the algorithms and functionalities are. This includes not shying away from stating limitations -- which as I see it, is just the reality of no tool being universal, but because of that is one of the most important information to be transmitted to potential users. My following comments point out various directions in which I think the manuscript can be improved.
The biggest point of confusion for me was whether the 3D environment functionality of BonVision is the same as that provided by virtual spatial environment packages such as ViRMEn and gaming engines such as Unity. In the latter software, the virtual environment is specified by geometrically laying out the shape of the traversable world and locations of objects in it. The subject then essentially controls an avatar in this virtual world that can move and turn, and the software engine computes the effects of this movement (i.e. without any additional user code) then renders what the avatar should see onto a display device. I cannot figure out if this is how BonVision also works. My confusion can probably be cured by some additional description of what exactly the user has to do to specify the placement of 3D objects. From the text on cube mapping (lines 43 and onwards), I guessed that perhaps objects should be specified by their vectorial displacement from the subject, but I have very little confidence in my guess and also cannot locate this information either in the Methods or the software website. For Figure 5F it is mentioned that BonVision can be used to implement running down a virtual corridor for a mouse, so if some description can be provided of what the user has to do to implement this and what is done by the software package, that may address my confusion. If BonVision is indeed not a full 3D spatial engine, it would be important to mention these design/intent differences in the introduction as well as Supplementary Table 1.
More generally, it would be useful to provide an overview of what the closed-loop rendering procedure is, perhaps including a Figure (different from Supplementary Figure 2, which seems to be regarding workflow but not the software platform structure). For example, I imagine that after the user-specified texture/object resources have been loaded, then some engine runs a continual loop where it somehow decides the current scene. As a user, I would want to know what this loop is and how I can control it. For example, can I induce changes in the presented stimuli as a function of time, whether this time-dependence has to be prespecified before runtime, or can I add some code that triggers events based on the specific history of what the subject has done in the experiment, and so forth. The ability to log experiment events, including any viewpoint changes in 3D scenes, is also critical, and most experimenters who intend to use it for neurophysiological recordings would want to know how the visual display information can be synchronized with their neurophysiological recording instrumental clocks. In sum, I would like to see a section added to the text to provide a high-level summary of how the package runs an experiment loop, explaining customizable vs. non-customizable (without directly editing the open source code) parts, and guide the user through the available experiment control and data logging options.
Having some experience myself with the tedium (and human-dependent quality) of having to adjust either the experimental hardware or write custom software to calibrate display devices, I found the semi-automated calibration capabilities of BonVision to be a strong selling point. However I did not manage to really understand what these procedures are from the text and Figure 2C-F. In particular, I'm not sure what I have to do as a user to provide the information required by the calibration software (surely it is not the pieces of paper in Fig. 2C and 2E..?). If for example, the subject is a mouse head-fixed on a ball as in Figure 1E, do I have to somehow take a photo from the vantage of the mouse's head to provide to the system? What about the augmented reality rig where the subject is free to move? How can the calibration tool work with a single 2D snapshot of the rig when e.g. projection surfaces can be arbitrarily curved (e.g. toroidal and not spherical, or conical, or even more distorted for whatever reasons)? Do head-mounted displays require calibration, and if so how is this done? If the authors feel all this to be too technical to include in the main text, then the information can be provided in the Methods. I would however vote for this as being a major and important aspect of the software that should be given air time.
As the hardware-limited speed of BonVision is also an important feature, I wonder if the same ~2 frame latency holds also for the augmented reality rendering where the software has to run both pose tracking (DeepLabCut) as well as compute whole-scene changes before the next render. It would be beneficial to provide more information about which directions BonVision can be stressed before frame-dropping, which may perhaps be different for the different types of display options (2D vs. 3D, and the various display device types). Does the software maintain as strictly as possible the user-specified timing of events by dropping frames, or can it run into a situation where lags can accumulate? This type of technical information would seem critical to some experiments where timings of stimuli have to be carefully controlled, and regardless one would usually want to have the actual display times logged as previously mentioned. Some discussion of how a user might keep track of actual lags in their own setups would be appreciated.
On the augmented reality mode, I am a little puzzled by the layout of Figure 3 and the attendant video, and I wonder if this is the best way to showcase this functionality. In particular, I'm not entirely sure what the main scene display is although it looks like some kind of software rendering — perhaps of what things might look like inside an actual rig looking in from the top? One way to make this Figure and Movie easier to grasp is to have the scene display be the different panels that would actually be rendered on each physical panel of the experiment box. The inset image of the rig should then have the projection turned on, so that the reader can judge what an actual experiment looks like. Right now it seems for some reason that the walls of the rig in the inset of the movie remain blank except for some lighting shadows. I don't know if this is intentional.
-
Reviewer #2 (Public Review):
BonVision is a package to create virtual visual environments, as well as classic visual stimuli. Running on top of Bonsai-RX it tries and succeeds in removing the complexity of the above mentioned task and creating a framework that allows non-programmers the opportunity to create complex, closed loop experiments. Including enough speed to capture receptive fields while recording different brain areas.
At the time of the review, the paper benchmarks the system using 60Hz stimuli, which is more than sufficient for the species tested, but leaves an open question on whether it could be used for other animal models that have faster visual systems, such as flies, bees etc.
The authors do show in a nice way how the system works and give examples for interested readers to start their first workflows with it. …
Reviewer #2 (Public Review):
BonVision is a package to create virtual visual environments, as well as classic visual stimuli. Running on top of Bonsai-RX it tries and succeeds in removing the complexity of the above mentioned task and creating a framework that allows non-programmers the opportunity to create complex, closed loop experiments. Including enough speed to capture receptive fields while recording different brain areas.
At the time of the review, the paper benchmarks the system using 60Hz stimuli, which is more than sufficient for the species tested, but leaves an open question on whether it could be used for other animal models that have faster visual systems, such as flies, bees etc.
The authors do show in a nice way how the system works and give examples for interested readers to start their first workflows with it. Moreover, they compare it to other existing software, making sure that readers know exactly what "they are buying" so they can make an informed decision when starting with the package.
Being written to run on top of Bonsai-RX, BonVision directly benefits from the great community effort that exists in expanding Bonsai, such as its integration with DeepLabCut and Auto-pi-lot. Showing that developing open source tools and fostering a community is a great way to bring research forward in an additive and less competitive way.
-
Reviewer #1 (Public Review):
In this project, the authors set out to create an easy to use piece of software with the following properties: The software should be capable of creating immersive (closed loop) virtual environments across display hardware and display geometries. The software should permit easy distribution of formal experiment descriptions with minimal changes required to adapt a particular experimental workflow to the hardware present in any given lab while maintaining world-coordinates and physical properties (e.g. luminance levels and refresh rates) of visual stimuli. The software should provide equal or superior performance for generating complex visual cues and/or immersive visual environments in comparison with existing options. The software should be automatically integrated with many other potential data streams …
Reviewer #1 (Public Review):
In this project, the authors set out to create an easy to use piece of software with the following properties: The software should be capable of creating immersive (closed loop) virtual environments across display hardware and display geometries. The software should permit easy distribution of formal experiment descriptions with minimal changes required to adapt a particular experimental workflow to the hardware present in any given lab while maintaining world-coordinates and physical properties (e.g. luminance levels and refresh rates) of visual stimuli. The software should provide equal or superior performance for generating complex visual cues and/or immersive visual environments in comparison with existing options. The software should be automatically integrated with many other potential data streams produced by 2-photon imaging, electrophysiology, behavioral measurements, markerless pose estimation processing, behavioral sensors, etc.
To accomplish these goals, the authors created two major software libraries. The first is a package for the Bonsai visual programming language called "Bonsai.Shaders" that brings traditionally low-level, imperative OpenGL programming into Bonsai's reactive framework. This library allows shader programs running on the GPU to seamlessly interact, using drag and drop visual programming, with the multitude of other processing and IO elements already present in numerous Bonsai packages. The creation of this library alone is quite a feat given the complexities of mapping the procedural, imperative, and stateful design of OpenGL libraries to Bonsai's event driven, reactive architecture. However, this library is not mentioned in the manuscript despite its power for tasks far beyond the creation of visual stimuli (e.g. GPU-based coprocessing) and, unlike BonVision itself, is largely undocumented. I don't think that this library should take center stage in this manuscript, but I do think its use in the creation of BonVision as well as some documentation on its operators would be very useful for understanding BonVision itself.
Following the creation of Bonsai.Shaders, the authors used it to create BonVision which is an abstraction on top of the Shaders library that allows plug and play creation of visual stimuli and immersive visual environments that react to input from the outside world. Impressively, this library was implemented almost entirely using the Bonsai visual programming language itself, showcasing its power as a domain-specific language. However, this fact was not mentioned in the manuscript and I feel it is a worthwhile point to make. The design of BonVision, combined with the functional nature of Bonsai, enforces hard boundaries between the experimental design of visual stimuli and (1) the behavioral input hardware used to drive them, (2) the dimensionality of the stimuli (i.e. 2D textures via 3D objects), (3) the specific geometry of 3D displays (e.g. dual monitors, versus spherical projection, versus head mounted stereo vision hardware), and (4) automated hardware calibration routines. Because of these boundaries, experiments designed using BonVision become easy to share across labs even if they have very different experimental setups. Since Bonsai has integrated and standardized mechanisms for sharing entire workflows (via copy paste of XML descriptions or upload of workflows to publicly accessible Nuget package servers), this feature is immediately usable by labs in the real world.
After creating these pieces of software, the authors benchmarked them against other widely used alternatives. IonVisoin met or exceeded frame rate and rendering latency performance measures when compared to other single purpose libraries. BonVision is able to do this while maintaining its generality by taking advantage of advanced JIT compilation features provided by the .NET runtime and using bindings to low-level graphics libraries that were written with performance in mind. The authors go on to show the real-world utility of BonVision's performance by mapping the visual receptive fields of LFP in mouse superior colliculus and spiking in V1. The fact that they were able to obtain receptive fields indicates that visual stimuli had sufficient temporal precision. However, I do not follow the logic as to why this is because the receptive fields seem to have been created using post-hoc aligned stimulus-ephys data, that was created by measuring the physical onset times of each frame using a photodiode (line 389). Wouldn't this preclude any need for accurate stimulus timing presentation?
Finally the authors use BonVision to perform one human psychophysical and several animal VR experiments to prove the functionality of the package in real-world scenarios. This includes an object size discrimination task with humans that relies on non-local cues to determine the efficacy of the cube map projection approach to 3D spaces (Fig 5D). Although the results seem reasonable to me (a non-expert in this domain), I feel it would be useful for the authors to compare this psychophysical discrimination curve to other comparable results. The animal experiments prove the utility of BonVision for common rodent VR tasks.
In summary, the professionalism of the code base, the functional nature of Bonsai workflows, the removal of overhead via advanced JIT compilation techniques, the abstraction of shader programming to high-level drag and drop workflows, integration with a multitude of input and output hardware, integrated and standardized calibration routines, and integrated package management and workflow sharing capabilities make Bonsai/BonVision serious competitors to widely-used, closed-source visual programming tools for experiment control such as LabView and Simulink. BonVision showcases the power of the Bonsai language and package management ecosystem while providing superior design to alternatives in terms of ease of integration with data sources and facilitation of sharing standardized experiments. The authors exceeded the apparent aims of the project and I believe BonVision will become a widely used tool that has major benefits for improving experiment reproducibility across laboratories.
-
Evaluation Summary:
Increasingly, neuroscience experiments require immersive virtual environments that approximate natural sensory motor loops while permitting high-bandwidth measurements of brain activity. BonVision is an open-source graphics programming library that allows experimenters to quickly implement immersive 3D visual environments across display hardware and geometry with automated calibration and integration with hundreds of different neural recording technologies, behavioral apparatuses, etc. BonVision standardizes sharing complex, closed-loop visual tasks between labs with vastly different equipment, provides a concrete and easy way to do so, and should be of interest to a wide array of visual systems neuroscientists.
-
-
-