CompoundRay, an open-source tool for high-speed and high-fidelity rendering of compound eyes
Curation statements for this article:-
Curated by eLife
Evaluation Summary:
This work reports on a mathematical modeling system and associated software implementation for compound eye vision. A critical advance reported here is the ability to model each ommatidium with independent properties and on a software implementation that runs in real time, with tantalizing applications in both modeling biological systems such as insect compound eyes, and the exploration of the possible applications of compound eye vision in robotics.
(This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #1 and Reviewer #2 agreed to share their name with the authors.)
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (eLife)
- Computational and Systems Biology (eLife)
Abstract
Revealing the functioning of compound eyes is of interest to biologists and engineers alike who wish to understand how visually complex behaviours (e.g. detection, tracking, and navigation) arise in nature, and to abstract concepts to develop novel artificial sensory systems. A key investigative method is to replicate the sensory apparatus using artificial systems, allowing for investigation of the visual information that drives animal behaviour when exposed to environmental cues. To date, ‘compound eye models’ (CEMs) have largely explored features such as field of view and angular resolution, but the role of shape and overall structure have been largely overlooked due to modelling complexity. Modern real-time ray-tracing technologies are enabling the construction of a new generation of computationally fast, high-fidelity CEMs. This work introduces a new open-source CEM software ( CompoundRay ) that is capable of accurately rendering the visual perspective of bees (6000 individual ommatidia arranged on 2 realistic eye surfaces) at over 3000 frames per second. We show how the speed and accuracy facilitated by this software can be used to investigate pressing research questions (e.g. how low resolution compound eyes can localise small objects) using modern methods (e.g. machine learning-based information exploration).
Article activity feed
-
-
Author Response
Reviewer #1 (Public Review):
In their manuscript "CompoundRay: An open-source tool for high-speed and high-fidelity rendering of compound eyes", the authors describe their software package to simulate vision in 3D environments as perceived through a compound eye of arbitrary geometry. The software uses hardware accelerated ray casting using NVIDIA Optix to generate simulations at very high framerates of ~5000 FPS on recent NVIDIA graphics hardware. The software is released under the permissive MIT license, publicly available at https://github.com/ManganLab/eye-renderer, and well documented. CompoundRay can be extraordinarily useful for computational neuroscience experiments exploring insect vision and robotics with insect like vision devices.
The manuscript describes the target of the work: realistic simulating …
Author Response
Reviewer #1 (Public Review):
In their manuscript "CompoundRay: An open-source tool for high-speed and high-fidelity rendering of compound eyes", the authors describe their software package to simulate vision in 3D environments as perceived through a compound eye of arbitrary geometry. The software uses hardware accelerated ray casting using NVIDIA Optix to generate simulations at very high framerates of ~5000 FPS on recent NVIDIA graphics hardware. The software is released under the permissive MIT license, publicly available at https://github.com/ManganLab/eye-renderer, and well documented. CompoundRay can be extraordinarily useful for computational neuroscience experiments exploring insect vision and robotics with insect like vision devices.
The manuscript describes the target of the work: realistic simulating vision as perceived by compound eyes in arthropods and thoroughly reviews the state of the art. The software CompoundRay is then presented to address the shortcomings of existing solutions which are either oversimplifying the geometry of compound eyes (e.g. assuming shared focal points), using an unrealistic rendering model (e.g. local geometry projection) or being slower than real-time.
The manuscript then details implementation choices and the conceptual design and components of the software. The effect of compound eye geometries is discussed using some examples. The speed of the simulator depending on SNR is assessed and shown for three physiological compound eye geometries.
I find the described open source compound eye vision simulation software extraordinarily useful and important. The manuscript reviews the state of the art well. The figures are well made and easy to understand. The description of the method and software, in my opinion, needs work to make it more succinct and easier to understand (details below). In general, I found relevant concepts and ideas buried in overly complicated meandering descriptions, and important details missing. Some editorial work could help a lot here.
Thank you for the very positive feedback.
Major:
- The transfer of the scene seen by an arbitrary geometry compound eye into a display image lacks information and discussion about the focal center/ choice of projection. I believe that only the orientation of ommatidia is used to generate this projection which leads to the overlap/ non-coverage in Fig. 5c. Correct? It would be great if, for such scenarios, a semi-orthogonal+cylindrical projection could be added? Also, please explain better.
For clarification, CompoundRay allows for a number of projection modes from any 3D sampling surface to visualised 2D projections. This has now been made clearer with an updated Methods section “From single ommatidia to full compound eye” (lines 171-188), and also a more clarified explanation of the display pipeline within the “CompoundRay Software Pipeline” section (lines 245-247).
We note that Fig 5 is simply intended as an example of the extreme differences in information that can be provided by nodel (the current state of the art) and non-nodal imagers (as in biological systems). A user could indeed produce custom projections (as now noted in the future work section of the Discussion), such as semi-orthgonal+cylindrical projections by modifying the projection shaders but we do not feel that this adds substantially to the desired message of Fig 5 as currently all view images are generated using the same projection method allowing them to be compared. Further to this, a semi-orthogonal+cylindrical projection would only serve to display these types of eyes and not be of significant use outside of this category of design. Rather, the utility of CompoundRay for research is now demonstrated by the inclusion of an entirely new example experiment (lines 394-467) (Fig 10) which compares artificial and realistic compound eye models in a visual tracking task.
In additional we note that specific references to the “orientation-wise spherical mapping” of images have been added to appropriate image captions (Fig 5 & 6).
Finally, we have attempted to be more explicit about about the way that 2D projection systems work within CompoundRay (182-185)
- It is clear that CompoundRay is fast and addresses complex compound eyegeometries. It remains unclear, why global illumination models are discussed while the implementation uses ray casting to sample textures without illumination which is equivalent to projection rendering which runs fast on much simpler hardware. If the argument is speed and simplicity of writing the code, that's great, write it so. If it is an intrinsic advantage of the ray-casting method, then comparison with the 'many-cameras' approach sketched below should be done:
In your model, each ommatidium is an independent pin-hole camera. Instead of sampling this camera by ray-casting, you could use projection rendering to generate a small image per ommatidium-camera, then average over the intensities with an appropriate foveation function (Gaussian in your scenario, but could be other kernels). The resolution of the per-camera image defines the number of samples for anti-aliasing, randomizing will be harder than with ray-casting ;). What else is better when using ray-casting? Fewer samples? Hardware support? Possible to increase recursion depth and do more global things than local illumination and shadows? Easier to parallelize on specific hardware and with specific software libraries? Don't you think it would make sense to explain the entire procedure like that? That would make the choice to use ray-casting much easier to understand for naive readers like me.
Thanks for this feedback, and can see that it was misleading to include this in our previous Methods section. We have now reduced and moved discussion of global illumination models to the future work section at the end of the Discussion. We have also added a clarification to the end of this document that summarises this point as it was raised by multiple reviewers (see Changes Relating to Colour and Light Sampling)
- CompoundRay, as far as I understand, currently renders RGB images at 8-bitprecision. This may not be sufficient to simulate the vision of arthropod eyes that are sensitive to other wavelengths and at variable sensitivity.
Thanks for pointing out this easy-to-miss implementation detail. Indeed, you are correct that the native output is at 8-bit level as is standard to match display equipment. However, we note that the underlying on-GPU implementation operates at a 32-bit depth, so exposing this to the higher-level Python API should be possible, which could then be used as you suggest. We view adding enhanced lighting properties including shadows, illumination and higher bit depths so as to better support increased-bandwidth visual sensor simulation as future updates which we have now outlined in the Discussion (line 549-553).
Reviewer #2 (Public Review):
In this paper, the authors describe a new software tool which simulates the spatial geometry of insect compound eyes. This new tool improves on existing tools by taking advantage of recent advances in computer graphics hardware which supports high performance real-time ray tracing to enable simulation of insect eyes with greater fidelity than previously. For example, this tool allows the simulation of eyes in which the optical axes of the ommatidia do not converge to a single point and takes advantage of ray tracing as a rendering modality to directly sample the scene with simulated light rays. The paper states these aims clearly and convincingly demonstrates that the software meets these aims. I think the availability of a high-quality, open-source software tool to simulate the geometry of compound eyes will be generally useful to researchers studying vision and visual behavior in insects and roboticists working on bio-inspired visual systems, and I am optimistic that the describe tool could fill that role well.
Thankyou for the positive feedback.
As far as weaknesses of the paper, the most major issue for me is that I could not find any example of why the additional modeling fidelity or speed is useful in understanding a biological phenomenon. While the work is technically impressive, I think such a demonstration would increase its impact substantially.
An example experiment has been added as requested.
I can identify a few more, relatively minor, weaknesses: the software tool is not particularly easy to install but I think this is due primarily to the usage of advanced graphics hardware and software libraries and hence not something the authors can easily correct. In fact, the authors provide substantial documentation to help with installation.
Indeed, we have tried to ease installation as much as possible by provided detailed documentation. This has been updated since initial submission and proven sufficient for multiple users. We have looked into dockerising the code but as correctly identified by the reviewer there are significant challenges due to proprietory hardware and their drivers.
Another weakness of the tool, which the authors might like to address in the paper, is that there are some aspects of insect vision and optics which are not directly addressed. For example, the wavelength and polarization properties of light rays are hardly addressed despite extensive research into the sensation of these properties. Furthermore, the optical model employed here is purely ray based and would not allow investigating the wave nature of light which is important for propagation from the corneal surface to the photoreceptors in many species.
Indeed, it is correct that the current implementation does not allow such advanced light modellign features but as our initial aim was to allow arbitrary surface shapes this was considered beyond the scope of this work. However, we have added a short description of extensions that the method would allow without significant architectural changes which include many of those listed by the reviewer. As the renderer simulates light as it reaches the lens surface, it is hoped that further works will be able to use this natural boundary between the eye surface and it’s internals to build further computational models that use the data generated in CompoundRay as a starting point to then simulate inside-eye light transport.
-
Evaluation Summary:
This work reports on a mathematical modeling system and associated software implementation for compound eye vision. A critical advance reported here is the ability to model each ommatidium with independent properties and on a software implementation that runs in real time, with tantalizing applications in both modeling biological systems such as insect compound eyes, and the exploration of the possible applications of compound eye vision in robotics.
(This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #1 and Reviewer #2 agreed to share their name with the authors.)
-
Reviewer #1 (Public Review):
In their manuscript "CompoundRay: An open-source tool for high-speed and high-fidelity rendering of compound eyes", the authors describe their software package to simulate vision in 3D environments as perceived through a compound eye of arbitrary geometry. The software uses hardware accelerated ray casting using NVIDIA Optix to generate simulations at very high framerates of ~5000 FPS on recent NVIDIA graphics hardware. The software is released under the permissive MIT license, publicly available at https://github.com/ManganLab/eye-renderer, and well documented. CompoundRay can be extraordinarily useful for computational neuroscience experiments exploring insect vision and robotics with insect like vision devices.
The manuscript describes the target of the work: realistic simulating vision as perceived by …
Reviewer #1 (Public Review):
In their manuscript "CompoundRay: An open-source tool for high-speed and high-fidelity rendering of compound eyes", the authors describe their software package to simulate vision in 3D environments as perceived through a compound eye of arbitrary geometry. The software uses hardware accelerated ray casting using NVIDIA Optix to generate simulations at very high framerates of ~5000 FPS on recent NVIDIA graphics hardware. The software is released under the permissive MIT license, publicly available at https://github.com/ManganLab/eye-renderer, and well documented. CompoundRay can be extraordinarily useful for computational neuroscience experiments exploring insect vision and robotics with insect like vision devices.
The manuscript describes the target of the work: realistic simulating vision as perceived by compound eyes in arthropods and thoroughly reviews the state of the art. The software CompoundRay is then presented to address the shortcomings of existing solutions which are either oversimplifying the geometry of compound eyes (e.g. assuming shared focal points), using an unrealistic rendering model (e.g. local geometry projection) or being slower than real-time.
The manuscript then details implementation choices and the conceptual design and components of the software. The effect of compound eye geometries is discussed using some examples. The speed of the simulator depending on SNR is assessed and shown for three physiological compound eye geometries.
Opinion:
I find the described open source compound eye vision simulation software extraordinarily useful and important. The manuscript reviews the state of the art well. The figures are well made and easy to understand. The description of the method and software, in my opinion, needs work to make it more succinct and easier to understand (details below). In general, I found relevant concepts and ideas buried in overly complicated meandering descriptions, and important details missing. Some editorial work could help a lot here.
Major:
1. The transfer of the scene seen by an arbitrary geometry compound eye into a display image lacks information and discussion about the focal center/ choice of projection. I believe that only the orientation of ommatidia is used to generate this projection which leads to the overlap/ non-coverage in Fig. 5c. Correct? It would be great if, for such scenarios, a semi-orthogonal+cylindrical projection could be added? Also, please explain better.
2. It is clear that CompoundRay is fast and addresses complex compound eye geometries. It remains unclear, why global illumination models are discussed while the implementation uses ray casting to sample textures without illumination which is equivalent to projection rendering which runs fast on much simpler hardware. If the argument is speed and simplicity of writing the code, that's great, write it so. If it is an intrinsic advantage of the ray-casting method, then comparison with the 'many-cameras' approach sketched below should be done:
In your model, each ommatidium is an independent pin-hole camera. Instead of sampling this camera by ray-casting, you could use projection rendering to generate a small image per ommatidium-camera, then average over the intensities with an appropriate foveation function (Gaussian in your scenario, but could be other kernels). The resolution of the per-camera image defines the number of samples for anti-aliasing, randomizing will be harder than with ray-casting ;). What else is better when using ray-casting? Fewer samples? Hardware support? Possible to increase recursion depth and do more global things than local illumination and shadows? Easier to parallelize on specific hardware and with specific software libraries? Don't you think it would make sense to explain the entire procedure like that? That would make the choice to use ray-casting much easier to understand for naive readers like me.
3. CompoundRay, as far as I understand, currently renders RGB images at 8-bit precision. This may not be sufficient to simulate the vision of arthropod eyes that are sensitive to other wavelengths and at variable sensitivity.
-
Reviewer #2 (Public Review):
In this paper, the authors describe a new software tool which simulates the spatial geometry of insect compound eyes. This new tool improves on existing tools by taking advantage of recent advances in computer graphics hardware which supports high performance real-time ray tracing to enable simulation of insect eyes with greater fidelity than previously. For example, this tool allows the simulation of eyes in which the optical axes of the ommatidia do not converge to a single point and takes advantage of ray tracing as a rendering modality to directly sample the scene with simulated light rays. The paper states these aims clearly and convincingly demonstrates that the software meets these aims. I think the availability of a high-quality, open-source software tool to simulate the geometry of compound eyes …
Reviewer #2 (Public Review):
In this paper, the authors describe a new software tool which simulates the spatial geometry of insect compound eyes. This new tool improves on existing tools by taking advantage of recent advances in computer graphics hardware which supports high performance real-time ray tracing to enable simulation of insect eyes with greater fidelity than previously. For example, this tool allows the simulation of eyes in which the optical axes of the ommatidia do not converge to a single point and takes advantage of ray tracing as a rendering modality to directly sample the scene with simulated light rays. The paper states these aims clearly and convincingly demonstrates that the software meets these aims. I think the availability of a high-quality, open-source software tool to simulate the geometry of compound eyes will be generally useful to researchers studying vision and visual behavior in insects and roboticists working on bio-inspired visual systems, and I am optimistic that the describe tool could fill that role well.
As far as weaknesses of the paper, the most major issue for me is that I could not find any example of why the additional modeling fidelity or speed is useful in understanding a biological phenomenon. While the work is technically impressive, I think such a demonstration would increase its impact substantially. I can identify a few more, relatively minor, weaknesses: the software tool is not particularly easy to install but I think this is due primarily to the usage of advanced graphics hardware and software libraries and hence not something the authors can easily correct. In fact, the authors provide substantial documentation to help with installation. Another weakness of the tool, which the authors might like to address in the paper, is that there are some aspects of insect vision and optics which are not directly addressed. For example, the wavelength and polarization properties of light rays are hardly addressed despite extensive research into the sensation of these properties. Furthermore, the optical model employed here is purely ray based and would not allow investigating the wave nature of light which is important for propagation from the corneal surface to the photoreceptors in many species.
-
Reviewer #3 (Public Review):
Millward et al. introduce their software CompoundRay that aims at realistic, high-performance, real-time modeling of compound eye function, and pose it as a tool to explore compound eye-mediated behaviors. The design and operation of the software, and its critical advantages over competing software, are spelled out, and some illustrations are offered.
Overall, the paper figures, both in presentation and content, could be strengthened significantly. In particular, specific compound eye configurations with heterogeneous ommatidia, which the software supports, are largely lacking. One such comparison could have been done with the eyes of rober flies, which present a fovea of high visual acuity ommatidia.
-
