Towards Mixed Reality-Navigated Surgery: Point Cloud Surface Registration for Automated 3D Image Overlay

Purpose Minimally invasive and robotically assisted surgery relies on guidance systems to aid in navigation. Using high accuracy tracking, surface registration algorithms and extended reality (XR) devices, we aim to develop a system to provide a real-time, immersive visualization of previously acquired computed tomography (CT) data through automated surface registration as an intuitive and radiation free alternative to conventional image guiding methods. Methods Combining state-of-the art XR devices, high-precision tracking and scanning and advanced surface registration algorithms, we propose a modular and immersive system to provide a holographic overlay of relevant tools and anatomical structures in a surgical context. We measure and verify the accuracy of various modalities to provide a suitable foundation for further technical enhancements and clinical studies. Results Accuracy measurements for the XR holographic overlay as well as automatically calculated positioning of CT data with surface registration are in line with conventional image-guided approaches. Analysis of the acquired data indicates probable avenues to further optimize the system. Conclusion The system developed in this paper provides a promising foundation for an immersive, real-time guidance system for interventional, minimally invasive, and robotically assisted procedures. Further studies to verify the accuracy and usability in a clinical context are initiated.