Advanced Computer Assisted Surgery: CT Imaging to Augmented Reality projection of lung model using deep learning segmentation technique and 3D biomechanical generation

New medical imaging and deep learning-based segmentation technologies, as well as augmented reality (AR) technologies have greatly contributed to the evolution of computer-assisted surgery (CAS). Nonetheless, biomechanical models adaptation on the real time continues to be a significant challenge because of tissue deformation during surgical operations. The traditional visualization techniques will be based on the pre-operative CT and MRI that is inherently static and will not capture the changes in anatomy during the surgery. A detailed framework is suggested in this study to improve surgical navigation that incorporates the use of CT image segmentation through deep learning, 3D reconstruction, biomechanical modeling, and real-time visualization through AR. The technique relies on the ResUNet++ architecture to attain high-quality lung segmentation. Image-to-Mesh (Im2Mesh) framework is used to reconstruct the 3D model from the segmented images. Finally, the Simulation Open Framework Architecture (SOFA) is used to simulate the biomechanical behaviour of the patient-specific lung model through the application of finite element modeling (FEM). Lastly, the reconstructed model would be displayed in real time using Meta 2 AR headset. The performance of the segmentation is exceptional with experimental results showing a Dice similarity coefficient of 0.9738 and the overall accuracy of 0.9599. All these results justify the benefit of the presented framework in providing an accurate segmentation and realistic anatomical representation of the biomechanical plausible model, which contributes to the success of AR-guided surgical procedures.