Three-dimensional H&E histopathology powered by deep learning-assisted multimodal nonlinear microscopy

The diagnostic and prognostic potential of histopathology can be significantly enhanced with three-dimensional (3D) imaging. Histopathology relies on the gold-standard 2D inspection of hematoxylin and eosin (H&E)-stained tissue sections with a brightfield microscope. We introduce a novel methodology termed nonlinear-H&E (N-H&E) that utilizes inherent optical sectioning of nonlinear microscopy for 3D volumetric investigations. Multimodal multiphoton excitation fluorescence (MPF) and third-harmonic generation (THG) microscopy is employed to image fluorescent and nonfluorescent structures. The optical sections are used to create 3D brightfield-equivalent H&E volumes by employing a generative adversarial network (GAN)-based image-to-image translation. N-H&E generates realistic 2D and 3D brightfield images with enhanced axial resolution. Furthermore, polarimetric second-harmonic generation (P-SHG) is applied to visualize the 3D collagen architecture. The overlay of volumetric P-SHG and GAN-generated H&E images provides an enriched view of tumor microenvironment for 3D histopathology. Reconstructed images meet standards for visual quality, structural accuracy, and clinical assessment reliability.