Large-scale segmentation model facilitates intraoperative histopathology by third harmonic generation microscopy

Histopathological assessment is the corner stone for tumor diagnosis in the clinic, but it is in its optimal form (using paraffin-embedded sections) not readily available during surgery. The intraoperative alternative, frozen section analysis, also takes about 30 minutes. Label-free third harmonic generation (THG) microscopy shows great potential to fill this gap, by providing tissue information of histopathological quality instantly. Compact THG imaging devices have been developed and are tested at clinical sites, however, one major issue remaining is the lack of robust image quantification methods. Developing such methods is challenged by limited data resource and the need for high model generalization capability. Here, we present 1) an image generation model based on diffusion bridge to convert histological images to THG images, overcoming the issue of limited THG data, and 2) the first large-scale foundation model Cell-SAM for quantification of key tumor elements present in THG images, with few-shot generalization capability. Cell-SAM provides two complementary modes. The boxes-prompted SAM mode, boosted by a developed object detection model whose backbone is pre-trained by a designed unsupervised strategy, can not only segment, classify and separate cells, but also produce large scale data for fine-tuning SAM. The fine-tuned SAM mode can directly segment blood vessels and all cells regardless of cell types, with improved accuracy. These two modes of Cell-SAM enable segmentation of normal cells, tumor cells, immune cells and blood vessels. We validate the models on data obtained from various CNS tumor types and lymphomas, collected with a portable THG imaging device in the Princess Máxima Center for Pediatric Oncology, achieving a segmentation accuracy of higher than 98%. Statistical analysis of cell density and vascular density further suggests the potential of THG imaging for tumor typing. Combined with the short imaging time of THG, we conclude that the combination of Cell-SAM and compact THG imaging provides a unique approach for ‘instant pathology’ intraoperatively.