Towards H&E Referenced Multiplex Immunofluorescence Interpretation: Spatial Co-localization, Cell Feature Validation, and Virtual H&E Generation

Multiplexed Immunofluorescence (MxIF) enables detailed immune cell phenotyping, providing critical insights into cell behavior within the tumor immune microenvironment (TIME). However, signal integrity can be compromised due to the complex cyclic staining processes inherent to MxIF. Hematoxylin and Eosin (H&E) staining, on the other hand, offers complementary information through its depiction of cell morphology and texture patterns and is often visually cross-referenced with MxIF in clinical settings. In this study, we proposed a novel framework to align H&E and MxIF images for precise cross-modal cell feature validation. Using cell detection outputs from each modality as anchors, we formulated the multimodal image registration problem as point set alignment. Coherent Point Drift (CPD) is employed for initial alignment, followed by Graph Matching (GM) for refinement. Evaluations on ovarian cancer tissue microarrays (TMAs) demonstrate that our method achieves high alignment accuracy, enabling reliable validation of cell-level features across modalities for both restained and serial sections. Our results indicate that restained H&E enhances confidence in findings derived from MxIF. Additionally, we demonstrated the feasibility of generating high-quality virtual H&E images from MxIF data when restained H&E is unavailable, offering a viable alternative for integrated multimodal analysis.