Impact of deep feature extraction strategies on clinical outcome prediction: a comparative analysis

Background Deep features (DFs) extracted from medical images using convolutional neural networks (CNNs) have shown promising results for predictive modeling in oncology. However, there is no consensus on optimal DF extraction strategies, and methodological choices related to network architecture, training paradigm, and input configuration may substantially affect predictive performance. Objectives To systematically evaluate different DF extraction strategies across imaging modalities and clinical endpoints, and to assess their impact on predictive performance in oncology applications. Methods Multiple DF extraction approaches were evaluated, including 2D and 3D autoencoders trained from scratch, fine-tuned pretrained networks, and different input configurations (whole images versus lesion-centered crops and varying spatial resolutions). These strategies were assessed in two clinical scenarios: prediction of best overall response (BOR) from computed tomography (CT) in non-small cell lung cancer, and one-year overall survival (OS) from magnetic resonance imaging (MRI) in glioblastoma. Results Predictive performance varied substantially depending on the DF extraction strategy. For BOR prediction, pretrained models achieved moderate performance, with AUC values as low as 0.68, whereas combining radiomics with DFs extracted using a 2D autoencoder trained from scratch improved performance up to an AUC of 0.85. In glioblastoma, a fine-tuned VGG16 model achieved an AUC of 0.87 using single-modality MRI. Models relying exclusively on DFs showed comparable performance to those combining radiomics and DFs, indicating robustness and reduced dependence on precise lesion segmentation. Conclusions The choice of DF extraction methodology has a critical impact on predictive performance. Carefully designed DF strategies can serve as reliable imaging biomarkers and support predictive modeling across different imaging modalities and oncological endpoints.