Transformer-based multimodal model for estimation of appendicular lean mass using incomplete chest radiographs and electronic health record

Background Sarcopenia is a muscle disease that increases the risk of falls, fractures, and mortality. Appendicular lean mass (ALM) assessment is central to its diagnosis, but standard methods like dual-energy X-ray absorptiometry (DXA) have accessibility and cost issues. Previous artificial intelligence (AI) studies for sarcopenia assessment have been limited to single modalities and have not adequately addressed missing data modalities. This study aimed to develop and validate a multimodal AI model using frontal and lateral chest radiographs and electronic health record (EHR) data to estimate ALM and detect low muscle mass, and to investigate the robustness of a transformer-based algorithm to missing modalities. Methods This model development and validation study adhered to TRIPOD guidelines. The derivation cohort included 3,295 observations from 1,524 participants (mean age 61.9 years, 30% female). External validation was performed on an independent cohort of 3,771 observations from 1,976 participants (mean age 58.4 years, 59% female). Our multimodal model uses a transformer-based TabPFN to predict ALM from 75 features, which integrate 18 features extracted from each of the frontal and lateral chest radiographs by TorchXray with 39 features from EHR data (demographics and blood tests). Model performance for ALM estimation was evaluated using the root mean square error (RMSE), and mean absolute error (MAE), and Pearson correlation coefficient. Estimation of appendicular lean mass performance was assessed using the area under the receiver operating characteristic curve (AUROC). Results The prevalence of low muscle mass was 30% in the derivation set and 29% in the external validation set. The multimodal model achieved high accuracy in ALM estimation, with an RMSE of 1.23 kg, MAE of 0.96 kg, and correlation coefficient of 0.958 in the internal test set. In external validation, the model yielded an RMSE of 1.87 kg, and MAE of 1.41 kg, and correlation of 0.930. For estimation of appendicular lean mass in the external validation set, the model yielded an AUROC of 0.825 for males and 0.730 for females. Sensitivity analyses demonstrated the model's robustness to missing modalities, maintaining stable prediction errors in participants missing part of the EHR (RMSE: 2.00 kg), frontal (RMSE: 1.99 kg), or lateral (RMSE: 2.08 kg) chest radiographs. Conclusions Our transformer-based multimodal AI model accurately estimates ALM and detects low muscle mass from routinely collected clinical data, outperforming unimodal approaches. The model demonstrated robustness even with missing data modalities, supporting its potential utility as a screening tool in clinical settings where data completeness varies. This approach has the potential to serve as an accessible screening tool for low muscle mass, especially in settings where DXA is not readily available.