Comparative Analysis of Clinical Finding Retrieval Difficulty in Chest X-ray Retrieval Models

  1. Areesha farid
  2. Hifsa Farooq

Reviewed by

Read the full article

Listed in

Log in to save this article

Abstract

Sentence-level retrieval generates chest X-ray (CXR) reports by selecting clinical sentences from a predefined pool. Retrieval performance depends on the composition of this pool, yet aggregate metrics can conceal differences between clinical findings. This study examined whether finding-level retrieval difficulty reflects model limitations or retrieval pool composition. Three sentence-level models (a JoImTeRNet-based model developed by the authors, CXR-RePaiR and CXR-ReDonE) retrieved the top 10 sentences for 10,000 frontal CheXpert images. Sentences were drawn from an original pool of 129,906 MIMIC-CXR sentences, 30 balanced pools of approximately 6,200 sentences each (up to 450 positive sentences per finding, drawn with different random seeds) and 50 random pools of equal size. Sentences were labeled with the CheXbert labeler. Micro-averaged recall, precision and F1, finding-level recall at K=5 and cross-model Spearman correlation were evaluated. Balancing raised micro-recall for every model at every K and finding-level recall for 13 of 14 findings, whereas No Finding recall fell by 0.119 to 0.187 (mean of 30 balanced pools). For 36 of 42 finding-model combinations the change exceeded that of random pools in at least half of the balanced pools, and at K=5 balanced-pool recall exceeded the random-pool mean by 0.07 to 0.08. Precision decreased and F1 varied. Cross-model agreement across 13 findings declined from ρ=0.687–0.808 (original pool) to ρ=0.153–0.577 (balanced pools), while random pools gave ρ=0.712–0.788; with only 13 findings, this decrease should be regarded as descriptive. Lung Lesion remained among the five lowest-recall findings for all three models under the original pool and in all 30 balanced pools. Retrieval pool composition influences finding-level recall and apparent difficulty in CXR sentence retrieval, and part of the difficulty observed under the full pool is pool-dependent. Evaluations should report pool composition and finding-level results alongside aggregate scores.

Article activity feed

  1. Version published to 10.21203/rs.3.rs-11003136/v3 on Research Square
  2. Version published to 10.21203/rs.3.rs-11003136/v2 on Research Square
  3. PREreview

    This Zenodo record is a permanently preserved version of a PREreview. You can view the complete PREreview at https://prereview.org/reviews/22788887.

    In order to evaluate the impact of the candidate-sentence pool on the retrieval of clinical findings for chest X-ray images, we investigate three different sentence-retrieval models, namely the proposed JoImTeRNet model as well as CXR-RePaiR and CXR-ReDonE. We assess these models under two candidate-sentence pool conditions: an original, unbalanced pool of 129,906 sentences and a balanced pool of 6,216 sentences, capped at 450 positive sentences per finding. Our results show that candidate-pool composition strongly affects both finding-level retrieval performance and cross-model agreement on which findings are "hard" to retrieve. Cross-model agreement was strong for the original pool of sentences (ρ=0.749–0.807), but dropped for the balanced pool of sentences (ρ=0.240–0.495). Lung Opacity and Pleural Effusion were consistently the two hardest findings to retrieve, whereas Support Devices was consistently the easiest.

    Field contribution

    It is novel to empirically demonstrate that the apparent "consensus" between independently trained retrieval models in a multi-model setup does not necessarily stem from the models themselves, but rather may be influenced by the design of the evaluation pool.

    Major issues

    • Confounded variables: The authors note that the "balanced" pool differs from the original pool in terms of size, class balance, and even the included sentences. However, rather than subsampling to a fixed size, the authors use a substantially smaller balanced pool. An equal-size random subsample would be needed to isolate the effect of balancing from the effect of pool size.

    • Possible bias favoring the proposed model: The sentence pool for evaluation was selected from the MIMIC-CXR database used to train the proposed model, while CXR-ReDonE was trained on the stylistically different MIMIC-PRO. This could confer an advantage unrelated to genuine retrieval quality.

    • Evaluation labels not expert-verified: The evaluation set is a random sample from the CheXpert training set, not the expert-adjudicated validation set. Moreover, the ground truth was generated by an automated labeler (CheXbert), hence introducing label noise.

    • Small number of findings (14) for correlation analysis: With Spearman's rank correlation based on only 14 findings, estimates will inherently be unstable, even with the use of bootstrap confidence intervals.

    • Speculative explanation for Lung Opacity's difficulty: The hypothesis in the Figure that the findings' semantic/definitional ambiguity contributes to the difficulty of retrieving Lung Opacity is plausible but was never directly tested.

    Minor issues

    • Some redundancy between the abstract, results summary, and conclusion.

    • The "Related Work" section is rather long in comparison to the content of the paper; it could be shortened.

    • Some of the tables, especially Table 3, contain a lot of numbers and therefore would benefit from a visual summary alongside Figure 1.

    • Justification for the specific number of 450 positive sentences for the balancing cap is not clear.

    • Reference [21] is listed as a source but is not used in the paper.

    Competing interests

    The author declares that they have no competing interests.

    Use of Artificial Intelligence (AI)

    The author declares that they did not use generative AI to come up with new ideas for their review.

  4. Version published to 10.21203/rs.3.rs-11003136/v1 on Research Square