Metrics for Evaluating Biological AI Model Prediction Accuracy at the Data-Substrate Level
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Summary
Reports in the biological literature disagree on whether a given model can predict a biological outcome from a given data sample — one study finding a model capable, another, on the same kind of data, finding it is not. This is particularly a challenge in relation to LLMs–where the models are large and opaque, with weights and training data inaccessible. Such disagreements cannot be settled by directly inspecting the model. To address this challenge, we consider an alternative approach: assessing whether the data sample is adequate to support the prediction asserted. For a given dataset, its substrate — the underlying structure of the data — determines what any model can recover, independent of architecture or capacity. At the same time, predicting the present state of a biological process and predicting the direction of its future change are different tasks; the second is supportable among AI models only where the data encode direction as determinable from the state — a property we call encoding — and is unsupportable where the same observed state precedes change in opposite directions — a property we call non-identifiability, in the informational rather than the statistical sense. We introduce two generic metrics, Predictive Blindness Risk (PBR) and Prediction Indeterminacy Measure (PIM), that evaluate a data substrate for predictive accuracy directly — without access to model weights, architecture, or training data — and locate the regions of a data substrate where a predictive claim can be supported and where it cannot. Using human biological subjects, we employ the Yale Brain Metastases Longitudinal Data (1,430 human subjects; 11,892 MRI studies; four sequences) and show that direction of change was non-identifiable across regions encompassing the majority of transitions; a nonlinear AI model gained essentially nothing over majority-direction prediction there while recovering direction near-perfectly where the state encoded it; and model accuracy tracked data-substrate resolvability continuously (Spearman ρ = −0.95 to −1.00). The metrics adjudicate, before any model is trusted and from the data alone, where claims of predictive accuracy — of state, or of the law of change — can be supported.
Background
Model prediction of a biological outcome from a given data sample is often highly contested — particularly in predicting biological change. The literature reports conflicting capability for the same kind of data and when complex models or opaque LLMs are used, the disagreement cannot be resolved by inspecting the model or its benchmark training data. Such disagreement has two sources — instability of the model, and indeterminacy of the data — of which only the first has been studied. We address the second by relocating the question from the model to the data substrate.
Methods
We analysed human biological data taken from the Yale Brain Metastases Longitudinal Data (1,430 human subjects; 11,892 MRI studies; four sequences) using transitions between consecutive scans. An observed-measure proxy grouped subjects into matched observed states. We defined two data-substrate metrics — Predictive Blindness Risk (PBR), the minority-direction fraction at a matched state, and Prediction Indeterminacy Measure (PIM), the separation between admissible trajectories at a state — and evaluated state-only prediction with subject-level-split logistic regression and a multilayer perceptron against a majority-direction baseline. At bin level across all eight bins per sequence, we computed the Spearman correlation between PBR and held-out model accuracy. Formal results established when a single-valued predictor cannot recover direction at a shared state (Appendix); a controlled simulation with a known governing law served as comparison.
Results
Across sequences, 6–7 of 8 matched-state bins contained transitions in both directions, encompassing 75–88% of transitions; weighted PBR ranged 0.30–0.35 (mean 0.33, SD 0.01). In these non-identifiable bins the nonlinear model did not exceed majority-direction prediction (model−majority −0.014 to +0.001 across sequences and splits); in the complementary encoding bins (12–25% of transitions) accuracy reached 0.958–1.000. Across all bins, model directional accuracy declined monotonically with PBR (Spearman ρ = −0.952 FLAIR, −0.976 T1 post-contrast, −1.000 T1 pre-contrast, −0.976 T2; P ≤ 3.3×10 −5 in three sequences, 2.6×10 −4 in FLAIR), establishing a continuous gradient of data-substrate resolvability rather than a binary partition. PBR was stable across binning resolution (range 0.299–0.353; CV 5.4%). In the simulation the state-only model predicted the wrong direction for every minority-direction subject while retaining significant overall association ( ρ = 0.46); theoretical PBR tracked empirical failure ( ρ = 0.96).
Conclusions
State-only prediction recovers directional structure where the data substrate encodes it and cannot where the substrate is non-identifiable, with model accuracy varying continuously between the two as a near-deterministic function of data-substrate resolvability. The contested question of model predictive capability is therefore, in part, a prior question about the data: PBR and PIM answer it from the data alone, without access to model weights, architecture, or training data, and discriminate the regions of a data substrate where a claim of predictive accuracy — of state, or of the law of change — is and is not supportable.
