Measurement Reliability Confounds Cross-Regime Comparisons of Postprandial Glucose Prediction: Evidence from Bidirectional Cross-Cohort Transfer
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (PREreview)
Abstract
Postprandial glucose response (PPGR) prediction is a central promise of personalized nutrition, yet published models are rarely evaluated against the reliability of the measurement they try to predict. We reproduce and extend the CGMacros baseline (45 subjects, 1,699 meals; leave-one-subject-out) and subject it to a cross-cohort stress test on BIG IDEAs (16 subjects, 656 meals). The replication matches the published breakfast result (iAUC r = 0.655 against approx. 0.64) and extends it to all analysable meals (iAUC 0.451; AUC 0.838). Duplicate reference breakfasts in CGMacros give a repeatability of rho = 0.799, a ceiling the model stays below (sqrt(rho) approx. 0.894) and so does the published baseline. A frozen model transfers forward at r = 0.227 for iAUC; trained on the external cohort and transferred back it reaches 0.085, so iAUC transfer fails in both directions. AUC behaves differently: it survives transfer (0.569 forward, 0.748 reverse) and is largely a baseline-glucose quantity - pre-prandial glucose alone recovers 0.807 of AUC across all meals, 0.861 at breakfast, and 0.963 at the person level. Read against its own regime, the within-cohort breakfast model captures 0.54 of the reliable variance, a same-device cross-population transfer in a prior study also 0.54, and the cross-device transfer 0.11. Cross-regime comparisons of PPGR models are therefore confounded by measurement reliability and cannot be read as comparisons of model quality: reliability explains part of the gap, and the remainder is attributable to features, device, annotation and model class in proportions this design cannot separate. The claim is conditional and falsifiable, and we state the study that would overturn it. We recommend that PPGR reports name their measurement regime, place a reliability estimate beside every correlation, and report per-subject alongside marginal interval coverage.
Article activity feed
-
This Zenodo record is a permanently preserved version of a PREreview. You can view the complete PREreview at https://prereview.org/reviews/24389074.
The study, Measurement Reliability Confounds Cross-Regime Comparisons of Postprandial Glucose Prediction, investigates how measurement reliability influences the performance and generalizability of machine-learning models predicting postprandial glucose responses (PPGR).
Using two independent continuous glucose monitoring (CGM) datasets (CGMacros and BIG IDEAs), the authors reproduce an existing prediction model, evaluate its performance within each cohort, and conduct bidirectional cross-cohort validation.
The key findings are:
1. Measurement reliability strongly influences prediction performance. Duplicate-meal analysis revealed substantial differences in repeatability depending on …
This Zenodo record is a permanently preserved version of a PREreview. You can view the complete PREreview at https://prereview.org/reviews/24389074.
The study, Measurement Reliability Confounds Cross-Regime Comparisons of Postprandial Glucose Prediction, investigates how measurement reliability influences the performance and generalizability of machine-learning models predicting postprandial glucose responses (PPGR).
Using two independent continuous glucose monitoring (CGM) datasets (CGMacros and BIG IDEAs), the authors reproduce an existing prediction model, evaluate its performance within each cohort, and conduct bidirectional cross-cohort validation.
The key findings are:
1. Measurement reliability strongly influences prediction performance. Duplicate-meal analysis revealed substantial differences in repeatability depending on meal type, with reference breakfasts showing higher reliability (r = 0.799) than free-choice dinners (r = 0.374) and snacks (r = 0.376).
2. Poor cross-cohort generalizability. The model showed substantially reduced performance when transferred between cohorts. Prediction of incremental glucose response (iAUC) declined to r = 0.227 in forward transfer and r = 0.085 in reverse transfer, suggesting limited portability across different measurement and data-collection regimes.
3. AUC can overestimate predictive performance. The authors demonstrate that total area under the glucose curve (AUC) is strongly influenced by pre-meal glucose levels. Consequently, AUC may reflect baseline glycemic status rather than the actual meal-induced glucose excursion. iAUC provides a more appropriate measure for evaluating meal-specific glucose responses.
4. Aggregation does not necessarily indicate better meal-level prediction. Although averaging predictions at the individual level improved within-cohort correlations, it did not rescue cross-cohort iAUC prediction.
Contribution to the field: This work highlights an important but frequently overlooked limitation in personalized nutrition research: differences in measurement reliability can confound comparisons of predictive model performance across studies and populations. It emphasizes the need to report measurement reliability alongside prediction metrics, distinguish meal-level from person-level prediction, and evaluate models under comparable measurement conditions. These findings provide useful methodological guidance for developing and validating more reliable and generalizable PPGR prediction.
Major issues
1. Measurement reliability is not established in both cohorts. The study estimates repeatability using CGMacros but does not measure it directly in BIG IDEAs. Consequently, the central argument that reliability explains cross-cohort performance differences remains partly inferential rather than experimentally demonstrated.
2. Multiple confounding factors limit interpretation. Device type, meal annotation methods, and participant characteristics differ simultaneously between the two cohorts. The study cannot independently determine how much each factor contributes to the observed decline in prediction performance.
3. Limited sample size and external validation. BIG IDEAs includes only 16 participants. Although bidirectional validation strengthens the analysis, the relatively small number of individuals limits statistical power and the generalizability of the conclusions.
4. Limited evidence for the proposed reliability framework. The comparison with previous studies is informative, but the reliability estimates come from different populations and experimental protocols. In particular, the use of an external device-level reliability estimate as a proxy for BIG IDEAs introduces uncertainty into the comparison.
5. Clinical applicability remains uncertain. The study demonstrates limitations in meal-level prediction and reports substantial variability in prediction intervals across individuals. However, it does not establish whether the proposed framework or prediction results can improve personalized dietary recommendations or clinical decision-making.
Minor issues
1. Clarify terminology and abbreviations. The distinction between AUC, iAUC, peak glucose rise, and measurement reliability could be explained more consistently, particularly for readers less familiar with CGM-based glucose prediction.
2. Improve presentation of statistical uncertainty. Confidence intervals should be reported consistently alongside key performance metrics, especially for cross-cohort correlations and reliability estimates.
3. Improve visualization of cohort differences. A concise comparison table highlighting differences in participant characteristics, CGM devices, sampling frequencies, and meal annotation protocols would improve readability.
4. Expand discussion of alternative explanations. The discussion could more clearly distinguish the effects of measurement reliability from differences in population characteristics, feature distributions, model architecture, and meal-recording accuracy.
5. Improve the discussion of practical implications. The recommendations for reporting reliability and evaluating individual-level prediction intervals are valuable, but more specific guidance on implementing these recommendations in future PPGR studies would strengthen the manuscript.
Competing interests
The author declares that they have no competing interests.
Use of Artificial Intelligence (AI)
The author declares that they did not use generative AI to come up with new ideas for their review.