Derivation and external validation of clinical prediction rules identifying children at risk of linear growth faltering
Curation statements for this article:-
Curated by eLife
eLife assessment
This work would be of interest to global health scientists, particularly in low- and middle-income countries where childhood stunting is an ongoing challenge, and to statisticians interested in building clinical prediction rules. The authors leveraged large, rich datasets from multi-center studies to build and validate predictive models. But by using change in growth, rather than absolute growth, as the only outcome, it may be missing children of concern who are already experiencing growth failure and require intervention but have reached a growth faltering floor.
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (eLife)
Abstract
Nearly 150 million children under-5 years of age were stunted in 2020. We aimed to develop a clinical prediction rule (CPR) to identify children likely to experience additional stunting following acute diarrhea, to enable targeted approaches to prevent this irreversible outcome.
Methods:
We used clinical and demographic data from the Global Enteric Multicenter Study (GEMS) to build predictive models of linear growth faltering (decrease of ≥0.5 or ≥1.0 in height-for-age z -score [HAZ] at 60-day follow-up) in children ≤59 months presenting with moderate-to-severe diarrhea, and community controls, in Africa and Asia. We screened variables using random forests, and assessed predictive performance with random forest regression and logistic regression using fivefold cross-validation. We used the Etiology, Risk Factors, and Interactions of Enteric Infections and Malnutrition and the Consequences for Child Health and Development (MAL-ED) study to (1) re-derive, and (2) externally validate our GEMS-derived CPR.
Results:
Of 7639 children in GEMS, 1744 (22.8%) experienced severe growth faltering (≥0.5 decrease in HAZ). In MAL-ED, we analyzed 5683 diarrhea episodes from 1322 children, of which 961 (16.9%) episodes experienced severe growth faltering. Top predictors of growth faltering in GEMS were: age, HAZ at enrollment, respiratory rate, temperature, and number of people living in the household. The maximum area under the curve (AUC) was 0.75 (95% confidence interval [CI]: 0.75, 0.75) with 20 predictors, while 2 predictors yielded an AUC of 0.71 (95% CI: 0.71, 0.72). Results were similar in the MAL-ED re-derivation. A 2-variable CPR derived from children 0–23 months in GEMS had an AUC = 0.63 (95% CI: 0.62, 0.65), and AUC = 0.68 (95% CI: 0.63, 0.74) when externally validated in MAL-ED.
Conclusions:
Our findings indicate that use of prediction rules could help identify children at risk of poor outcomes after an episode of diarrheal illness. They may also be generalizable to all children, regardless of diarrhea status.
Funding:
This work was supported by the National Institutes of Health under Ruth L. Kirschstein National Research Service Award NIH T32AI055434 and by the National Institute of Allergy and Infectious Diseases (R01AI135114).
Article activity feed
-
-
Author Response
Public Evaluation Summary:
This work would be of interest to global health scientists, particularly in low- and middleincome countries where childhood stunting is an ongoing challenge, and to statisticians interested in building clinical prediction rules. The authors leveraged large, rich datasets from multi-center studies to build and validate predictive models. But by using change in growth, rather than absolute growth, as the only outcome, it may be missing children of concern who are already experiencing growth failure and require intervention but have reached a growth faltering floor.
Thank you for this suggestion. We have added additional models for the following predictions: a) growth faltering in those NOT stunted (HAZ≥-2) at presentation, b) any stunting (HAZ<-2) at follow-up, and c) any stunting at …
Author Response
Public Evaluation Summary:
This work would be of interest to global health scientists, particularly in low- and middleincome countries where childhood stunting is an ongoing challenge, and to statisticians interested in building clinical prediction rules. The authors leveraged large, rich datasets from multi-center studies to build and validate predictive models. But by using change in growth, rather than absolute growth, as the only outcome, it may be missing children of concern who are already experiencing growth failure and require intervention but have reached a growth faltering floor.
Thank you for this suggestion. We have added additional models for the following predictions: a) growth faltering in those NOT stunted (HAZ≥-2) at presentation, b) any stunting (HAZ<-2) at follow-up, and c) any stunting at follow-up in those not stunted at presentation. While we agree the addition of these models improves the manuscript, we also want to highlight that these models have distinct outcomes and therefore have separate clinical uses. Our original goal was to identify children whose growth was likely to slow down after diarrhea. As we show, top predictors and predictive performance is similar for growth faltering across baseline stunting status. We present any stunting at follow-up as a comparison, but argue that this is a different clinical outcome that may warrant different intervention. We have edited the manuscript for clarity as follows.
P.22 L339-343: . In sensitivity analyses, we demonstrated our ability to predict any stunting at follow-up with high accuracy (Table 1, Table S5). However, this represents a related but distinct outcome from our original aim, namely a slowing down of growth as opposed to stunting, and may warrant different clinical intervention.
P.23 L.353-357: Current malnutrition recommendations are based on patient presentation – whether a child is underweight when they present to the clinic. Our CPR could be used to identify children not currently stunted and therefore not currently recommended for nutritional interventions, but who are likely to slow down in growth and therefore at higher risk of incident stunting.
P.23 L352-361: Our CPR provides a tool for identifying patients likely to experience additional growth faltering after acute diarrhea. Current malnutrition recommendations are based on patient presentation – is a child underweight when they come to the clinic. Our CPR could be used to identify children not currently stunted and therefore not currently recommended for nutritional interventions, but who are likely to slow down in growth and therefore at higher risk of incident stunting. Identifying these children would allow clinicians to connect patients with community-based nutrition interventions (e.g. maternal support for safe introduction of weening foods, small quantity lipid nutrient supplements (SQ-LNS), etc.(45-48)) to prevent additional effects of chronic malnutrition, namely irreversible stunting.
P.25 L.390-393: Our findings indicate that use of prediction rules, potentially applied as clinical decision support tools, could help to identify additional children at risk of poor outcomes after an episode of diarrheal illness, i.e. not currently stunted but likely to decelerate growth.
Reviewer #1 (Public Review):
In this manuscript, the authors built logistic regression prediction models for linear growth faltering using demographic, socioeconomic, and clinical variables, with the objective of developing a clinical prediction rule that could be applied by healthcare workers to identify and treat high-risk children. A model with 2 variables selected by random forest variable importance performed similarly to a model with 10 variables. Age and HAZ at baseline were selected for the 2-variable model, consistent with existing literature. The authors externally validated the 2variable model and found similar discriminative ability. Based on typical rule-of-thumb cutoffs, model performance was moderate (AUCs of ~0.65-0.75, depending on model specification); models may still be useful in practice, but this should be further discussed by the authors.
We agree that our overall ability to predict growth faltering was moderate. As we present in-depth below, we do not intend for our clinical prediction rule (CPR) to replace existing guidelines. Therefore, we are not proposing that our CPR be used to withhold nutritional treatment. Rather, we intend for our CPR to be used in conjunction with existing clinical practices to identify additional children who may or may not be currently stunted, but at are increased risk of decelerated growth and therefore would also benefit from nutritional interventions.
Strengths:
Linear growth faltering is a pressing issue with broad, negative impacts on the health, development, and well-being of children worldwide. In this work, the authors applied clearly explained, thoughtful approaches to variable selection, model specification, and model validation, with large, multi-country cohorts used for training and external validation. Appropriate datasets for external validation can be challenging to find, but the MAL-ED data used here is well-suited to the task, with similar predictor and outcome measurements to the GEMS training data. The well-characterized studies allowed the authors to explore a wide range of potential predictors for stunting, including socioeconomic factors, antibiotic use, and diarrheal etiology.
Weaknesses:
This work would benefit from additional discussion around the clinical relevance of the results. For example, what is the current standard of care for prevention of stunting, and how much would this model improve the status quo? Is specificity of 0.47 in the context of sensitivity of 0.80 an acceptable tradeoff with regards to the interventions that would be used? More discussion around these points is necessary to support the authors' conclusions that these models could potentially be used to support clinical decisions and target resources.
Current practice focuses on the identification and treatment of malnutrition, with malnutrition classified based on mid-upper arm circumference (MUAC), weight for length or height z-score, or bipedal oedema. None of these measurements compare child size to their age. At the International Centre for Diarrhoeal Research, Bangladesh (ICDDRB), children are only evaluated for stunting if their weight for age z-score is too low. While stunting can be the result of chronic malnutrition, it can also be a contributing factor to future health problems (see first paragraph of Introduction). Therefore, while related to malnutrition, stunting is a distinct health outcome that would benefit from explicit identification strategies. Furthermore, current practice only identifies children who are already stunted when they present to care. A CPR to identify children whose growth is likely to slow down and therefore who are at risk of new or additional stunting could help prevent additional stunting and its downstream health outcome. The Discussion now includes the following:
P.23 L.353-361: Current malnutrition recommendations are based on patient presentation – whether a child is underweight when they present to the clinic. Our CPR could be used to identify children not currently stunted and therefore not currently recommended for nutritional interventions, but who are likely to slow down in growth and therefore at higher risk of incident stunting. Identifying these children would allow clinicians to connect patients with communitybased nutrition interventions (e.g. maternal support for safe introduction of weening foods, small quantity lipid nutrient supplements (SQ-LNS), etc.(46-49)) to prevent additional effects of chronic malnutrition, namely irreversible stunting.
In addition to the external validation, further investigation of model performance in key subpopulations would strengthen the importance and applicability of the work. For example, performance of prediction models may vary widely by setting; it would be valuable to show that the model has similar performance in each country. Another key sensitivity analysis would be to show consistent model performance by HAZ at baseline. The authors note that stunting may be challenging to reverse (p.20), and many of the children are already below the typical cutoff of HAZ<-2 at baseline; it would be valuable to show model performance among the subgroup of children for whom treatment would be most beneficial.
We appreciate this suggestion. We have added additional analysis regarding stunting at baseline as described above. We have added country-specific CPRs in the Supplement. We have also added a sensitivity analysis whereby we fit models to all data from one continent in GEMS, and then validated that model on the other continent in GEMS data. As you can see from Supplementary Table S5, top predictors and discriminative performance were similar across countries and continents
P.10 L.171-173: Finally, we conducted a quasi-external validation within the GEMS data by fitting a model to one continent and validating it on the other.
P.24 L.380-383: The quasi-external validation between continents within GEMS data, as well as the country-specific models within GEMS, all had similar top predictors and discriminative performance, further supporting the overall validity of our CPR. Finally, we explored a range of AFe cutoffs for etiology, with consistent results.
Reviewer #2 (Public Review):
The manuscript documents a thorough and well-validated clinical prediction model for risk of severe child linear growth faltering after diarrheal disease episodes, using data from multiple studies and countries. They identified a parsimonious model of child age and current size with relatively good predictive accuracy. However, I don't believe the prediction rule should be used in it's current form due to the outcome used the danger of missing treating children who require nutritional supplementation.
As described in-depth above, we do not intend for this CPR to replace existing guidelines, but rather to function as a complementary tool to identify additional children not currently stunted but who are at risk of their growth slowing down.
The outcome used for prediction in a binary indicatory for a decrease in height-for-age Z-score >= 0.5. A child who fails to gain height by future measurements is of concern, but this outcome also misses children who are already experiencing growth failure, and is vulnerable to regression to the mean effect. The two most important predictors were age and current size, with current size having a positive association with risk of growth faltering. As mentioned in the discussion, there is "the possibility that children need to have high enough HAZ in order to have the potential to falter." Additionally, there may be children with erroneously high height measurements at the first measurement, so that the HAZ change >= 0.5 associated with high baseline HAZ is from measurement-error regression to the mean. I recommend also predicting absolute HAZ (or stunting status) as a secondary outcome and comparing if the important predictors change.
See above.
In its current form, the results and conclusions from the results have problematic implications for the treatment of child malnutrition. The conclusion states: "In settings with high mortality and morbidity in early childhood, such tools could represent a cost-effective way to target resources towards those who need it most." If the current CPR was used in a resourceconstrained setting, it would recommend that larger children should be prioritized for nutritional supplementation over already stunted children who may have reached their growth faltering floor. In addition, with a sensitivity of 80%, the tool would miss treating a large number of children who would experience growth faltering. The results of the clinical prediction tool need to be presented with care in how it could be used to prioritize treatment without missing treating children who would benefit from nutritional supplementation. Including absolute HAZ as an outcome will help, along with additional discussion of how the CPR fits alongside current treatment recommendations. For example, does this rule indicate treating children who aren't currently treated, or are there children who don't need treatment given current guidelines and the created CPR.
We thank the Reviewers for pointing out this oversight. We have edited the Discussion for clarity as follows.
P.23 L.352-361: Our CPR provides a tool for identifying patients likely to experience additional growth faltering after acute diarrhea. Current malnutrition recommendations are based on patient presentation – is a child underweight when they come to the clinic. Our CPR could be used to identify children not currently stunted and therefore not currently recommended for nutritional interventions, but who are likely to slow down in growth and therefore at higher risk of incident stunting. Identifying these children would allow clinicians to connect patients with community-based nutrition interventions (e.g. maternal support for safe introduction of weening foods, small quantity lipid nutrient supplements (SQ-LNS), etc.(45-48)) to prevent additional effects of chronic malnutrition, namely irreversible stunting.
P.25 L.390-393: Our findings indicate that use of prediction rules, potentially applied as clinical decision support tools, could help to identify additional children at risk of poor outcomes after an episode of diarrheal illness, i.e. not currently stunted but likely to decelerate growth.
In sum, this is a thorough, well done, clearly explained exercise in creating a clinical prediction tool for predicting child risk of future growth faltering. The writing and motivation is clear, and the methods have applicability far beyond the specific use-case.
-
eLife assessment
This work would be of interest to global health scientists, particularly in low- and middle-income countries where childhood stunting is an ongoing challenge, and to statisticians interested in building clinical prediction rules. The authors leveraged large, rich datasets from multi-center studies to build and validate predictive models. But by using change in growth, rather than absolute growth, as the only outcome, it may be missing children of concern who are already experiencing growth failure and require intervention but have reached a growth faltering floor.
-
Reviewer #1 (Public Review):
In this manuscript, the authors built logistic regression prediction models for linear growth faltering using demographic, socioeconomic, and clinical variables, with the objective of developing a clinical prediction rule that could be applied by healthcare workers to identify and treat high-risk children. A model with 2 variables selected by random forest variable importance performed similarly to a model with 10 variables. Age and HAZ at baseline were selected for the 2-variable model, consistent with existing literature. The authors externally validated the 2-variable model and found similar discriminative ability. Based on typical rule-of-thumb cutoffs, model performance was moderate (AUCs of ~0.65-0.75, depending on model specification); models may still be useful in practice, but this should be further …
Reviewer #1 (Public Review):
In this manuscript, the authors built logistic regression prediction models for linear growth faltering using demographic, socioeconomic, and clinical variables, with the objective of developing a clinical prediction rule that could be applied by healthcare workers to identify and treat high-risk children. A model with 2 variables selected by random forest variable importance performed similarly to a model with 10 variables. Age and HAZ at baseline were selected for the 2-variable model, consistent with existing literature. The authors externally validated the 2-variable model and found similar discriminative ability. Based on typical rule-of-thumb cutoffs, model performance was moderate (AUCs of ~0.65-0.75, depending on model specification); models may still be useful in practice, but this should be further discussed by the authors.
Strengths:
Linear growth faltering is a pressing issue with broad, negative impacts on the health, development, and well-being of children worldwide. In this work, the authors applied clearly explained, thoughtful approaches to variable selection, model specification, and model validation, with large, multi-country cohorts used for training and external validation. Appropriate datasets for external validation can be challenging to find, but the MAL-ED data used here is well-suited to the task, with similar predictor and outcome measurements to the GEMS training data. The well-characterized studies allowed the authors to explore a wide range of potential predictors for stunting, including socioeconomic factors, antibiotic use, and diarrheal etiology.
Weaknesses:
This work would benefit from additional discussion around the clinical relevance of the results. For example, what is the current standard of care for prevention of stunting, and how much would this model improve the status quo? Is specificity of 0.47 in the context of sensitivity of 0.80 an acceptable tradeoff with regards to the interventions that would be used? More discussion around these points is necessary to support the authors' conclusions that these models could potentially be used to support clinical decisions and target resources.
In addition to the external validation, further investigation of model performance in key subpopulations would strengthen the importance and applicability of the work. For example, performance of prediction models may vary widely by setting; it would be valuable to show that the model has similar performance in each country. Another key sensitivity analysis would be to show consistent model performance by HAZ at baseline. The authors note that stunting may be challenging to reverse (p.20), and many of the children are already below the typical cutoff of HAZ<-2 at baseline; it would be valuable to show model performance among the subgroup of children for whom treatment would be most beneficial.
-
Reviewer #2 (Public Review):
The manuscript documents a thorough and well-validated clinical prediction model for risk of severe child linear growth faltering after diarrheal disease episodes, using data from multiple studies and countries. They identified a parsimonious model of child age and current size with relatively good predictive accuracy. However, I don't believe the prediction rule should be used in it's current form due to the outcome used the danger of missing treating children who require nutritional supplementation.
The outcome used for prediction in a binary indicatory for a decrease in height-for-age Z-score >= 0.5. A child who fails to gain height by future measurements is of concern, but this outcome also misses children who are already experiencing growth failure, and is vulnerable to regression to the mean effect. …
Reviewer #2 (Public Review):
The manuscript documents a thorough and well-validated clinical prediction model for risk of severe child linear growth faltering after diarrheal disease episodes, using data from multiple studies and countries. They identified a parsimonious model of child age and current size with relatively good predictive accuracy. However, I don't believe the prediction rule should be used in it's current form due to the outcome used the danger of missing treating children who require nutritional supplementation.
The outcome used for prediction in a binary indicatory for a decrease in height-for-age Z-score >= 0.5. A child who fails to gain height by future measurements is of concern, but this outcome also misses children who are already experiencing growth failure, and is vulnerable to regression to the mean effect. The two most important predictors were age and current size, with current size having a positive association with risk of growth faltering. As mentioned in the discussion, there is "the possibility that children need to have high enough HAZ in order to have the potential to falter." Additionally, there may be children with erroneously high height measurements at the first measurement, so that the HAZ change >= 0.5 associated with high baseline HAZ is from measurement-error regression to the mean. I recommend also predicting absolute HAZ (or stunting status) as a secondary outcome and comparing if the important predictors change.
In its current form, the results and conclusions from the results have problematic implications for the treatment of child malnutrition. The conclusion states: "In settings with high mortality and morbidity in early childhood, such tools could represent a cost-effective way to target resources towards those who need it most." If the current CPR was used in a resource-constrained setting, it would recommend that larger children should be prioritized for nutritional supplementation over already stunted children who may have reached their growth faltering floor. In addition, with a sensitivity of 80%, the tool would miss treating a large number of children who would experience growth faltering. The results of the clinical prediction tool need to be presented with care in how it could be used to prioritize treatment without missing treating children who would benefit from nutritional supplementation. Including absolute HAZ as an outcome will help, along with additional discussion of how the CPR fits alongside current treatment recommendations. For example, does this rule indicate treating children who aren't currently treated, or are there children who don't need treatment given current guidelines and the created CPR.
In sum, this is a thorough, well done, clearly explained exercise in creating a clinical prediction tool for predicting child risk of future growth faltering. The writing and motivation is clear, and the methods have applicability far beyond the specific use-case.
-
