The influence of study selection criteria on statistical conclusion validity: A simulation study

Tier 2 interventions are designed for students who are not making sufficient progress despite receiving whole-group instruction. These interventions target a specific population, necessitating clear eligibility criteria for screening participants. While efficacy studies of Tier 2 interventions often generalize their findings to struggling readers, the eligibility criteria for selecting study samples can vary across studies. In our simulation study, we compared effect size estimates from different samples, each screened using commonly used eligibility criteria. Despite each method accurately recovering the true effect on average across 500 iterations, the range of estimates showed substantial variation, with significant under- and overestimation in individual iterations. We concluded that the perceived effectiveness of Tier 2 interventions may fluctuate based on how the sample is screened for eligibility. We propose future research to address this methodological challenge while balancing the practical demands of Tier 2 intervention studies.