The Dark Side of Sequential Testing: A Simulation Study on Questionable Research Practices

In response to growing replication failures attributed to underpowered studies and questionable research practices (QRPs) such as data peeking, researchers have increasingly turned to sequential testing frameworks -- most notably the Sequential Probability Ratio Test (SPRT). By evaluating evidence at each interim analysis, SPRTs achieve the same inferential rigor as fixed-sample protocols while requiring substantially fewer participants, all without sacrificing robustness to violations of their statistical assumptions. However, like any statistical method, SPRTs are susceptible to QRPs. We conducted a simulation study using a fictional researcher who applied various hacking strategies to favor the alternative hypothesis. These included running multiple parallel sequential ANOVAs, performing opportunistic subgroup or outlier analyses, flexibly redefining expected effect sizes, reshuffling observation order, and filtering datapoints that weakened interim likelihood ratios. Single hacking strategies and moderate combinations inflated the nominal 5% Type I error rate to 6-19 % (rising to 99 % under extreme data filtering), shifted effect-size estimates upward, increased sample size efficiency and reduced the rate of non-decisions. These findings underscore that while SPRTs offer substantial gains in efficiency, they are not immune to misuse, much like fixed-design approaches. It is therefore critical to promote transparency and preregistrations in sequential designs to prevent the adoption of QRPs early on.