The replicability of real-world evidence for the surgical treatment of proximal humeral fractures in older patients evaluated using sceptical p-value

Background: The replicability of study results plays a central role, particularly in the context of real-world evidence (RWE). However, the evaluation of the agreement between two study results is not straightforward and different metrics are available. As an example for RWE, the surgical treatment of proximal humeral fractures in older patients using locked plate fixation (LPF) or reverse total shoulder arthroplasty (RTSA) was analyzed by two different studies based on the data of the largest German health insurance company [1, 2], but the generalizability of the results should be further addressed. Methods: Independent data of a different German insurance company were used to emulate two existing studies for the surgical treatment of proximal humeral fracture in older patients using RTSA vs LPF. Study protocol was a priori registered. The sceptical p-value was used to judge the agreement between known original effect estimate (OES) and the observed emulated treatment effect estimate (EES) and compared to other existing binary agreement metrics. Results: Overall, 7 (64%) of the 11 evaluated endpoints were successfully replicated. The Pearson correlation coefficient (rp = 0.97) and the intra-class correlation coefficient (ICC3 = 0.94) indicated a good, positive correlation between original and emulation study. OES was successfully replicated for all short-term endpoints and surgical complications during follow-up (all pscep < 0.025). Conclusion: All treatment effects of the original study for short-term endpoints and for surgical complications during follow-up were successfully replicated using an independent data base. The sceptical p value was a useful and well-interpretable metric to evaluate the agreement between original and emulation study. Trial registration: A priori registered at ClinicalTrials.gov (Identifier: NCT06537024). Level of Evidence: Level III, retrospective replication study.