Meta-analysis models relaxing the random effects normality assumption: methodological systematic review and simulation study

Background Random effects meta-analysis is widely used for synthesizing the studies of a systematic review assuming a normal between-study distribution. However, this assumption might not always be plausible. Alternative options have been suggested but not used in published meta-analyses. Methods We conducted a systematic review to identify articles that proposed alternative meta-analysis models assuming non-normal distributions for the random effects, such as skewed or semi-parametric distributions. Subsequently, we performed a simulation study to evaluate the performance of the identified models and to compare them with the normal model. We considered 22 scenarios varying the amount of between study variance, the number of included studies, and the shape of the true distribution: normal, skew-normal, and mixture of two normal distributions. For each scenario, we generated 1000 meta-analyses datasets. To investigate additional aspects of the alternative models, we also applied them at three extracted simulated datasets representing three scenarios with different true distributions. Results We identified in total 27 articles suggesting 24 alternative models that can be classified into three broad categories: models based on long-tail and skewed distributions, on mixtures of distributions, and on Dirichlet process priors (DP). We compared 15 models in our simulation study implemented in the Frequentist or Bayesian framework. Results revealed small differences in bias between the different models but larger differences in the level of coverage probability. In scenarios with large between-study variance, all models were substantially biased in the estimation of the mean treatment effect. However, mixture and semi-parametric models revealed latent underlying clustering of studies and assisted to form subgroups of common characteristics. The three simulated datasets demonstrated similar patterns with the simulation study for the bias of the mean treatment effect. Conclusion Focusing only on the mean treatment effect of the random effects meta-analysis can be misleading when substantial heterogeneity is suspected or outliers are present. In such cases, identifying the factors that differentiate the studies and looking at the prediction intervals can be very informative. Based on our simulation, investigators could have the normal model as their starting point and consider alternative models as sensitivity analysis in view of seemingly non-normal data. Clinical trial number: not applicable