Registered Replication Report and Extension of Maas & Hox (2005), Sufficient Sample Sizes for Multilevel Models

The use of multilevel models in behavioral and social sciences has risen dramatically in the past several decades. Researchers using these models often consult highly-referenced simulation studies when designing their own studies, especially for sample-size planning.However, current literature suggests two issues with this practice. First, recent efforts to replicate findings from seminal simulation studies suggest the rate of replicability is far from ideal (Luijken et al., 2024). Second, researchers are now using more complex models but foundational simulation studies were conducted under much simpler conditions. To address these two issues, we propose a replication and extension of Maas & Hox (2005), a widely cited study that recommended Level-2 sample sizes of N = 50 to be sufficient under specific conditions. We replicate their core findings and propose several extensions based on a literature search of multilevel modeling practices employed in fifty recent studies.These extensions include examining conditions with imbalanced data across higher-level units, varying the number of random slopes and cross-level interactions, and varying effect sizes. These results could ensure the existing literature relies on a replicable simulation study while also informing multilevel study planning for complex models which are becoming increasingly popular in behavioral and social sciences.