What Do Assessment Center Ratings Reflect? Consistency and Heterogeneity in Variance Composition Across Multiple Samples

The question of what assessment centers measure has remained a controversial topic in research for decades, with a recent increase in studies that (1) use generalizability theory and (2) acknowledge the effects of aggregating post-exercise dimension ratings into higher-level assessment center scores. Building on these developments, we used Bayesian generalizability theory and random-effects meta-analyses to examine the variance explained by assessment center components such as assessees, exercises, dimensions, assessors, their interactions, and the interrater reliability of AC ratings in 19 different assessment center samples from various organizations (N = 4,963 assessees with 272,528 observations). This provides the first meta-analytic estimates of these effects, as well as insight into the extent to which findings from previous studies generalize to assessment center samples that differ in measurement design, industry, and purpose, and how heterogeneous these effects are across samples. Results were consistent with previous trends in the ranking of variance explained by key AC components (with assessee main effects and assessee-exercise effects being the largest variance components) and additionally emphasized the relevance of assessee-exercise-dimension effects. In addition, meta-analytic results suggested substantial heterogeneity in all reliable variance components (i.e., assessee main effect, assessee-exercise effect, assessee-dimension effect, and assessee-exercise-dimension effect) and in interrater reliability across assessment center samples. Aggregating AC ratings into higher-level scores (i.e., overall AC scores, exercise-level scores, and dimension-level scores) reduced heterogeneity only slightly. Implications of the findings for a multifaceted assessment center functioning are discussed.