Unwanted variability in child welfare decision-making: An empirical investigation

Assessing a parent's ability to provide a safe and nurturing environment for their children is a challenging task for child welfare workers. Given the complex and indefinite nature of parenting, decision-makers often rely on subjective criteria in their evaluations. In Study 1 (N = 49), we examined the variability among child welfare professionals as they independently assessed the overall quality of care in ten vignettes, each depicting different scenarios. We found an inter-rater reliability of .50, indicating a concerning level of unwanted variability (‘noise’), with 24.7 % of the variance attributable to average differences between raters. This suggests that raters have different thresholds for what constitutes unsafe parenting. In Study 2 (N = 44), we examined whether aggregating domain-specific evaluations would reduce noise, but the resulting improvement in inter-rater reliability to .54 was not statistically significant. We found 28.7 % level noise after aggregating. We conclude that noise-reduction strategies are appropriate and necessary to improve the consistency of assessments in child welfare decisions.