Evaluating DIF Using Various Detection Methods in Small Samples

When testing for differential item functioning (DIF) in an item, researchers are often confronted with small sample sizes. Six DIF detection methods, designed to produce more stable inferences for DIF parameters, were examined under small sample conditions using both real and simulated data. These methods included the chi-square test based on the classical Mantel-Haenszel (MH) procedure with raw data, common odds ratio based the classical MH procedure with raw data, common odds ratios based on the MH procedure with log-linear smoothing and an empirical Bayes (EB) approach, Rasch-based improved version of Lord’s Wald test (i.e., Wald-1), and Rasch-based likelihood ratio test (LRT), all of which have been used in previous research studies. The results indicated that the common odds ratios of MH with raw and smoothed data produced almost identical results, while the Rasch-based methods exhibited similarities in terms of the Type Ι error and power rates. The findings from the current study support the use of the Rasch-based methods for detecting DIF. Specifically, the Type Ι error rates for the Rasch-based methods were below the significance level of .05 across all conditions. In contrast, the Type Ι error rates for the MH procedures, both with raw and smoothed data, exceeded .05 when the sample sizes were 50 and 100. However, the Rasch-based methods showed power rates below the .80 threshold when the sample sizes were 50 and 100. Moreover, the Type Ι error rates for the EB approach to MH procedure were lower than those for other MH procedures in most cases. Nevertheless, the power rates for the EB approach to MH tended to be similar to those for other MH procedures as the percentages and magnitudes of DIF increased.