Guidelines for the interpretation of NCDIF as an effect size measure

Several statistics have been proposed to detect Differential Item Functioning (DIF) and quantify its magnitude. However, not all DIF statistics quantify the same population parameters and, therefore, cannot be interpreted interchangeably. We focus on deriving principled guidelines for the interpretation of the non-compensatory differential item functioning (NCDIF) parameter as a DIF effect size measure. This parameter quantifies the expected effect on the squared differences between the item scores for the focal population of examinees. We first investigate under which conditions NCDIF may be compared to the Delta Mantel-Haenszel (∆MH) so that the ETS cutoff values can serve meaningfully as a benchmark. We then examine the behavior of the NCDIF parameter as a function of uniform and non-uniform DIF, as well as the effect that the distribution of the focal group exerts on its magnitude. Lastly, using one estimator of the NCDIF parameter, we evaluate the accuracy of the derived classification rules for NCDIF and identify an approximate bias correction for this estimator. Overall, these results provide useful guidelines for interpreting the magnitude of NCDIF that are consistent with its specific nature and improve the alignment of DIF classifications with the magnitude of the NCDIF parameter.