Racial Bias in the ADOS: Comparison of Differential Item Functioning and Moderated Nonlinear Factor Analysis Among More Than 4,000 Participants from NDAR and SSC

Background: The Autism Diagnostic Observation Schedule (ADOS) is one of the most highly regarded autismevaluations. Unfortunately, multiple studies have demonstrated significant differential itemfunctioning (DIF) for individuals from different racial groups, and the mechanism for racial bias inthe ADOS remains unknown because DIF effects are inconsistently replicated across items.Although the ADOS consists of 5 modules and 2 historical versions (ADOS and ADOS-2), studiessearching for bias using item response theory-based DIF analyses often compromise statisticalpower to account for these differences by subdividing their samples. A moderated nonlinear factoranalysis (MNLFA) is more appropriate because it allows for the presence of multiple moderators,where there may be multiple sources of bias, and it allows the specification of bias receptors (e.g.,item-level, factor-level). To date, six studies have used MNLFA to assess differential itemfunctioning in the ADOS for age or sex. No study has used MNLFA to examine racial bias, nor hasit been used to examine differences across the different modules and versions of the ADOS.Objectives: To test for differential functioning in the ADOS items across race (African-American, Asian,White), module (1-4), and version (ADOS and ADOS-2).Methods:The current study is the largest examination of DIF in the ADOS to date (N = 4354+), utilizing datafrom the Simons Simplex Collection (n = 2851) and the NIMH Data Archive (n = 1503). Datacollection is complete for Module 3 and forthcoming for Modules 1, 2, and 4 (due to a recentshutdown of the NIMH Data Archive). First, item response theory-based DIF was conducted on the14 algorithm items using the lordif package in R. A graded response model was selected given theordinal response data. Chi-square significance tests were used to detect significant uniform andnon-uniform DIF, with an alpha cutoff value of 0.05. DIF analysis will be further conducted withdata from all modules and versions aggregated. Second, measurement invariance will be tested viaMNLFA, using the OpenMx package in R. A confirmatory one-factor model, specified with the 14algorithm ADOS items as manifest variables and with race, version, and module as moderatingeffects for the factor loadings and item intercepts in Figure 1, will be used.Preliminary Results:Within Module 3 aggregated across versions 1 and 2, four algorithm items showed significant DIF:B1 “Eye Contact” (Pr(χ212)=0.117, Pr(χ223)=0.001), B4 “Shared Enjoyment’ (Pr(χ212)=0.709,Pr(χ223)=0.004), B11 “Quality Rapport” (Pr(χ212)=0.001, Pr(χ223)=0.267), and D2 “ComplexMannerisms” (Pr(χ212)=0.021, Pr(χ223)=0.675). The item characteristic curves of B11 and D2suggest uniform DIF, whereas DIF in B1 and B4 is non-uniform (Figure 2).Conclusion:In the largest sample to date, we have identified bias in four ADOS Module 3 algorithm items. Wewill conduct a larger analysis with MNLFA to determine whether the source of racial bias is due tospecific modules or versions of the ADOS. This analysis will allow us to pinpoint specific moduleitems and offer solutions to improve the measure and thereby diagnostic accuracy and equity fordiverse individuals.