Implementing argument-based validity in second language research on individual differences

Quantitative meta-syntheses focusing on psychometric features of measurement tools in second and foreign language (L2) individual differences (ID) research have called scholars’ attention to data collection instrument quality (Plonsky & Derrick, 2016; Sudina, 2023; Teimouri et al., 2019). It appears that in many primary L2 ID studies systematically reviewed to date, little attention has been given to instrument validity issues, which may result in formulating unsubstantiated inferences and drawing unjustifiable conclusions. Although there have been some attempts at raising awareness concerning the importance of validity evidence linked to certain constructs (e.g., language learning motivation; see Al-Hoorie et al., 2024; Papi & Teimouri, 2024), an overarching theoretical and methodological framework of validation that L2 ID researchers could rely on seems to be missing. In our theoretical paper, we intend to provide such scaffolding to measurement in L2 ID studies by drawing on argument-based validation (Kane, 2013; for a recent synthesis, see Zhang & Aryadoust, 2022) practices in other areas of applied linguistics (Chapelle, 2021). For illustrative purposes, we used a quantitative measure with documented psychometric characteristics. Thus, we hope to provide empirically based guidelines for L2 ID researchers on reporting information about the validation process and validity evidence of their psychometric tools.