Machine learning for item parameter generation with small samples

Item response theory (IRT) is a widely used technique for modeling testing data and obtaining information about examinee performance and item characteristics. The most widely used IRT models typically require sample sizes of 200 or more in order to obtain accurate and efficient parameter estimates. However, in some cases such sample sizes are not available to psychometricians and measurement professionals because they are working with small and/or difficult to sample populations. The goal of the current study was to describe and investigate the performance of an alternative approach for estimating IRT item parameters using machine learning models. The basic framework involves generating a large set of training data using very general item parameter values and then calculating both IRT and classical test theory (CTT) item parameter estimates for the training data. The machine learning algorithms are then trained to predict the IRT parameter estimates using the CTT estimates. In turn, these trained models are then applied to a set of observed item responses from a small sample. A simulation study revealed that for samples of fewer than 100 individuals, the machine learning approach outperformed standard IRT estimation algorithms in terms of both estimation accuracy and efficiency. Implications for practice are discussed.