Field-testing multiple-choice questions with AI examinees: English grammar items

Field-testing is a necessary but resource-intensive step in the development of high-quality educational assessments. I present an innovative method for field-testing newly written exam items by replacing human examinees with artificially intelligent (AI) examinees. The proposed approach is demonstrated using 466 four-option multiple-choice English grammar questions. Pre-trained transformer language models are fine-tuned based on the 2-parameter logistic (2PL) item response model to respond like human test-takers. Each AI examinee is associated with a latent ability θ , and the item text is used to predict response selection probabilities for each of the four response options. For the best modeling approach identified, the overall correlation between the true and predicted 2PL correct response probabilities was .82 (bias = 0.00, root-mean-squared-error = 0.18). The simulation study results were promising, showing that item response data generated from AI can be used to calculate item proportion correct, item discrimination, conduct item calibration with anchors, distractor analysis, dimensionality analysis, and latent trait scoring. However, the proposed approach still fell short of the accuracy of analyses that can be achieved with human examinee response data. If further refined, potential resource savings in transition from human to AI field-testing could be enormous. AI could shorten the field-testing timeline, prevent examinees from seeing low quality field-test items in real exams, shorten test lengths, eliminate test security, item exposure, and sample size concerns, reduce overall cost, and help expand the item bank. Example Python code from this study is available on Github: https://github.com/hotakamaeda/ai_field_testing1