Increasing the Routine Feasibility of Passage Independence Checks for Multiple Choice Tests of Reading Comprehension

A pure measure of reading comprehension would require the participant to actively engage with the reading material, before being able to correctly answer questions about the reading passage. This principle is known as passage dependence. Quality control of this principle in reading comprehension tests has been called for in the past, but has not been universally adopted. The strongest validation paradigm would require larger field tests where participants answer the reading-comprehension question-items under two test administration conditions: in absence of the associated reading passage (blinded condition) and in presence of the passage (control condition). This study proposes and explores a crude AI-equivalent for passage (in)dependence checks by playing into the strengths of large language models and mimicking the experimental blinding of the validation paradigm. The approach is showcased on a large pool of 79 items associated with 12 reading passages of the Progress in International Reading Literacy Study (PIRLS). We discuss the empirical results and the approach in more detail, with attention to insights, limitations, and directions for future work. The feasibility of the approach calls for a routine implementation of passage (in)dependence checks during the development and review of reading comprehension tests, both in research and in practice.