Developing NOVA: A Next-Generation Open Vocabulary Assessment

In psychological assessment, vocabulary tests are commonly used as reliable and efficient indicators of crystallized intelligence, as retrospective proxies for premorbid intelligence, and as measures of language proficiency. However, many of the widely used German vocabulary tests are outdated, proprietary, and lack a clear rationale for item selection. To address these limitations, we developed a new, openly available vocabulary test: the Next-Generation Open Vocabulary Assessment (NOVA). Therefore, we first constructed 110 multiple-choice vocabulary items with support from ChatGPT and administered them to 1,052 German-speaking adults using a multiple-matrix design, along with a declarative knowledge test for validation purposes. In a second step, we used Ant Colony Optimization to compile two parallel 30-item short forms, optimized for reliability and item difficulty and discrimination. The resulting tests assessed vocabulary unidimensionally and reliably, covered a large ability range, and correlated strongly with declarative knowledge. We provide a Shiny app for the calculation of standard values based on individual test results. Additional analyses revealed that 57% of the variance in item difficulties could be explained by word frequency and word length, which may be particularly useful for streamlining the future development of vocabulary tests.