Incremental Alternative Sampling as a Lens into the Temporal and Representational Resolution of Linguistic Prediction

This study introduces a new information-theoretic model of predictive uncertainty in incremental language comprehension, which builds on and extends the computational framework of Surprisal theory. Incremental Alternative Sampling (IAS) posits that a comprehender's predictive uncertainty over upcoming linguistic units in a given sentential or discourse context is proportional to the units' representational distance from her expected alternative continuations. We formally define the information measure derived from this model—incremental information value—and situate it within the broader family of uncertainty measures. This formalism offers greater flexibility for characterising incremental prediction than the classical measure of surprisal. Our analysis focuses on the representational domain and forecast horizon of prediction; however, the theoretical framework provides a practical foundation for investigating a wide range of aspects of prediction. Results indicated that estimates of incremental information value—computed via Monte Carlo sampling of alternatives using artificial neural network language models—possess significant predictive power across various behavioural and neural responses. These include explicit predictability judgements, cloze probabilities and surprisals, as well as signatures of processing effort like eye-tracked and self-paced reading times, and event-related brain potentials. Perhaps more importantly, IAS offers insights into the potential predictive processing strategies underlying these responses: different psycholinguistic measurements are better predicted by distinct combinations of representational and temporal resolutions, as captured by different IAS models. Beyond serving as a robust cognitive model of human comprehension, IAS also provides a window into language models' predictive processing. Analysis of language model estimates of next-word surprisal through the lens of incremental alternative sampling reveals that, despite its seemingly narrow focus on the immediate next unit, next-word surprisal estimates implicitly encode predictive uncertainty over multiple future lexical items across varying levels of representation.