Robust assessment of the cortical encoding of word-level expectations using the temporal response function

Speech comprehension involves detecting words and interpreting their meaning according to the preceding semantic context. This process is thought to be underpinned by a predictive neural system that uses that context to anticipate upcoming words. Recent work demonstrated that such a predictive process can be probed from neural signals recorded during ecologically-valid speech listening tasks by using linear lagged models, such as the temporal response function. This is typically done by extracting stimulus features, such as the estimated word-level surprise, and relate such features to the neural signal. While modern large language models (LLM) have led to a substantial leap forward on how word-level features and predictions are modelled, there has been little progress made towards the metrics used for evaluating how well a model is relating stimulus features and neural signals. In fact, previous studies relied on evaluation metrics that were designed for studying continuous univariate sound features, such as the sound envelope, without considering the different requirements of word-level features, which are discrete and sparse in nature. As a result, studies probing lexical prediction mechanisms in ecologically-valid experiments typically exhibit small effect-sizes, severely limiting the type of observations that can be drawn and leaving considerable uncertainty on how exactly our brains build lexical predictions. First, the present study discusses and quantifies these limitations on both simulated and actual electroencephalography signals capturing responses to a speech comprehension task. Second, we tackle the issue by introducing two assessment metrics for the neural encoding of lexical surprise that substantially improve the state-of-the-art. The new metrics were tested on both the simulated and actual electroencephalography datasets, demonstrating effect-sizes over 140% larger than those for the vanilla temporal response function evaluation.