Attention is all you need (in the brain): semantic contextualization in human hippocampus

In natural language, word meanings are contextualized, that is, modified by meanings of nearby words. Inspired by self-attention mechanisms in transformer-based large language models (LLMs), we hypothesized that contextualization in the brain results from a weighted summation of canonical neural population responses to words with those of the words that contextualize them. We examined single unit responses in the human hippocampus while participants listened to podcasts. We first find that neurons encode the position of words within a clause, that they do so at multiple scales, and that they make use of both ordinal and frequency-domain positional encoding (which are used in some transformer models). Critically, neural responses to specific words correspond to a weighted sum of that word’s non-contextual embedding and the embedding of the words that contextualize it. Moreover, the relative weighting of the contextualizing words is correlated with the magnitude of the LLM-derived estimates of self-attention weighting. Finally, we show that contextualization is aligned with next-word prediction, which includes prediction of multiple possible words simultaneously. Together these results support the idea that the principles of self-attention used in LLMs overlap with the mechanisms of language processing within the human hippocampus, possibly due to similar prediction-oriented computational goals.