A neural network with episodic memory learns causal relationships between narrative events

Humans reflect on past memories to make sense of an ongoing event. Past work has shown that people retrieve causally related past events during comprehension, but the exact process by which this causal inference occurs remains elusive. Here, we employed a recurrent neural network augmented with an episodic memory buffer to examine how memories are retrieved and integrated based on causal relationships between events. The model was trained to predict upcoming scenes as it watched a television episode. At every time step, the model transformed the current scene into two distinct representations—"value" representing memory content and "key" representing memory address—both of which were stored as episodic memory. The model learned to retrieve selective past values by applying self-attention over stored keys, and it integrated these retrieved values with the current scene representation to predict an upcoming scene. By separating representations used for encoding from those used for retrieval, the model learned to retrieve memories in ways that go beyond simple pattern similarity. In turn, the model represented causally related events with similar patterns beyond perceptual or semantic similarities, suggesting that it organized event representations based on latent causal structure. Memories retrieved by the model were similar to those retrieved by human participants who watched the same television episode. The model also exhibited hippocampus-like pattern separation and pattern completion, and its representational structure aligned more closely with human fMRI data than a comparison model without an episodic memory buffer. These findings suggest that the model captures the way humans represent events and retrieve memories based on causal relationships. Together, this work proposes a key-value episodic memory system as a candidate computational mechanism for how humans retrieve causally related memories to comprehend naturalistic events.