Multimodal inference through mental simulation

The ability to infer the past from what we perceive in the present is a key capacity of human cognition. Witnessing a broken vase, humans will automatically bring to mind a causal story of what happened. Multiple sources of sensory evidence can support this inference. Seeing the broken pottery tells you something, but hearing the crash tells you even more. In this work, we explore people's inferences about the past from multimodal evidence. We present a physical reasoning paradigm called Plinko. In the prediction task, participants must determine where a ball that is dropped into a box with obstacles will land. A computational model that uses mental simulation in an Intuitive Physics Engine captures participant predictions very well. In the inference task, participants must infer which hole in the box the ball fell from. Across conditions, participants are presented with different combinations of visual and auditory cues, and must combine this information to determine what happened. We develop a sequential sampling model that selectively simulates from promising hypotheses, and demonstrate that this model accurately captures participants' judgments and eye-movements. By coordinating sensory evidence in an underlying causal representation of the physical world, this simulation approach is able to capture complex multimodal inferences that go beyond traditional approaches to multimodal integration.