No Coincidence, George: Processing Limits in Cognitive Function Reflect the Curse of Generalization

The striking constraints of some human cognitive processes stand in stark contrast to the nearlimitless capability of others. While we can acquire and flexibly use vast amounts of information,the amount we can process at any one time is often stiflingly limited. Here, we integrate ideasfrom information-theory, cognitive science, and neuroscience to offer a unified account of whyprocessing is often so limited. We argue that this reflects a fundamental tradeoff betweenrepresentational efficiency and processing efficiency. ‘Representational efficiency’ refers to howmuch and how compactly information is represented by an agent, that is directly related to itscapacity for generalization. We distinguish this from ‘processing efficiency’, which refers to howmany representations can be processed at the same time. We show that maximizingrepresentational efficiency to optimize the capacity for generalization — a characteristicallyhuman cognitive strength — comes at the expense of surprisingly strict limits in processingcapacity, an equally characteristic human weakness that has been observed in a variety ofcognitive tasks. We refer to this as the “curse of generalization,” and formulate this first ininformation theoretic form, and then demonstrate it in a neurally motivated model of a set ofcanonical cognitive tasks that have been used to demonstrate the strict limits in humanprocessing capacity. We suggest that the tension between representational efficiency andprocessing efficiency imposes a fundamental constraint on information processing, that mayprovide a unified explanation for a wide range of psychological phenomena, from performancein the tasks on which we focus to representational learning and skill acquisition more broadly, aswell as the performance of modern machine learning architectures that exhibit generalizationcapabilities comparable to humans.