The effects of object category training on the responses of macaque inferior temporal cortex are consistent with performance-optimizing updates within a visual hierarchy

How does the primate brain coordinate plasticity to support its remarkable ability to learn object categories? To address this question, we measured the consequences of category learning on the macaque inferior temporal (IT) cortex, a key waypoint along the ventral visual stream that is known to support object recognition. First, we observed that neural activity across task-trained monkeys’ IT showed increased object category selectivity, enhanced linear separability (of categories), and overall more categorical representations compared to those from task-naïve monkeys. To model how these differences could arise, we next developed a computational hypothesis-generation framework of the monkeys’ learning process using anatomically-mapped artificial neural network (ANN) models of the primate ventral stream that we combined with various choices of learning algorithms. Our simulations revealed that specific gradient-based, performance-optimizing updates of the ANN’s internal representations substantially approximated the observed changes in the IT cortex. Notably, we found that such models predict novel training-induced phenomena in the IT cortex, including changes in category-orthogonal representations and IT’s alignment with behavior. This convergence between experimental and modeling results suggests that plasticity in the visual ventral stream follows principles of task optimization that are well approximated by gradient descent. We propose that models like the ones developed here could be used to make accurate predictions about visual plasticity in the ventral stream and its transference – or lack thereof – to any future test image.