Resolving a paradox about how vision is transformed into familiarity

While humans and other primates are generally good at remembering the images they have seen, they systematically remember some images better than others (“image memorability”). Here, we leverage image memorability to resolve a puzzle around how the brain transforms seeing into familiarity. Namely, the neural signal driving familiarity reports is thought to be repetition suppression, an overall reduction in the vigor of the population response in brain regions, including inferotemporal cortex (ITC). However, within ITC, more memorable images evoke higher firing rates than less memorable ones, even when they are repeated. These two observations conflict: if reduced firing leads to stronger memory signaling, then why are the images that induce greater firing more memorable? To investigate this “familiarity-memorability paradox”, we compared neural activity in ITC and the hippocampus (HC) as two macaque monkeys performed a single-exposure image familiarity task. We found a resolution to the paradox, reflected in two pieces of evidence. First, memory and memorability were not completely overlapping in ITC, suggesting that familiarity signals could be extracted independently of memorability modulation. Second, responses in HC reflected a transformation of ITC responses that extracted memory, leaving behind stimulus-evoked firing modulation. This transformation could be accounted for by a linear decoder that selectively extracts memory and preserves larger ITC memory signals for more memorable images. These results resolve the familiarity-memorability paradox by demonstrating that ITC familiarity signals are a component (but not the entirety) of repetition suppression, and they establish the existence of a previously undescribed medial temporal lobe computation.