Visual Regularities Underlie Hierarchical Object Representations in the Human Brain and Self-supervised DNN

NeuroAI develops the interplay of neuroscience and artificial intelligence, especially on visual processing. Human visual system organizes objects based on a representational hierarchy. However, it remains unclear whether this hierarchy arises from visual or semantic information. One hypothesis posits that the visual system is structured around statistical regularities of visual information. Here, we test this hypothesis using the THINGS datasets and pure-visual deep neural networks (DNN). We constructed a low-dimensional object space based on multiple abstract object properties, reflecting statistical patterns of visual regularities. By applying voxelwise encoding models, we identified clusters in the higher visual cortex based on their property tuning, and they were found to support specific object categories. These clusters serve as the middle level to reveal a property-cluster-object hierarchical organization. Subsequently, we investigated whether this hierarchical structure could be captured by a self-supervised DNN. Through activity similarity analysis, we mapped the brain clusters onto the DNN and independently found that the DNN’s clusters exhibited distinct property tuning and influenced the classification accuracy of corresponding object categories, mirroring the effects observed in the human brain. Our results demonstrate similar hierarchical structures in the human brain and self-supervised DNN, suggesting that the visual regularities shape neural architecture of visual system. This study highlights the great potential of neural computational model in neuroscience study.

Index Terms Visual Processing, Abstract Property, Hierarchical Representation, Self-supervised Visual DNN