Steerable autoencoders underlying remapping, spatiotopy, and visual stability.

The encoding of locations in world coordinates is often called spatiotopy; it captures how we experience the layout of things around us. The visual system, on the other hand, appears to rely on retinotopic representations. The search for some critical area of the brain that recovers locations in world coordinates from the sequence of fixations as a spatiotopic map, has not met with much success. Here we review earlier suggestions that replace spatiotopic maps with the tracking or remapping of a small number of attended targets to keep their positions updated on retinotopic maps (e.g., Wurtz, 2018; Golomb & Mazer, 2021). This remapping process achieves a simulation of spatiotopy – the target locations are updated and motion responses to the shift of the retinal image are suppressed and discounted. Importantly, we propose that each attended target’s identity is connected to the locations of its features in early visual cortex through a steerable autoencoder network that shifts the feature activity to the target’s new location with each saccade, explaining the many reports of trans-saccadic integration. This autoencoder process is not limited to the remapping function but is part of the overall architecture of the visual system, acting to link target properties together, even as the target or the eyes move.