Rank Dependency of Rescaled Pruning in Recurrent Neural Networks

Throughout development and maturity, neural circuits undergo massive synaptic pruning, yielding highly sparse connectivity while preserving robust population-level computations. These population dynamics are often low-dimensional, allowing task-related computations to be formalized as trajectories within latent subspaces. How such low-dimensional dynamics are preserved amid widespread network sparsification remains unclear. Here, we investigate how different synaptic pruning rules shape low-dimensional dynamics and task performance in recurrent neural networks (RNNs). Moving beyond previous approaches focused on random sparsification of low-rank networks or networks with strictly constrained structures, we systematically evaluate how biologically motivated pruning rules interact with a network’s underlying rank. We show that post-pruning dynamics and task performance depend critically on the network’s initial rank due to distinct eigenspectral characteristics across rank regimes. Combining mathematical analysis with simulations, we demonstrate that pruning with synaptic rescaling preserves low-dimensional dynamics with minimal distortion in low-rank RNNs, but degrades in the high-rank regime. Our findings suggest that low-rank structure, combined with homeostatic synaptic rescaling, is essential for maintaining stable, low-dimensional dynamics in sparse networks.