NeuroCore: A Framework for Neuromodulation-Regulated Self-Modifying Modular Neural Architectures

We introduce the NeuroCore framework, a formal mathematical treatment of modular neural architectures in which a minimal executive Core—possessing no higher cognitive capabilities—autonomously orchestrates a heterogeneous collection of specialist modules through learned continuous-representation interfaces. The Core’s behavior is governed by two neuromodulation-inspired subsystems: a Dopamine System implementing distributional reinforcement learning with prediction-error intrinsic motivation and a stagnation penalty, and a Serotonin System formulated as a meta-reinforcement-learning controller that learns to optimize long-horizon constraint satisfaction. We make four theoretical contributions. First, we formalize the stagnation-modification tradeoff—proving that without explicit anti-stagnation pressure, optimal policies in self-modifying systems converge to modification-avoidance, and deriving the conditions under which the stagnation penalty restores non-trivial self-modification behavior (Theorem 1). Second, we prove a general non-convergence result for coupled self-modifying multi-objective systems, showing that the joint optimization does not admit guaranteed convergence to fixed points or bounded attractors in the parameter space (Theorem 2). Third, we establish partial stability guarantees: bounded representational drift via homeostatic Lyapunov functions (Theorem 3), local convergence under frozen modules via two-timescale stochastic approximation (Proposition 1), and modification frequency bounds (Proposition 2). Fourth, we derive information-theoretic costs for module manipulation operations that serve as principled proxies for true disruption. We propose seven falsifiable empirical predictions and discuss implications for the design of autonomous self-organizing AI systems.