AURA: An Adaptive Unified Regularization Approach for Gradient-Based Optimization

In the engineering world, optimization plays an important role i.e. almost every industry tries to optimize their systems as much as possible in order to increase efficiency. Hence, in the machine learning universe as well, optimization is quite famous and when it comes to optimisation, one algorithm which often comes to our mind is gradient descent. Currently, there are many customized optimiser techniques which describe different kinds of techniques and processes to converge faster and obtain a low error rate. We propose AURA (Adaptive Unified Regularized Algorithm), a novel stochastic optimizer that shifts adaptation from the learning rate to the momentum parameter. Unlike conventional adaptive methods such as Adam and RMSProp, which primarily rely on per-parameter learning rate scaling, AURA maintains a fixed learning rate and instead adaptively modulates momentum ($\beta$) through three synergistic signals: (i) loss-trend awareness, which captures short-term dynamics in optimization stability, (ii) gradient-norm sensitivity, which prevents instability under varying gradient magnitudes, and (iii) cosine-similarity modulation, which aligns current updates with historical trajectories to enhance directional consistency. Empirical evaluations on classification and regression benchmarks demonstrate that AURA achieves competitive or superior convergence behavior compared to widely used optimizers.