A Hybrid Dropout Framework for Enhanced Generalization in Convolutional Neural Networks

Regularization is a major challenge when training deep neu- ral networks, especially with small and medium-sized datasets. Tradi- tional dropout methods like standard dropout, Monte Carlo dropout, variational dropout, structured dropout, and concrete dropout aim to reduce overfitting. However, each method behaves inconsistently when used alone across different layers of the network. This paper presents a Hybrid Dropout Framework with a specific regularization approach for a specific dataset. For the CIFAR-10 dataset, the framework uses fixed- rate standard dropout in convolutional layers, adaptive concrete dropout in the dense layer, and structured dropout to improve spatial regulariza- tion. For the UCI Digits dataset, the hybrid approach combines standard dropout, concrete dropout, and Monte Carlo averaging during inference to improve the performance. The Monte Carlo averaging is applied dur- ing inference to predict uncertainty and avoid unnecessary computation overhead during the training phase. The framework is easy to implement and works with various architectures. The results obtained for applying this framework on both CIFAR-10 and UCI Digits dataset shows positive performance, highlighting the effectiveness irrespective of the size of the dataset. Using the proposed hybrid approach, this work achieves an ac- curacy of 90.5% on the CIFAR-10 dataset and 98.61% on the UCI Digits dataset. This shows improvements of 0.78% and 0.28%, respectively, in comparison to the best-performing individual dropout method.