Optimizing Multi-GPU Training with Data Parallelism and Batch Size Selection

This study investigated the optimization of deep learning model training in multi-GPU environments, focusing on the impact of batch size on computational efficiency and model performance. The authors conducted experiments using the MobileNetV2 architecture on a large-scale image dataset, employing single-GPU and multi-GPU setups. It was found that multi-GPU setups significantly reduced training times and mitigated memory constraints. Batch size configurations of 16, 32, 64, and 128 were analyzed to determine their influence on validation accuracy and convergence rates. The study established that a batch size of 64 provided the best balance between training efficiency and model generalization. The research highlights the benefits of data parallelism and multi-GPU systems while addressing the trade-offs between computational speed and accuracy. Suggestions for future work include developing adaptive batch size techniques and extending the analysis to other architectures and datasets.