The Phase-Switch Hybrid Adam–SGD Optimizer for Non-Convex Deep Learning: Evaluation on the MNIST and CIFAR-10 Dataset

Deep neural networks are highly likely to face non-convex optimization issues during training because of local minima, saddle points, and highly complex loss surfaces. Existing optimizers, such as Adam and Stochastic Gradient Descent (SGD), optimize faster or generalize better; however, they cannot optimize both properties effectively for non-convex optimization problems. This study proposes a phase switch hybrid optimization method to optimize and improve the training of deep neural networks. The proposed method uses Adam for faster convergence during the initial phase and SGD with momentum for better generalization during the latter phase. The hybrid optimization method combines the advantages of both Adam and SGD, enabling faster convergence and better generalization during training. The method is evaluated on the well-known datasets MNIST and CIFAR-10. The results obtained using the proposed method are better or comparable to existing methods on different metrics such as accuracy and loss minimization. The introduced method achieves improved performance, with accuracy gains of up to 0.4–0.6% on the MNIST dataset and up to 1–2% on the CIFAR-10 dataset compared to baseline optimizers, along with lower test loss in several learning rate settings. The study demonstrates that the proposed method is an effective solution for handling non-convex optimization problems and improves the robustness of training on different datasets.