Comparative analysis of algorithmic approaches in ensemble learning: bagging vs. boosting

Ensemble learning is widely applied in various real-world settings, with Bagging and Boosting being two core algorithms. Although these techniques have been extensively investigated through experimental comparisons of their performance in various scenarios, few studies have analyzed and quantified their benefits, costs, and complexities to support algorithm-aware decision making. In this study, we develop a theoretical model to compare Bagging and Boosting in terms of performance, computational costs, and ensemble complexity, and validate it through experiments on four datasets (MNIST, CIFAR-10, CIFAR-100, IMDB) with varying data complexity and computational environments. The results show that, for MNIST, as ensemble complexity increases (e.g., from 20 to 200), Bagging's performance improves from 0.932 to 0.933 before plateauing, while Boosting improves from 0.930 to 0.961 before showing signs of overfitting. At the same ensemble complexity, such as 200 base learners, Boosting requires approximately 14 times more computational time than Bagging, indicating substantially higher computational costs. Similar patterns are observed across the other three datasets, confirming the generality of our findings and revealing consistent trade-offs between performance and computational costs. Taken together, these results confirm the robustness of our theoretical predictions and provide a foundation for practical guidance. Specifically, decision-makers prioritizing cost-efficiency may prefer Bagging, whereas those focusing on maximizing performance might find Boosting more beneficial. For simpler datasets on average-performing devices, Boosting can be effective, whereas Bagging is more suitable for complex datasets on high-performing devices. Overall, this study contributes by integrating analytical modeling with empirical validation across multiple datasets to provide theoretical insights and practical guidance. It systematically compares Bagging and Boosting in terms of performance, computational costs, and ensemble complexity, thereby enabling practitioners to choose the most appropriate method under varying data complexities, performance needs, and resource constraints.