Rigorous validation of machine learning models for orthodontic extraction prediction using nested cross-validation and smotenc-integrated pipelines.

BACKGROUND/PURPOSE: Machine learning (ML) in orthodontics often suffers from optimistic bias due to data leakage and improper handling of class imbalance. This study implemented and evaluated a production-grade validation framework - nested cross-validation (CV) - for three distinct algorithm classes (logistic regression, random forest, and gradient boosting) using a consensus-labeled orthodontic dataset. MATERIALS AND METHODS: A retrospective dataset of 500 patients (334 non-extraction/166 extraction) was curated with 41 processed features. Two stratified split protocols (70/30 and 80/20) were evaluated to assess model robustness. To ensure methodological rigor, a 5-fold outer/3-fold inner nested cross-validation was implemented. Crucially, SMOTENC (synthetic minority over-sampling technique for nominal and continuous) was integrated inside the training pipeline to prevent data leakage. RESULTS: All models achieved high discrimination (ROC-AUC >0.87). XGBoost achieved the highest discrimination (ROC-AUC: 0.9177) and demonstrated the best overall balance for clinical decision support (balanced accuracy: 0.8566, F1-score: 0.8125). Logistic regression showed the highest internal stability (CV mean balanced accuracy: 0.8521 ± 0.0281) despite lower performance on the hold-out test set. CONCLUSION: The use of nested CV combined with pipeline-integrated SMOTENC provides a conservative and reliable estimate of model performance. While gradient boosting (XGBoost) offers superior predictive power, the framework highlights the trade-off between stability (linear models) and performance (boosting) in dentistry.