Understanding calibration robustness under extreme spectral multicollinearity: A systematic evaluation of high-dimensional NIR spectroscopy.

High-dimensional near-infrared (NIR) spectroscopy datasets are characterized by strong wavelength correlation, limited sample sizes, and structured experimental variability, creating significant challenges for multivariate calibration. Under these conditions, extreme multicollinearity can lead to unstable regression coefficients, inflated estimator variance, and reduced model robustness. Although numerous calibration algorithms have been proposed, comparatively little attention has been devoted to understanding how multicollinearity influences calibration behaviour across different modelling paradigms. This study systematically investigates the performance of classical chemometric, regularized linear, dimensionality reduction, wavelength selection, and nonlinear machine learning approaches using a publicly available glucose NIR spectroscopy dataset comprising 4200 wavelength variables and structured experimental perturbations. Partial Least Squares (PLS), Ridge Regression, Elastic Net, Principal Component Analysis (PCA), Genetic Algorithm-based PLS (GA-PLS), Artificial Neural Networks (ANN), Support Vector Regression (SVR), Random Forest, Gradient Boosting, and Gaussian Process Regression were evaluated using repeated cross-validation and independent validation. Model assessment was complemented by multicollinearity diagnostics, residual analysis, and wavelength-selection stability analysis to investigate the mechanisms governing calibration robustness. SNV-PLS achieved the best overall predictive performance on the independent validation dataset (RMSE = 5.75 mM; R2 = 0.864), followed by GA-PLS (RMSE = 5.82 mM; R2 = 0.861) and SNV-Ridge (RMSE = 5.84 mM; R2 = 0.860), while SNV-Elastic Net achieved comparable performance (RMSE = 5.93 mM; R2 = 0.856). GA-PLS reduced the spectral dimensionality from 4200 to 50 wavelengths; however, repeated GA runs showed only 0-2% exact wavelength overlap, increasing to 17-29% within ±5 nm and 30-47% within ±10 nm, indicating regional rather than exact-wavelength stability. Variance Inflation Factor analysis revealed extreme predictor collinearity, while residual and domain-coverage analysis showed that the major validation deterioration was localized at 50 mM glucose, where the six validation samples were systematically underpredicted and originated from an experimental condition absent from the calibration set. The findings suggest that effective management of covariance structure and estimator variance may be more influential than increasing model complexity for the investigated high-dimensional NIR calibration problem. These results provide practical guidance for selecting robust and interpretable calibration strategies for high-dimensional spectroscopic applications affected by severe multicollinearity.