Uncertainty explanation of artificial intelligence models by SHAP

• Uncertainty estimation and XAI are both solutions to “black box” characteristics. • Use the model-agnostic XAI methods (i.e., SHAP) to explain AI model uncertainty. • Find that the estimated uncertainty comes from the distribution of training data. • Emphasize the uncertainty explanation is different from general model explanation. The “black box” characteristic of AI models has received widespread attention from scientists. This has promoted the development of explainable artificial intelligence (XAI) methods aimed at explaining AI model decisions, e.g., evaluating the importance of input features for the model output. In addition to XAI, another way to open the “black box” is uncertainty estimation of AI models (e.g., output ranges for regression tasks), which provides users with confidence in model decisions. However, little attention has been paid to the explanation of model uncertainty, e.g., which input features contribute to model uncertainty? This study takes SHapley Additive exPlanations (SHAP) as an example of perturbation-based XAI methods and utilizes it to explain AI model uncertainty by changing the model input to observe the variation of the estimated uncertainty. The experimental results in three cases (i.e., house price prediction, remote sensing image classification, and electrocardiogram classification) show that SHAP can effectively explain the uncertainty of AI models. Moreover, further analysis reveals that the uncertainty of AI models is affected by the frequency of occurrence of input features in the training data. Specifically, if a certain feature of the sample rarely appears in the training data, it will lead to greater uncertainty. This conclusion aligns with common sense and supports the validity of the uncertainty explanation.