Diagnosing Model Performance Under Distribution Shift

Diagnosing Why Models Fail Under Distribution Shift and What to Do Next Predictive models often perform worse when deployed in a new target setting, but it is rarely clear why. In “Diagnosing Model Performance Under Distribution Shift,” Cai, Namkoong, and Yadlowsky introduce a diagnostic, distribution shift decomposition (DISDE), that attributes the change in performance from the training to target distributions into terms for (i) an increase in harder but previously seen inputs from training, (ii) changes in how outcomes relate to inputs, and (iii) poor performance on new input regions absent from the training data. Applications to employment prediction demonstrate how this decomposition can inform potential modeling improvements, guiding whether to use domain adaptation techniques, adjust model covariates, or collect new samples. Additionally, DISDE is used to help explain why certain domain adaptation methods fail to improve model performance for satellite image classification.