Generalized Active Stratified Sampling for Non-IID Data

Active learning (AL) is a semi-supervised learning paradigm with human-machine interaction and a limited annotation budget. However, few AL studies have explored distribution inconsistency between the data and the population. In this paper, we consider a basic form of the aforementioned issue,i.e., the training data is non-independently and identically distributed (non-IID) sampled from a class uniformly distributed population. Accordingly, we propose a naïve sample selection plugin, namely generalized active stratified sampling (GASS), to rebalance the sample size of each class during AL iterative process, resulting in a progressive approximation to the population. We generalize statistical stratified sampling to support the uncertainty strata criterion, forming the statistical foundation of GASS. This method, as a plugin, can seamlessly collaborate with popular information-based strategies. GASS shows superior rebalancing capabilities by analyzing the statistical moment and the class imbalanced index under the Probably Approximately Correct (PAC) theory. Furthermore, models derived with GASS have low Rademacher complexity (RC), indicating low generalization error bounds, and GASS also exhibits strong robustness to prediction perturbations. Experiments were conducted on 5 benchmark image datasets, and the results show that GASS significantly boosts the test accuracy by about$2.38\%$/$3.19\%$(paired$t$-test$p=0.01$/0.04) and reduces the empirical RC by about$1.42\%$/$1.94\%$(paired$t$-test$p=0.01$/0.05) on average in class imbalanced/balanced scenarios, respectively. This study establishes a potential benchmark for information-based AL.