ProReGen: Progressive Residual Generation under Attribute Correlations.

Attribute correlations in the training data will compromise the ability of a deep generative model (DGM) to synthesize images with under-represented attribute combinations (i.e., minority samples). Existing approaches mitigate this by data re-sampling to remove attribute correlations seen by the DGM, using a classifier to provide pseudo-supervision on generated counterfactual samples, or incorporating inductive bias to explicitly decompose the generation into independent submechanisms. We present ProReGen, a progressive residual generation approach inspired by the classical Robinson's transformation, to partial out from an image attribute x 2 its component m x 1 that is predictable by other image attributes x 1 , and the residual γ = x 2 - m x 1 that is not. This simplifies the problem of learning a DGM g x 1 , x 2 conditioned on correlated inputs, to learning g ~ x 1 , γ conditioned on orthogonal inputs. It further allows us to progressively learn g ~ by first shifting the burden to abundant majority samples to learn g ~ x 1 , γ = 0 , and then expanding it with additional layers g res to resolve its difference to g ~ x 1 , γ using residual attribute γ on limited minority samples. On three benchmark datasets with varying strengths of attribute correlation and one dataset with natural attribute correlation, we demonstrate that ProReGen-with input orthogonalization and progressive residual learning-improved the correctness of minority generations compared to existing strategies.