Maëlys Solal, Pascaline André, Ninon Burgos, For The Alzheimer's Disease Neuroimaging Initiative Group
We highlight a largely overlooked source of variability in deep generative models, their random seed, which can lead to substantially different anomaly detection performance and biased evaluation. Our experimental analysis emphasizes that it is crucial to design robust anomaly maps and to mitigate the impact of randomness-induced variability, by accounting for model reconstruction errors for instance using Z -score ensembling, or healthy-control normalization, to support more stable and trustworthy deployment in real-world clinical practice.
PURPOSE: Unsupervised anomaly detection allows identifying anomalies from unlabeled data, making it useful for neuroimaging analysis and computer-aided diagnosis. Given an individual's scan, we use a generative model to construct a subject-specific image of healthy appearance and compare both images with identify anomalies. Such approach has drawbacks as the reconstructions are imperfect, and model variability is not taken into account.
APPROACH: We study model variability arising from using different random seeds during training and explore strategies to mitigate the effect of unwanted reconstruction errors and variability. The strategies include model ensembling, anomaly map normalization, and anomaly map designs based on single or multiple pseudo-healthy reconstructions. We compare these approaches in the context of dementia-related anomalies on 3D FDG PET from ADNI using variational autoencoder models.
RESULTS: Our experiments suggest that variance between models can be reduced by aggregating their reconstructions in a Z -score based anomaly map. This strategy is highly effective, but computationally expensive, as it requires training several instances of the model. An alternative strategy is to normalize the anomaly map using statistics computed from a healthy validation set. We show that this normalization strategy substantially reduces performance variability across models and even increases anomaly detection performance in certain cases.
CONCLUSION: We highlight a largely overlooked source of variability in deep generative models, their random seed, which can lead to substantially different anomaly detection performance and biased evaluation. Our experimental analysis emphasizes that it is crucial to design robust anomaly maps and to mitigate the impact of randomness-induced variability, by accounting for model reconstruction errors for instance using Z -score ensembling, or healthy-control normalization, to support more stable and trustworthy deployment in real-world clinical practice.