Julien Michel, Jordi Inglada
This paper introduces a general formulation for the fusion of Satellite Image Time Series (SITS) of variable length from several sensors at different spatial resolutions and acquisition times over the same geographical area. In this formulation, all the spectral bands from all the input sensors are predicted at the best input spatial resolution, and at any observed or non-observed acquisition time requested. To address this general problem, an advanced Masked Auto-Encoder training strategy is proposed, utilising two new loss functions: a Linear-Regression Learned Perceptual Image Similarity term to favor high spatial frequency details, and a mask-contrastive term to ignore clouds and other non-informative areas in the input data. This strategy is applied to the training of Temporal Attention Multi-Resolution Fusion of Satellite Image Time-Series (TAMRF-SITS), a novel Deep Learning architecture designed to implement the proposed general formulation. Experiments with joint Landsat-8/9 and Sentinel-2 time-series were conducted on four different tasks from the literature and demonstrate that a single pre-trained TAMRF is on par with or better than existing ad-hoc methods. For instance, TAMRF provides a gain of 0.01 surface reflectance Root Mean Square Error on a Spatio-Temporal Fusion task when compared to competing algorithms, while showing the highest spatial frequency content. Moreover, the proposed method relaxes unrealistic assumptions routinely found in the literature, including: same or similar spectral bands in different sensors, same-day acquisitions, and scale-invariance of the relationship between high and low resolution images. To the best of our knowledge, our method is the first to achieve this range of capabilities with a single model, without making any of these assumptions. The complete source code for training and experiments is available here: https://github.com/Evoland-Land-Monitoring-Evolution/tamrfsits .