SEN12MS-CR-TS DATASET

On average, over half of all optical observations acquired via spaceborne Earth imagery are affected by clouds. As cloud coverage severely impedes the ongoing observation of Earth, the automated reconstruction of noisy or cloud covered information is a persistent problem in signal processing and remote sensing. While classical remote sensing applications oftentimes focused on narrowly-defined areals and case studies, the increasing availability of public-access, daily and large-scale satellite monitoring shifted the community's interest towards globally applicable methodology. To support the development of modern machine learning techniques for the purpose of cloud removal on whole-planet satellite data, we curated a large data set for training and evaluating new approaches.

This data set, SEN12MS-CR-TS, is a multi-modal and multi-temporal data set for cloud removal. It contains paired and co-registered sequences of space-borne radar measurements practically unaffected by clouds, as well as cloud-covered and cloud-free multi-spectral optical satellite observations. Complementary to the radar modality's cloud-robust information, historical satellite data is collected to provide additional sensor information for more faithfully reconstructing cloud-covered pixels. The radar and optical data are collected via Sentinel-1 and Sentinel-2 satellites from European Space Agency's Copernicus mission, respectively. The Sentinel satellites provide public access data and are among the most prominent satellites in Earth observation.

SEN12MS-CR-TS is the first public data set for cloud removal in Earth observation to provide repeates measures of multi-sensor data at a large-scale global and all-season coverage. Based on the observation that cloud coverage varies widely in practice, all scenarios ranging from clear skies to absolute coverage are contained in the train and test data. By making this curated and readily pre-processed data set available to the research community, we hope to hope to help advance automated cloud removal in optical satellite data.

SEN12MS-CR-TS is a global data set for multi-modal and multi-temporal cloud removal. It contains observations covering 53 globally distributed Regions of Interest, backward-compatible with the SEN12MS-CR data set. For each region, 30 temporally aligned synthetic aperture radar Sentinel-1 as well as optical multi-spectral Sentinel-2 images are collected throughout the whole year of 2018. 30 time samples for each of the 15578 patch-wise observations, for every S1 and S2 measurement. The approximate cloud coverage of all data is about 50%---reaching from clear-view images (e.g. for validation purposes), over semi-transparent haze or small clouds to dense and wide cloud coverage. The geospatial locations of ROI and our train and test split definitions are compatible with the mono-temporal SEN12MS-CR data set for cloud removal, so both data sets can complement one another and be combined for developing and benchmarking cloud removal solutions.

Cloud Removal Models

Model	Weights Download	Description
CR-TS Net (t=3)	pCloud Share	[PyTorch] The network trained on SEN12MS-CR-TS in Ebel et al 2022 on t=3 temporal samples of paired radar and cloudy optical satellite observations.
CR-TS Net (t=4)	pCloud Share	[PyTorch] The network trained on SEN12MS-CR-TS in Ebel et al 2022 on t=4 temporal samples of paired radar and cloudy optical satellite observations.
CR-TS Net (t=5)	pCloud Share	[PyTorch] The network trained on SEN12MS-CR-TS in Ebel et al 2022 on t=5 temporal samples of paired radar and cloudy optical satellite observations.
UnCRtainTS	pCloud Share	[PyTorch] All varieties of the uncertainty prediction networks trained on SEN12MS-CR-TS in Ebel et al 2023 on paired radar and cloudy optical satellite observations. Code here.

Complementary Models

Model	Weights Download	Description
VGG16	pCloud Share	[PyTorch] A VGG16 trained on 13-bands Sentinel-2 samples for global scene and land cover type classification, trained on SEN12MS. Can be used for perceptual losses, style transfer, multi-spectral Fréchet inception distance and other metrics, or for other purposes.
ResNet	pCloud Share	[PyTorch] A ResNet16 pre-trained for mono-temporal cloud removal on SEN12MS-CR. Takes radar and multi-spectral satellite data as inputs to make cloud-removed multispectral optical predictions.