The majority of all optical observations collected via spaceborne satellites are affected by haze or clouds. Consequently, persistent cloud coverage affects the remote sensing practitioner's capabilities of a continuous and seamless monitoring of our planet. The mission of this site is to facilitate research and development around optical satellite image reconstruction, and to provide the data needed for reconstructing cloud-covered information in satellite observations around the world.
Preceding efforts in this domain were typically confined to narrowly-defined regions of interest, not addressing the matter of generalization to previously unseen areas or across seasons. The data sets presented on this site provide curated observations to train and test contemporary deep learning approaches for the purpose of satellite image reconstruction. For this purpose, the sampled regions of interest are distributed across all continents of Earth, captured throughout the year and across seasons. Moreover, all extent of cloud coverage is represented—from filmy and semi-transparent haze to wide and thick cloud coverage, to model and reflect conditions encountered in practice.
While the Earth observation community enjoys an evergrowing amount of satellite missions equipped with specialised sensors, the image modalities of relevance and considered in cloud removal are typically two-fold:
To facilitate future research addressing the challenge of cloud removal, we offer two global, multi-seasonal and readily pre-processed data sets, which are as follows.
SEN12MS-CR is a multi-modal and mono-temporal data set for cloud removal. It contains observations covering 175 globally distributed Regions of Interest recorded in one of four seasons throughout the year of 2018. For each region, paired and co-registered synthetic aperture radar (SAR) Sentinel-1 measurements as well as cloudy and cloud-free optical multi-spectral Sentinel-2 observations from European Space Agency's Copernicus mission are provided. The Sentinel satellites provide public access data and are among the most prominent satellites in Earth observation.
SEN12MS-CR-TS is a multi-modal and multi-temporal data set for cloud removal. It contains time-series of paired and co-registered Sentinel-1 and cloudy as well as cloud-free Sentinel-2 data from European Space Agency's Copernicus mission. Each time series contains 30 cloudy and clear observations regularly sampled throughout the year 2018. Our multi-temporal data set is readily pre-processed and backward-compatible with SEN12MS-CR.
UnCRtainTS is a state-of-the-art neural network for multi-modal multi-temporal cloud removal. At the time of its release, it outperforms all baselines on both SEN12MS-CR and SEN12MS-CR-TS. In addition to providing cloud-removed satellite images, UnCRtainTS also predicts pixel-wise uncertainty maps. The uncertainty maps are indicative of the potential error on a per-sample basis, and are available even at inference time without ground truth references. This allows to control the quality of reconstructed satellite images, which may be required for safety-critical downstream applications in remote sensing.
If you utilize our data sets in your work, please use the following citations:
@article{sen12mscr,
title = {{Multisensor Data Fusion for Cloud Removal in Global and All-Season Sentinel-2 Imagery}},
author = {Ebel, Patrick and Meraner, Andrea and Schmitt, Michael and Zhu, Xiao Xiang},
journal = {IEEE Transactions on Geoscience and Remote Sensing},
year = {2020},
publisher = {IEEE},
url = {"https://doi.org/10.1109/TGRS.2020.3024744"}
}
@article{sen12mscrts,
title = {{SEN12MS-CR-TS: A Remote Sensing Data Set for Multi-modal Multi-temporal Cloud Removal}},
author = {Ebel, Patrick and Xu, Yajin and Schmitt, Michael and Zhu, Xiao Xiang},
journal = {IEEE Transactions on Geoscience and Remote Sensing},
year = {2022},
publisher = {IEEE},
url = {"https://doi.org/10.1109/TGRS.2022.3146246"}
}
@inproceedings{uncrtaints,
title = {{UnCRtainTS: Uncertainty Quantification for Cloud Removal in Optical Satellite Time Series}},
author = {Ebel, Patrick and Garnot, Vivien Sainte Fare and Schmitt, Michael and Wegner, Jan and Zhu, Xiao Xiang},
booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops},
year = {2023},
organization = {IEEE},
url = {"https://arxiv.org/abs/2304.05464"}
}
You can find my dissertation, enveloping all of our work on cloud removal, publicly accessible here.
Beyond this, you may also be interested in our following related studies:
© Patrick Ebel www.pwjebel.com.