Heterogeneous Change Detection on Remote Sensing Data with Self-Supervised Deep Canonically Correlated Autoencoders





remote sensing, earth observation, change detection, multimodal image analysis, machine learning, deep learning, neural networks, canonical correlation analysis


Change detection is a well-known topic of remote sensing. The goal is to track and monitor the evolution of changes affecting the Earth surface over time. The recently increased availability in remote sensing data for Earth observation and in computational power has raised the interest in this field of research. In particular, the keywords “multitemporal” and “heterogeneous” play prominent roles. The former refers to the availability and the comparison of two or more satellite images of the same place on the ground, in order to find changes and track the evolution of the observed surface, maybe with different time sensitivities. The latter refers to the capability of performing change detection with images coming from different sources, corresponding to different sensors, wavelengths, polarizations, acquisition geometries, etc. This thesis addresses the challenging topic of multitemporal change detection with heterogeneous remote sensing images. It proposes a novel approach, taking inspiration from recent developments in the literature. The proposed method is based on deep learning - involving autoencoders of convolutional neural networks - and represents an exapmple of unsupervised change detection. A major novelty of the work consists in including a prior information model, used to make the method unsupervised, within a well-established algorithm such as the canonical correlation analysis, and in combining these with a deep learning framework to give rise to an image translation method able to compare heterogeneous images regardless of their highly different domains. The theoretical analysis is supported by experimental results, comparing the proposed methodology to the state of the art of this discipline. Two different datasets were used for the experiments, and the results obtained on both of them show the effectiveness of the proposed method.


Download data is not yet available.


Metrics Loading ...

Author Biography

Federico Figari Tomenotti, University of Genoa

Electrical, Electronics and Telecommunications Engineering and Naval Architecture Department


Allen-Zhu, Zeyuan ; Li, Yuanzhi ; Song, Zhao: A Convergence Theory for Deep Learning via Over-Parameterization. In: Chaudhuri, Kamalika (Hrsg.) ; Salakhutdinov, Ruslan (Hrsg.): Proceedings of the 36th International Conference on Machine Learning Bd. 97. Long Beach, California, USA : PMLR, 09–15 Jun 2019, S. 242–252

Alpaydin, Ethem: Introduction to machine learning. MIT press, 2014

Andrew, Galen ; Arora, Raman ; Bilmes, Jeff ; Livescu, Karen: Deep canonical correlation analysis. In: International conference on machine learning, 2013, S. 1247–1255

Bovolo, Francesca ; Bruzzone, Lorenzo: The Time Variable in Data Fusion: A Change Detection Perspective. In: IEEE Geoscience and Remote Sensing Magazine 3 (2015), 09, S. 8–26. https://doi.org/10.1109/MGRS.2015.2443494

Cohen, J.: A coefficient of agreement for nominal scales. In: Educational and Psychological Measurement 20 (1960), S. 37–46. https://doi.org/10.1177/001316446002000104

Csaji ´ , Bal´azs C.: Approximation with Artificial Neural Networks, Faculty of Sciences; E¨otv¨os Lor´and University, Hungary, Diplomarbeit, 2001

Cybenko, G: Approximation by superpositions of a sigmoidal function. In: Mathematics of Control, Signals and Systems 4 (1989), December, S. 303, 314. – https://doi.org/10.1007/BF02551274

Figari Tomenotti, Federico ; Luppino, Luigi T. ; Hansen, Mads A. ; Moser, Gabriele ; Anfinsen, Stian N.: Heterogeneous Change Detection with Self-Supervised Deep Canonically Correlated Autoencoders. (submitted), January

Fung, Tung ; LeDrew, Ellsworth: Application of principal components analysis to change detection. In: Photogrammetric engineering and remote sensing 53 (1987), Nr. 12, S. 1649–1658

Goodfellow, Ian ; Bengio, Yoshua ; Courville, Aaron: Deep Learning. MIT Press, 2016. – http://www.deeplearningbook.org

Govender, Megandhren ; Chetty, Kershani ; Bulcock, Hartley: A review of hyperspectral remote sensing and its application in vegetation and water resource studies. In: Water S.A 33 (2007), 05. https://doi.org/10.4314/wsa.v33i2.49049

Handcock, Rebecca ; Torgersen, Christian ; Cherkauer, Keith ; Gillespie, Alan ; Tockner, Klement ; Faux, Russel ; Tan, Jing: Thermal Infrared Remote Sensing of Water Temperature in Riverine Landscapes. In: Fluvial Remote Sensing for Science and Management (2012), 08, S. 85–113. ISBN 9781119940791

Hardoon, David R. ; Szedmak, Sandor ; ShaweTaylor, John: Canonical correlation analysis: An overview with application to learning methods. In: Neural computation 16 (2004), Nr. 12, S. 2639–2664. https://doi.org/10.1002/9781119940791.ch5

Inglada, Jordi ; Giros, Alain: On the real capabilities of remote sensing for disaster management-feedback from real cases. In: IGARSS 2004. 2004 IEEE International Geoscience and Remote Sensing Symposium Bd. 2 IEEE (Veranst.), 2004, S. 1110–1112

Krähenbühl, Philipp ; Koltun ; Vladlen: Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials. In: Shawe-Taylor, J. (Hrsg.) ; Zemel, R. S. (Hrsg.) ; Bartlett, P. L. (Hrsg.) ; Pereira, F. (Hrsg.) ; Weinberger, K. Q. (Hrsg.): Advances in Neural Information Processing Systems 24. Curran Associates, Inc., 2011, S. 109–117

Lavigne, D. M.: Counting Harp Seals with ultra-violet photography. In: Polar Record 18 (1976), Nr. 114, S. 269–277. https://doi.org/10.1017/S0032247400000310

Liu, J. ; Gong, M. ; Qin, K. ; Zhang, P.: A Deep Convolutional Coupling Network for Change Detection Based on Heterogeneous Optical and Radar Images. In: IEEE Trans. Neural Netw. Learn. Syst. 29 (2018), March, Nr. 3, S. 545–559. https://doi.org/10.1109/TNNLS.2016.2636227

Lu, Zhou ; Pu, Hongming ; Wang, Feicheng ; Hu, Zhiqiang ; Wang, Liwei: The Expressive Power of Neural Networks: A View from the Width. In: Guyon, I. (Hrsg.) ; Luxburg, U. V. (Hrsg.) ; Bengio, S. (Hrsg.) ; Wallach, H. (Hrsg.) ; Fergus, R. (Hrsg.) ; Vishwanathan, S. (Hrsg.) ; Garnett, R. (Hrsg.): Advances in Neural Information Processing Systems 30. Curran Associates, Inc., 2017, S. 6231–6239

Luppino, Luigi T. ; Bianchi, Filippo M. ; Moser, Gabriele ; Anfinsen, Stian N.: Unsupervised Image Regression for Heterogeneous Change Detection. In: IEEE Trans. Geosci. Remote Sens. 57 (2019), Nr. 12, S. 9960–9975. https://doi.org/10.1109/TGRS.2019.2930348

Luppino, Luigi T. ; Kampffmeyer, Michael C. ; Bianchi, Filippo M. ; Jenssen, Robert ; Moser, Gabriele ; Serpico, Sebastiano B. ; Anfinsen, Stian N.: Deep image translation with an affinity-based change prior for unsupervised multimodal change detection. Oct 2020. – arXiv:2001.04271. https://doi.org/10.1109/TGRS.2021.3056196

Maas, Andrew L. ; Hannun, Awni Y. ; Ng, Andrew Y.: Rectifier nonlinearities improve neural network acoustic models. In: Proc. icml Bd. 30, 2013, S. 3

Mardia, Kantilal Varichand ; Kent, John T. ; Bibby, John M.: Multivariate analysis. London [u.a.] : Acad. Press, 1979 (Probability and mathematical statistics). – ISBN 0124712509

Mercier, G. ; Moser, G. ; Serpico, S. B.: Conditional Copulas for Change Detection in Heterogeneous Remote Sensing Images. In: IEEE Transactions on Geoscience and Remote Sensing 46 (2008), May, Nr. 5, S. 1428–1441. – ISSN 1558-0644. https://doi.org/10.1109/TGRS.2008.916476

Minnett, P.J. ; Alvera-Azcarate ´ , A. ; Chin, T.M. ; Corlett, G.K. ; Gentemann, C.L. ; Karagali, I. ; Li, X. ; Marsouin, A. ; Marullo, S. ; Maturi, E. ; Santoleri, R. ; Picart, S. S. ; Steele, M. ; Vazquez-Cuervo, J.: Half a century of satellite remote sensing of sea-surface temperature. In: Remote Sensing of Environment 233 (2019), S. 111366. – ISSN 0034-4257. https://doi.org/10.1016/j.rse.2019.111366

Moser, G. ; Serpico, S. B.: Generalized minimum-error thresholding for unsupervised change detection from SAR amplitude imagery. In: IEEE Transactions on Geoscience and Remote Sensing 44 (2006), Oct, Nr. 10, S. 2972–2982. – ISSN 1558-0644. https://doi.org/10.1109/TGRS.2006.876288

Moser, Gabriele ; Serpico, Sebastiano B. ; Benediktsson, Jon A.: Land-cover mapping by Markov modeling of spatial–contextual information in very-high-resolution remote sensing images. In: Proceedings of the IEEE 101 (2012), Nr. 3, S. 631–651. https://doi.org/10.1109/JPROC.2012.2211551

NASA: Missions - Climate Observation. – URL https://climate.nasa.gov/nasa_science/missions. – website visited 26/02/2020

Niu, X. ; Gong, M. ; Zhan, T. ; Yang, Y.: A Conditional Adversarial Network for Change Detection in Heterogeneous Images. In: IEEE Geosci. Remote Sens. Lett. 16 (2019), Jan, Nr. 1, S. 45–49. https://doi.org/10.1109/LGRS.2018.2868704

Otsu, N.: A Threshold Selection Method from Gray-Level Histograms. In: IEEE Transactions on Systems, Man, and Cybernetics 9 (1979), Jan, Nr. 1, S. 62–66. – ISSN 2168-2909. https://doi.org/10.1109/TSMC.1979.4310076

Racah, Evan ; Beckham, Christopher ; Maharaj, Tegan ; Kahou, Samira E. ; Prabhat ; Pal, Christopher J.: ExtremeWeather: A large-scale climate dataset for semi-supervised detection, localization, and understanding of extreme weather events. In: NIPS, 2016, S.

Volpi, Michele: Kernel-based methods for change detection in remote sensing images, Faculte des Geosciences et de l’Environnement, Dissertation, 2013

Volpi, Michele ; Camps-Valls, Gustau ; Tuia, Devis: Spectral alignment of multi-temporal cross-sensor images with automated kernel canonical correlation analysis. In: ISPRS Journal of Photogrammetry and Remote Sensing 107 (2015), 03. https://doi.org/10.1016/j.isprsjprs.2015.02.005

Wang, Weiran ; Arora, Raman ; Livescu, Karen ; Bilmes, Jeff: On deep multi-view representation learning. In: International Conference on Machine Learning, 2015, S. 1083–1092

Zhan, Tao ; Gong, Maoguo ; Jiang, Xiangming ; Li, Shuwei: Log-Based Transformation Feature Learning for Change Detection in Heterogeneous Images. In: IEEE Geosci. Remote Sens. Lett. 15 (2018), Nr. 9, S. 1352–1356. https://doi.org/10.1109/LGRS.2018.2843385

Zhou, Yuan ; Liu, Hui ; Li, Dan ; Cao, Hai ; Yang, Jing ; Li, Zizi: Cross-Sensor Image Change Detection Based on Deep Canonically Correlated Autoencoders. S. 251–257. In: Artificial Intelligence for Communications and Networks, 07 2019. – ISBN 978-3-030-22967-2. https://doi.org/10.1007/978-3-030-22968-9_22

Zhu, Jun-Yan ; Park, Taesung ; Isola, Phillip ; Efros, Alexei A.: Unpaired Image-To-Image Translation Using Cycle-Consistent Adversarial Networks. In: The IEEE International Conference on Computer Vision (ICCV), Oct 2017, S. 2223–2232. https://doi.org/10.1109/ICCV.2017.244




How to Cite

Figari Tomenotti, F. (2021). Heterogeneous Change Detection on Remote Sensing Data with Self-Supervised Deep Canonically Correlated Autoencoders. Septentrio Reports, (4). https://doi.org/10.7557/7.5763