MineNetCD: A Benchmark for Global Mining Change Detection on Remote Sensing Imagery
Monitoring land changes triggered by mining activities is crucial for industrial control, environmental management and regulatory compliance, yet it poses significant challenges due to the vast and often remote locations of mining sites. Remote sensing technologies have increasingly become indispensable to detect and analyze these changes over time. We thus introduce MineNetCD, a comprehensive benchmark designed for global mining change detection using remote sensing imagery. The benchmark comprises three key contributions. First, we establish a global mining change detection dataset featuring more than 70k paired patches of bitemporal high-resolution remote sensing images and pixel-level annotations from 100 mining sites worldwide. Second, we develop a novel baseline model based on a change-aware Fast Fourier Transform (ChangeFFT) module, which enhances various backbones by leveraging essential spectrum components within features in the frequency domain and capturing the channel-wise correlation of bitemporal feature differences to learn change-aware representations. Third, we construct a unified change detection (UCD) framework that currently integrates 20 change detection methods. This framework is designed for streamlined and efficient processing, utilizing the cloud platform hosted by HuggingFace.
Related Work:
W. Yu, X. Zhang, R. Gloaguen, X. X. Zhu and P. Ghamisi, "MineNetCD: A Benchmark for Global Mining Change Detection on Remote Sensing Imagery," in IEEE Transactions on Geoscience and Remote Sensing, doi: 10.1109/TGRS.2024.3491715.
Responsible AI for Earth Observation
The convergence of artificial intelligence (AI) and Earth observation (EO) technologies has brought geoscience and remote sensing into an era of unparalleled capabilities. AI's transformative impact on data analysis, particularly derived from EO platforms, holds great promise in addressing global challenges such as environmental monitoring, disaster response and climate change analysis. However, the rapid integration of AI necessitates a careful examination of the responsible dimensions inherent in its application within these domains. In this paper, we represent a pioneering effort to systematically define the intersection of AI and EO, with a central focus on responsible AI practices. Specifically, we identify several critical components guiding this exploration from both academia and industry perspectives within the EO field: AI and EO for social good, mitigating unfair biases, AI security in EO, geo-privacy and privacy-preserving measures, as well as maintaining scientific excellence, open data, and guiding AI usage based on ethical principles. Furthermore, the paper explores potential opportunities and emerging trends, providing valuable insights for future research endeavors.
Related Work:
P. Ghamisi, W. Yu, A. Marinoni, C. M. Gevaert, C. Persello, S. Selvakumaran, M. Girotto, B. P. Horton, P. Rufin, P. Hostert, F. Pacifici, and P. M. Atkinson, “Responsible ai for earth observation,” 2024.
MaskCD: A Remote Sensing Change Detection Network Based on Mask Classification
Change detection from remote sensing images using deep learning has been widely investigated in the literature. It is typically regarded as a pixel-wise labeling task that aims to classify each pixel as changed or unchanged. Although per-pixel classification networks in encoder-decoder structures have shown dominance, they still suffer from imprecise boundaries and incomplete object delineation at various scenes. For high-resolution RS images, partly or totally changed objects are more worthy of attention rather than a single pixel. Therefore, we revisit the CD task from the mask prediction and classification perspective and propose MaskCD to detect changed areas by adaptively generating categorized masks from input image pairs.
Related Work:
Yu, W., Zhang, X., Das, S., Zhu, X. X., and Ghamisi, P., 2024. MaskCD: A Remote Sensing Change Detection Network Based on Mask Classification. arXiv preprint.
SpectralGPT: Spectral Remote Sensing Foundation Model
The foundation model has recently garnered significant attention due to its potential to revolutionize the field of visual representation learning in a self-supervised manner. While most foundation models are tailored to effectively process RGB images for various visual tasks, there is a noticeable gap in research focused on spectral data, which offers valuable information for scene understanding, especially in remote sensing (RS) applications. To fill this gap, we created a universal RS foundation model named SpectralGPT for the first time. It is purpose-built to handle spectral RS images using a novel 3D generative pretrained transformer (GPT).
Related Work:
Hong, D., Zhang, B., Li, X., Li, Y., Li, C., Yao, J., Yokoya, N., Li, H., Ghamisi, P., Jia, X. and Plaza, A., 2024. SpectralGPT: Spectral remote sensing foundation model. IEEE Transactions on Pattern Analysis and Machine Intelligence.
Universal Adversarial Defense
Deep neural networks have shown remarkable success in remote sensing applications. However, their vulnerability to adversarial perturbations is a critical concern. To address this challenge, our research introduces a novel Universal Adversarial Defense approach (UAD-RS) that uses pre-trained diffusion models to protect common DNNs against unknown adversarial attacks.
Related Work:
W. Yu, Y. Xu, and P. Ghamisi, "Universal Adversarial Defense in Remote Sensing Based on Pre-trained Denoising Diffusion Models," arXiv preprint arXiv:2307.16865, 2023.
Hyperspectral Point Clouds Segmentation
We designed a fusion model for hyperspectral point clouds segmentation. Although there are numerous studies on point cloud segmentation, most of them focus on the typical point cloud data. Segmentation on hyperspectral point cloud data, however, has rarely been investigated. In this study, we present how to design a multi-stream network for multi-modal point cloud.
Related Work:
A. Rizaldy, A.J. Afifi, R. Gloaguen, P. Ghamisi, "Segmentation of Hyperspectral Point Clouds of Open-pit Mines with Multi-Stream Graph Network" under preparation, 2023.
3D Point Clouds Classification and Segmentation
In AI4RS, we also cover researchs on 3D point clouds. Unlike image classification, point cloud classification has different challenges due to the irregularity, unstructured, and unorderedness of the data. Leveraging by the advanced development of deep neural networks, we conducted researchs on various point cloud datasets.
Visual Question Answering
Contemporary approaches to RS VQA often involve resource-intensive techniques, such as full fine-tuning of large models or the extraction of image-text features from pre-trained multimodal models, followed by modality fusion using decoders. These approaches demand significant computational resources and time, and a considerable number of trainable parameters are introduced. To address these challenges, we introduce a novel method known as RSAdapter, which prioritizes runtime and parameter efficiency.
Related Work:
Y. Wang, P. Ghamisi, "RSAdapter: Adapting Multimodal Models for Remote Sensing Visual Question Answering," IEEE Trans. Geosci. Remote Sens., vol. 62, pp. 1−13, 2024.
ChangeMinds: Multi-task Framework for Detecting
and Describing Changes in Remote Sensing
Recent advancements in Remote Sensing (RS) for Change Detection (CD) and Change Captioning (CC) have seen substantial success by adopting deep learning techniques. Despite these advances, existing methods often handle CD and CC tasks independently, leading to inefficiencies from the absence of synergistic processing. In this paper, we present ChangeMinds, a novel unified multi-task framework that concurrently optimizes CD and CC processes within a single, end-to-end model. We propose the change-aware long short-term memory module (ChangeLSTM) to effectively capture complex spatiotemporal dynamics from extracted bi-temporal deep features, enabling the generation of universal change-aware representations that effectively serve both CC and CD tasks. Furthermore, we introduce a multi-task predictor with a cross-attention mechanism that enhances the interaction between image and text features, promoting efficient simultaneous learning and processing for both tasks. Extensive evaluations on the LEVIR-MCI dataset, alongside other standard benchmarks, show that ChangeMinds surpasses existing methods in multi-task learning settings and markedly improves performance in individual CD and CC tasks.
Related Work:
Wang, Y., Yu, W., Kopp, M., & Ghamisi, P. (2024). ChangeMinds: Multi-task Framework for Detecting and Describing Changes in Remote Sensing. arXiv preprint arXiv:2410.10047.
Text-to-Image Generation
Although deep neural networks have achieved great success in many important remote sensing tasks, generating realistic remote sensing images from text descriptions is still very difficult. To address this challenge, we develop novel algorithms for high-resolution remote sensing image synthesis from text descriptions.
Related Work:
Y. Xu, W. Yu, P. Ghamisi, M. Kopp, S. Hochreiter, "Txt2Img-MHN: Remote sensing image generation from text using modern hopfield networks," arXiv preprint arXiv:2208.04441, 2022.
Image/Change Captioning
As an interdisciplinary field at the intersection of CV and NLP, image captioning has gained significant attention in recent years, with many researchers applying advanced artificial intelligence techniques to improve the state of the art. To address the challenge of generating accurate captions, we conduct interesting research for high-resolution remote sensing images.
Related Work:
S. Chang and P. Ghamisi, "Changes to Captions: An Attentive Network for Remote Sensing Change Captioning," arXiv preprint arXiv:2304.01091, 2023.
Adversarial Attacks and Defenses
While deep learning algorithms have achieved state-of-the-art performance in many remote sensing interpretation tasks, their vulnerability to adversarial examples should not be neglected. To tackle this challenge, we develop novel adversarial attack and defense methods and collect benchmark datasets for remote sensing images.
Related Work:
Y. Xu, P. Ghamisi, "Universal adversarial examples in remote sensing: Methodology and benchmark," IEEE Trans. Geosci. Remote Sens., vol. 60, pp. 1−15, 2022.
Natural Hazards Monitoring
Natural hazards are naturally occurring phenomena of geological, hydrological, or meteorological origin that might harm humans or the environment. To monitor and analyze natural hazards like landslides, we develop novel algorithms and collect benchmark datasets with multi-source remote sensing data.
Related Work:
O. Ghorbanzadeh, Y. Xu, P. Ghamisi, M. Kopp, D. Kreil, "Landslide4sense: Reference benchmark data and deep learning models for landslide detection," arXiv preprint arXiv:2206.00515, 2022.
AI for Social Good
New developments and a plethora of applications are already changing the way humanity will face the challenges of our planet. To better monitor and achieve the sustainable development goals, we develop novel AI approaches for social good.
Related Work:
C. Persello, J. Dirk Wegner, R. Hänsch, D. Tuia, P. Ghamisi, M. Koeva, G. Camps-valls, "Deep learning and earth observation to support the sustainable development goals: Current approaches, open challenges, and future opportunities," IEEE Geosci. Remote Sens. Mag., 2022.
Image Restoration
The quality of data acquired by remotely sensed imaging sensors (active and passive) is often degraded by a variety of noise types and artifacts. To tackle this challenge, we develop novel image restoration methods for remote sensing data.
Related Work:
B. Rasti, Y. Chang, E. Dalsasso, L. Denis, P. Ghamisi, "Image restoration for remote sensing: Overview and toolbox," IEEE Geosci. Remote Sens. Mag., 2021.
Spectral Unmixing
Due to the limited spatial resolution and scattering of the light, a pixel spectrum is generally a complex mixture of the pure spectra of its constituent materials, i.e., the endmember spectra. To accurately estimate the fractional abundances of those endmembers, we develop advanced deep learning-based unmixing methods.
Related Work:
B. Rasti, B. Koirala, P. Scheunders, P. Ghamisi, "Undip: Hyperspectral unmixing using deep image prior," IEEE Trans. Geosci. Remote Sens., vol. 60, pp. 1−15, 2022.
Data Clustering
Traditional clustering models essentially work in the Euclidean domain, leading to insufficient handling of graph-structured data. To tackle this challenge, we develop novel graph neural network-based approaches for high-dimensional remote sensing data.
Related Work:
Y. Cai, Z. Zhang, Z. Cai, X. Liu, Y. Ding, P. Ghamisi, "Fully linear graph convolutional networks for semi-supervised learning and clustering," arXiv preprint arXiv:2111.07942, 2021.
Learning with Sparse Annotations
Most of the existing interpretation approaches, especially deep learning-based ones, require a large number of high-quality labeled samples, which are expensive and time-consuming to be collected in practice. To tackle this challenge, we develop novel algorithms that can achieve satisfactory performance with sparse annotations.
Related Work:
Y. Xu, P. Ghamisi, "Consistency-regularized region-growing network for semantic segmentation of urban scenes with point-level annotations," IEEE Trans. Image Process., vol. 31, pp. 5038–5051, 2022.
​
J. Yue et al., "Optical Remote Sensing Image Understanding With Weak Supervision: Concepts, methods, and perspectives," IEEE Geoscience and Remote Sensing Magazine, vol. 10, no. 2, pp. 250-269, June 2022,
Anomaly (Change) Detection
Traditional representation-based detectors suffer from high computation burden and can hardly explore the relationship among different instances. To tackle this challenge, we develop novel collaborative representation methods for high-dimensional remote sensing data without supervision.
Related Work:
S. Chang, P. Ghamisi, "Nonnegative-constrained joint collaborative representation with union dictionary for hyperspectral anomaly detection," IEEE Trans. Geosci. Remote. Sens., vol. 60, pp. 1-13, 2022.
​
S. Chang, M. Kopp, and P. Ghamisi, "Sketched Multiview Subspace Learning for Hyperspectral Anomalous Change Detection," IEEE Trans. Geosci. Remote. Sens., vol. 60, pp. 1-12, 2022.
Data Fusion
Remote sensing data acquired from different sensors generally possess complementary information which is beneficial to the interpretation of ground objects. To make full use of this advantage, we develop novel data fusion approaches with deep learning for multi-source remote sensing data.
Related Work:
K. Rafiezadeh Shahi, P. Ghamisi, B. Rasti, P. Scheunders, R. Gloaguen, "Unsupervised data fusion with deeper perspective: A novel multisensor deep clustering algorithm," IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., vol. 15, pp. 284−296, 2022.
​
P. Ghamisi et al., "Multisource and Multitemporal Data Fusion in Remote Sensing: A Comprehensive Review of the State of the Art," IEEE Geoscience and Remote Sensing Magazine, vol. 7, no. 1, pp. 6-39, March 2019.