PolyGNN: Polyhedron-based Graph Neural Network for 3D Building Reconstruction from Point Clouds

We present PolyGNN, a polyhedron-based graph neural network for 3D building reconstruction from point clouds. PolyGNN learns to assemble primitives obtained by polyhedral decomposition via graph node classification, achieving a watertight and compact reconstruction. To effectively represent arbitrary-shaped polyhedra in the neural network, we propose a skeleton-based sampling strategy to generate polyhedron-wise queries. These queries are then incorporated with inter-polyhedron adjacency to enhance the classification. PolyGNN is end-to-end optimizable and is designed to accommodate variable-size input points, polyhedra, and queries with an index-driven batching technique. To address the abstraction gap between existing city-building models and the underlying instances, and provide a fair evaluation of the proposed method, we develop our method on a large-scale synthetic dataset with well-defined ground truths of polyhedral labels. We further conduct a transferability analysis across cities and on real-world point clouds. Both qualitative and quantitative results demonstrate the effectiveness of our method, particularly its efficiency for large-scale reconstructions. The source code and data are available at https://github.com/chenzhaiyu/polygnn.

Reference: Zhaiyu Chen, Yilei Shi, Liangliang Nan, Zhitong Xiong, Xiao Xiang Zhu (2024). PolyGNN: Polyhedron-based Graph Neural Network for 3D Building Reconstruction from Point Clouds. ISPRS Journal of Photogrammetry and Remote Sensing, 218 (2024): 693-706 (link).

Assessing the Macro-scale Patterns of Urban Tree Canopy Cover in Brazil Using High-resolution Remote Sensing Images

Urban trees typically provide many environmental, social, and economic urban ecosystem services to people living in urban areas. Along streets, they mitigate air and noise pollution and urban forests provide a habitat for wildlife, such as insects, birds, and serrated animals, while also providing a cool living environment by mitigating urban heat. Therefore, studying patterns of UTC cover is important to promote environmental justice and support urban sustainable development in Brazil. To better understand the UTC cover patterns in Brazil, this study creates UTC products for 472 cities in Brazil based on high-resolution satellite images and an advanced CNN segmentation model and then conducts a comprehensive assessment of macro-scale patterns of UTC cover in Brazil by taking tree canopy coverage, human exposure, and determines into account. The results revealed that the UTC cover of Brazilian cities is spatially heterogeneous, ranging from 5% to 34%. There was a difference in UTC coverage between the old and new urban areas, with the average largest difference near 5%. More than 76% urban population exposure to UTC coverage of 0~0.2. Most cities have a relatively high inequality in human exposure to urban tree-covered spaces, especially in northeastern and southeastern Brazil. Results from the geographical detector models show climatic factors play a major role in determining the UTC cover patterns in Brazilian cities, followed by socioeconomic, geographical, soil, and urbanization factors. This study suggests the Brazilian government pay more attention to greening renovation projects in old urban areas and formulate effective urban tree irrigation policies for cities with limited autumn and winter rainfall. The study also suggests follow-up research on UTC cover patterns that consider the effects of race, urban history, city structure, land use, and local government policy factors to further support the goals of sustainable development in Brazilian cities.     

Reference: Jianhua Guo, Zhiheng Liu, Xiao Xiang Zhu (2023). Assessing the Macro-scale Patterns of Urban Tree Canopy Cover in Brazil Using High-resolution Remote Sensing Images. Sustainable Cities and Society, 100, 105003.(link)

HTC-DC Net: Monocular Height Estimation from Single Remote Sensing Images

Three-dimensional geoinformation is of great significance for understanding the living environment; however, 3-D perception from remote sensing data, especially on a large scale, is restricted, mainly due to the high costs of 3-D sensors such as light detection and ranging (LiDAR). To tackle this problem, we propose a method for monocular height estimation from optical imagery, which is currently one of the richest sources of remote sensing data. As an ill-posed problem, monocular height estimation requires well-designed networks for enhanced representations to improve the performance. Moreover, the distribution of height values is long-tailed with the low-height pixels, e.g., the background (BG), as the head, and thus, trained networks are usually biased and tend to underestimate building heights. To solve the problems, instead of formalizing the problem as a regression task, we propose HTC-DC Net following the classification–regression paradigm, with the head-tail cut (HTC) and the distribution-based constraints (DCs) as the main contributions. HTC-DC Net is composed of the backbone network as the feature extractor, the HTC-AdaBins module, and the hybrid regression process. The HTC-AdaBins module serves as the classification phase to determine bins adaptive to each input image. It is equipped with a vision transformer (ViT) encoder to incorporate local context with holistic information and involves an HTC to address the long-tailed problem in monocular height estimation for balancing the performances of foreground (FG) and BG pixels. The hybrid regression process does the regression via the smoothing of bins from the classification phase, which is trained via DCs. The proposed network is tested on three datasets of different resolutions, namely ISPRS Vaihingen (0.09 m), Data Fusion Contest 19 (DFC19) (1.3 m), and Global Building Height (GBH) (3 m). The experimental results show the superiority of the proposed network over existing methods by large margins. Extensive ablation studies demonstrate the effectiveness of each design component

Reference: Sining Chen, Yilei Shi, Zhitong Xiong, Xiao Xiang Zhu (2023). HTC-DC Net: Monocular Height Estimation from Single Remote Sensing Images. IEEE Transactions Geoscience and Remote Sensing, 61 (2023): 1-18. (link)

From Easy to Hard: Learning Language-guided Curriculum for Visual Question Answering on Remote Sensing Data.

Visual question answering (VQA) for remote sensing scene has great potential in intelligent human-computer interaction system. Although VQA in computer vision has been widely researched, VQA for remote sensing data (RSVQA) is still in its infancy. There are two characteristics that need to be specially considered for the RSVQA task. 1) No object annotations are available in RSVQA datasets, which makes it difficult for models to exploit informative region representation; 2) There are questions with clearly different difficulty levels for each image in the RSVQA task. Directly training a model with questions in a random order may confuse the model and limit the performance. To address these two problems, in this paper, a multi-level visual feature learning method is proposed to jointly extract language-guided holistic and regional image features. Besides, a self-paced curriculum learning (SPCL)-based VQA model is developed to train networks with samples in an easy-to-hard way. To be more specific, a language-guided SPCL method with a soft weighting strategy is explored in this work. The proposed model is evaluated on three public datasets, and extensive experimental results show that the proposed RSVQA framework can achieve promising performance.

Reference: Zhenghang Yuan, Lichao Mou, Qi Wang, Xiao Xiang Zhu (2022). From Easy to Hard: Learning Language-guided Curriculum for Visual Question Answering on Remote Sensing Data. IEEE Transactions on Geoscience and Remote Sensing, 60 (2022): 1-11. (link)

γ-Net: Superresolving SAR Tomographic Inversion via Deep Learning

Synthetic aperture radar tomography (TomoSAR) has been extensively employed in 3-D reconstruction in dense urban areas using high-resolution SAR acquisitions. Compressive sensing (CS)-based algorithms are generally considered as the state-of-the art in super-resolving TomoSAR, in particular in the single look case. This superior performance comes at the cost of extra computational burdens, because of the sparse reconstruction, which cannot be solved analytically, and we need to employ computationally expensive iterative solvers. In this article, we propose a novel deep learning-based superresolving TomoSAR inversion approach, γ -Net, to tackle this challenge. γ -Net adopts advanced complex-valued learned iterative shrinkage thresholding algorithm (CV-LISTA) to mimic the iterative optimization step in sparse reconstruction. Simulations show the height estimate from a well-trained γ -Net approaches the Cramér-Rao lower bound (CRLB) while improving the computational efficiency by one to two orders of magnitude comparing to the first-order CS-based methods. It also shows no degradation in the super-resolution power comparing to the state-of-the-art second-order TomoSAR solvers, which are much more computationally expensive than the first-order methods. Specifically, γ -Net reaches more than 90% detection rate in moderate super-resolving cases at 25 measurements at 6 dB SNR. Moreover, simulation at limited baselines demonstrates that the proposed algorithm outperforms the second-order CS-based method by a fair margin. Test on real TanDEM-X data with just
six interferograms also shows high-quality 3-D reconstruction with high-density detected double scatterers.

Reference: Kun Qian, Yuanyuan Wang, Yilei Shi, Xiao Xiang Zhu (2022). γ-Net: Superresolving SAR Tomographic Inversion via Deep Learning. IEEE Transactions on Geoscience and Remote Sensing, 60, pp. 1–16. (link)