Leveraging pretrained diffusion model for semantic 3-D reconstruction from monocular remote sensing image

Abstract

Semantic 3D reconstruction from monocular imagery serves as a cost-effective tool for many urban applications, such as energy system modeling, resilience analysis, and urban planning. However, the generalization of task-specific models for semantic 3D reconstruction remains limited by the available data scale and diversity. In contrast, visual foundation models (VFMs) are trained on large-scale, diverse datasets, enabling stronger adaptability and richer visual knowledge across different tasks. Unlike most VFMs that focus on discrimination or feature extraction, pretrained diffusion models (PDMs) are generative, combining high-level semantic understanding with the ability to produce high-fidelity details and textures. Building upon these advantages, this study proposes a novel task-adaptive framework that harnesses PDMs for semantic 3D reconstruction from monocular remote sensing images. Our framework employs low-rank adaptation to efficiently fine-tune the denoising network, effectively modeling the high-dimensional features required for semantic 3D reconstruction while only training a minimal fraction of parameters. We further design a lightweight, task-specific decoder to map these features into target elevation and semantic maps. In addition, we introduce an evidential height regression method, which incorporates uncertainty awareness into height estimation without introducing additional computational overhead. Experiments on the public US3D JAX and Open Data DC datasets demonstrate that our framework significantly outperforms other existing methods in both subtasks of height estimation and semantic segmentation, achieving high-fidelity semantic 3D reconstruction of remote sensing scenes. This technology holds significant potential for advancing urban modeling, enabling more accurate and efficient large-scale geographic analysis.