Hybrid panoramic visual SLAM and point cloud color mapping

Abstract

Visual SLAM (Simultaneous Localization and Mapping) asks if it is possible for a mobile platform with cameras to incrementally build a consistent map of an unknown environment while simultaneously determining its location within this map. Localization and mapping have become the fundamental technology both in traditional applications such as surveying and modeling, and in new applications such as robotics, autonomous driving, and VR/AR. However, the drawbacks and gaps of traditional visual SLAM are (1) tracking failure in complex scenes, (2) the wrong optimization caused by loop closing problems, and (3) sparse, colorless, noisy 3D maps generated that are useless in most applications. The robustness and accuracy of localization and mapping can be significantly improved by filling the three gaps. The thesis proposed a localization framework based on visual SLAM and a mapping framework based on point cloud color mapping to close the three gaps. The proposed localization framework of Hybrid Panoramic Visual SLAM integrates DF (Direct and Feature-based) pose estimation and target-based method for panoramic visual SLAM. Experiments show that the robustness of vSLAM is enhanced by the panoramic model and vSLAM using the newly emerged commercial dual-fisheye camera. The accuracy of vSLAM is improved by a target-based method and DF pose estimation. The proposed mapping framework of point cloud color mapping includes timestamp alignment, pose interpolation, video segmentation, occlusion detection, frame-to-frame colorization, highlight detection, ground inpainting, and point cloud post-processing. The experiments show that the framework can be used in various scenes to generate a vivid and photo-realistic map. In summary, the thesis is a systematic study on technologies of vSLAM- based localization and point cloud color mapping. Visual SLAM is performed first to estimate the pose and trajectory of the platform, and then point cloud color mapping takes full advantage of the localization results to generate a vivid and photo-realistic colorized point cloud. A prototype backpack system is designed to validate the proposed two frameworks. The two frameworks cover a complete process of visual simultaneous localization and mapping, which can be used for surveying, modeling, robotics, autonomous driving, and VR/AR.