Learning based methods for color constancy and image enhancement

Abstract

With the fast development of camera devices and social media, images are nowadays one of the most widely used media in our daily life. However, during the acquisition, formation and transmission processes, images are prone to various types of corruptions, leading to degradation in image quality. The on-camera image signal processing (ISP) algorithms and the image enhancement methods are too crucial to ensure and improve the quality of camera output images. Plenty of efforts have been devoted to the research of ISP and image enhancement, and the recently developed deep learning technique has achieved prominent results in these areas. In this thesis, we leverage deep learning for several fundamental tasks in camera ISP pipeline and image enhancement. Color constancy is the foremost unit in ISP to correct the color bias of the captured images to cater to the human vision system. In chapter 2, we introduce a multi-domain learning strategy for color constancy to relief from lacking training data by leveraging cross-device datasets. Our method achieves state-of-art performance on the commonly used benchmark datasets. Particularly, our model is capable of transferring to a new device with merely a few training samples, which largely reduces the cost of time-consuming data acquisition stage for camera manufacturers when developing color constancy models for new devices. Image diffraction blurring is another type of deterioration which blurs the image and degrades the image quality. In chapter 3, we conduct a pioneer work by constructing a real-world diffraction blur dataset. With the constructed real-world dataset, we further design a progressive learning strategy and a robust loss function to train a deep convolutional neural network for diffraction blur removal. Our model can effectively recover more textures and details from images with diffraction blur than the general image deblurring methods. Single image super-resolution (SISR) is a fundamental task in image enhancement, which aims to increase the resolution of given images. In this thesis, we focus on the more challenging real-world SISR task, where the image degradation process is much more complicated and unknown. In chapter 4, we learn the degradation model from existing real-world SISR datasets, and use the learned degradation model to synthesize large scale realistic training image pairs. By using the generated realistic SISR image pairs, more robust SISR models can be trained, which exhibit higher generalization performance than previous SISR models, presenting promising visual quality for real-world images. In chapter 5, we further investigate the real-world SISR problem. We work from another perspective, i.e., designing blind super-resolution models. Specifically, we first estimate the pixel-wise degradation map of the given image, and then utilize a deep CNN whose local filters are dependent on estimated degradation to achieve super-resolution. Our method is able to handle complex non-uniform image degradations in real-world scenarios and achieves leading performance on a wide variety of real-world images with good runtime efficiency. In summary, in this thesis we tackle several important tasks in camera ISP and image enhancement by leveraging deep learning techniques. Our methods demonstrate state-of-art performances on these tasks.