Machine learning for image super-resolution in real-world applications

Abstract

Single image super-resolution (SISR) is a fundamental task in computer vision, which aims to reconstruct high-resolution images from their low-resolution counterparts. In recent years, deep image SR methods have achieved remarkable performance, but there are still some challenging issues limiting their applications in real-world situations. In this thesis, we will investigate three challenging issues, including the degradation mismatch, high computational complexity, and the trade-off between the distortion and perceptual quality, for existing deep image SR models in real-world applications. In Chapter 3, we will focus on the degradation mismatch between real-world and synthetic degradations. Two specific degradation models are considered, i.e., the mixture of multiple degradations and the multiple-focal-length degradation. A deep progressive network is proposed to address the mixture of degradations, which leverages prior information from small-scale images to restore corrupted contents of large-scale images. In addition, we propose a deep feature mixture model for the multiple-focal-length degradation. The proposed model can adaptively aggregate local contextual information extracted from different scales for reconstruction. Furthermore, we propose a novel loss function, based on the Laplacian pyramid, to improve the performance. Experiments show that our proposed methods significantly outperform other promising image SR models. Then, we will introduce two proposed deep lightweight image SR models for resource-constrained devices in Chapter 4. The first method adopts the proposed lightweight compression-and-extraction module, which avoids unnecessary costs for redundant features. The second lightweight model is based on our proposed convolutional kernel which is dynamically generated in each pixel location. In addition, it shares kernel weights across the channels for feature extraction. Experiments show that our proposed methods can significantly reduce the number of model parameters, while maintaining the performance, leading to a better trade-off between the performance and the model complexity than other lightweight models. Next, in Chapter 5, we will focus on the trade-off between distortion and perceptual quality for super-resolved images. To address this issue, we propose an image-fusion method, which provides a novel interpolation method in the Wasserstein space. This novel interpolation method can effectively balance the distortion and perceptual quality of the images by manipulating their high-frequency components in the wavelet domain. Experiments show that our proposed method significantly improves the visual quality, while maintaining the distortion quality, leading to a better performance than compared methods. In addition, we show that our proposed method can be applied to any deep image SR model and solved very efficiently with CPU only. Finally, we will propose an extremely low-latency online video SR model for online applications in Chapter 6. A novel kernel knowledge transfer method is proposed to enhance a simple, lightweight video SR model, in terms of the distortion quality and latency. Experiments show that our proposed method can achieve remarkable performance, compared with other promising methods. In addition, we will show that our proposed method can process the video sequence at a rate of up to 110 FPS, leading to the best trade-off between the performance and model complexity.