Learning approaches for fast image interpolation and super-resolution

Abstract

Image interpolation and image super-resolution (SR) are fundamental and essential problems in image processing. They share a common purpose which is to enhance the resolution of a low-resolution (LR) digital image, as well as have some dissimilarities. Image interpolation aims to predict missing high-resolution (HR) pixels from the neighboring known ground truth HR pixels, while the objective of image super-resolution is to generate a HR image which preserves sharp edges and natural textures from a blurred and down-sampled version of the original HR image. The low-quality and small digital images become increasingly undesirable when the resolution of the display screens keeps growing. However, the existing image interpolation and image super-resolution algorithms either produce unsatisfactory HR images with blurry edges and annoying artifacts or require too long processing time to realize practical applications. We propose a two-stage framework for fast image interpolation via random forests (FIRF). The proposed FIRF method gives high accuracy, as well as requires low computation. The underlying idea of this proposed work is to apply random forests to classify the natural image patch space into numerous subspaces and learn a linear regression model for each subspace to map the LR image patch to HR image patch. The FIRF framework consists of two stages. Stage 1 of the framework removes most of the ringing and aliasing artifacts in the initial Bi-cubic interpolated image, while Stage 2 further refines the Stage 1 interpolated image. By varying the number of decision trees in the random forests and the number of stages applied, the proposed FIRF method can realize computationally scalable image interpolation. Extensive experimental results show that the proposed FIRF(3,2) method achieves more than 0.3 dB improvement in PSNR over the state-of-the-art Nonlocal Auto-Regressive Modeling (NARM) method. Moreover the proposed FIRF(1,1) obtains similar or better results as NAMR while requires only 0.3% of its computational time. We also propose to use a set of decision tree strategies for fast and high quality image SR. We take the divide-and-conquer strategy using decision tree for super-resolution (SRDT) which performs a few simple binary tests to classify an input LR patch into one of the leaf nodes and directly multiplies this LR patch with the regression model at that leaf node for regression. Both the classification process and the regression process take extremely small amount of computation. We formulate a hierarchical decision trees (SRHDT) method which cascades multiple layers of super-resolution decision trees to further boost the SR results. Inspired by the random forests approach which combines regression models from an ensemble of decision trees, we propose a hierarchical decision trees approach with fused regression models (SRHDT_f), which fuses the regression models from 4 relevant leaf nodes within the same decision tree to form a more robust regression model. This achieves another 0.1 dB improvement. Our experimental results show that our initial approach, the SRDT method achieves comparable SR results as the sparse representation based method and the deep learning based method but the speed of our method is much faster. Furthermore, our enhanced version, the SRHDT_f method achieves more than 0.3 dB higher PSNR over that of the A+ method which is the state-of-the-art method in SR. Hybrid DCT-Wiener-Based interpolation scheme using the learnt Wiener filter can significantly improve both objective and subjective performance by learning a suitable Wiener filter to fit the hybrid scheme with a good mix of spatial and transform domain process. Using the adaptive k-NN MMSE estimation for each block achieves extraordinary up-sampling results. However, it needs a large database and relatively long processing time. We have also investigated using multiple learnt Wiener filters and combined the information from both external training images and original low-resolution image. The proposed dual MMSE estimators adaptively resolve the problem of one general learnt Wiener filter and use less computation time compared with that of the k-NN MMSE estimation. Experimental results show that the proposed dual MMSE estimators achieve around 1dB PSNR improvement compared to the original hybrid DCT-Wiener-Based scheme and provide more natural visual quality.