Blind deep restoration : from face to natural images

Abstract

Blind image restoration (BIR) aims to generate photo-realistic results on the real-world degraded observations. Although it is valuable in many applications, the complex textures and difficulties in simulating the real-world degradation make it still challenging to perform on practical scenarios. Notably, face image, which is a specific category of natural image, owns strong structure prior, while blind face image restoration is not well investigated. In this thesis, we firstly explore the BIR of face images by utilizing their structure prior, which would be beneficial to restore face images with unknown degradation types. In contrast, natural images have more complex structures, thus are more intractable for BSR. We observe that they usually share the same degradation with face images. Based on this, we further extend BSR to natural images by learning the real-world degradation from face regions to synthesize their real-world degraded natural images. The main research contents can be summarized as follows: (1) To restore the low-quality (LQ) face images with unknown degradation by embedding facial structure prior, we introduce a high-quality (HQ) reference image of the same identity to guide the blind restoration process, and develop a GFRNet. Since the HQ reference and the LQ input usually have different poses, in this thesis we employ a semi-supervised manner to predict the optical flow, which is utilized to solve the spatial misalignment. Subsequently, the warped reference together with the LQ input are taken into the restoration network for accurate texture transformation. (2) The inconsistent expressions and poses brought by single reference may lead to limited improvements. Actually, one person usually has multiple HQ face images with different poses. We then extend the single reference to multiple exemplars for guided restoration, and develop an ASFFNet. Given a LQ input, we select the optimal reference with the smallest differences of poses, which is subsequently utilized through an adaptive feature fusion module to alleviate the inconsistent distribution. (3) The above two works require one or more HQ references from the same identity, which limits their application scenarios. We note that different faces usually have similar structure and texture (i.e., nose, eyes and mouth), and suggest a general face restoration method by constructing general texture dictionaries for each facial component. The developed DFDNet can guide the restoration of arbitrary degraded images without requiring references of the same identity. (4) While being able to transfer the identity-aware texture, a specific face restoration method may be limited by the consistency of poses and expressions. On the contrary, the general texture prior can cover most of the poses, but is limited in the identity details. In this thesis, we propose a dual memory dictionary based method, called DMDNet, by storing identity related features and numerous general texture priors respectively. We also propose a dictionary transform module to adaptive handle the cases when the references are not available. (5) Compared with face images, real-world LQ natural images are more difficult to restore due to their complex textures and the unknown degradation types. Since the face region usually shares the similar degradation with other image regions, in this thesis we use the real-world LQ face images and their restored HQ counterparts to explore the real-world degradation process. The developed ReDegNet can transfer the degradation that is learned from real-world face images to HQ natural images to synthesize their real-world LQ images for blind natural image restoration. In summary, we present five methods for BIR. Among them, GFRNet and ASFFNet adopt the references from the same identity for specific face restoration, while DFDNet uses the face component priors for general face restoration. DMDNet learns dual memory dictionaries to combine the benefits of both general and specific restoration. Finally, ReDegNet learns the real-world degradation from face images and then synthesizes the practical training pairs for blind natural image restoration. Experiments show their effectiveness in BIR and show great values in practical applications.