Generative non-photorealistic line drawing colorization

Abstract

Image colorization is a classic task in computer graphics that involves adding color to black-and-white photos, enhancing their visual appeal. Over the past two decades, this field has evolved significantly, developing a rich array of techniques. It is widely applied in areas such as old photo restoration, movie remastering, and scientific visualization. The rise of Artificial Intelligence Generated Content (AIGC) technology has further influenced creative media and computational design, impacting fields like fashion and animation design. My first research project is a comprehensive survey of image colorization, proposing a new taxonomy that views colorization as an application of computer graphics. This study analyzes the chronological development of colorization technology, highlighting its origins in graphics, its growth in vision, and its current trend toward integrating both fields. A key issue identified is the lack of specific evaluation metrics for colorization, with existing methods relying on image restoration metrics. To address this, we propose a colorization aesthetic evaluation and assess seven automatic colorization methods, emphasizing the importance of aligning color rendering with human visual perception. This has significant implications for computational photography, such as improving image texture quality in smartphone cameras. Image colorization can be categorized into natural image colorization and line drawing colorization. Line drawings, unlike natural grayscale images, consist of sparse lines that convey only structural information, making them more challenging to colorize. Current methods include user-guided, reference-based, and text-based approaches. User-guided methods require high aesthetic skills, unsuitable for non-professionals, while text-based methods struggle with complex structures. Reference-based methods, however, are user-friendly, allowing automatic coloring with a reference image. Despite their convenience, these methods often suffer from issues with color consistency and semantic correspondence. My second research focuses on designing an attention-aware anime line drawing colorization method using Generative Adversarial Networks (GANs). By employing self-supervised data enhancement, the network is trained with limited data. The integration of convolutional and self-attention mechanisms enhances color consistency and semantic correspondence while maintaining clear line structures. Given the challenges of GANs, such as unstable training and mode collapse, diffusion models offer a promising alternative. Inspired by their success in other image generation tasks, my third research introduces AnimeDiffusion, the first reference-based anime colorization diffusion model. This study employs a two-stage training strategy involving denoising and color reconstruction to improve image quality and accelerate training. A dataset of anime faces, comprising 31,696 training images and 579 testing images, was created for model training. Using XDoG as an intermediate line drawing representation, the method demonstrates strong generalization across different line styles. AnimeDiffusion is applicable in anime character recolorization, original character colorization, and fashion sketch colorization. In summary, the integration of computer graphics and AI is crucial in today's AIGC technology. As a graphics researcher, I focus on valuable topics, exploring the development of colorization technology, proposing a novel aesthetic assessment method, and developing new models based on generative algorithms. My research enhances creative efficiency for artists and also extends to fashion illustration design.