COLORIZATION USING OPTIMIZATION

Colorizing  grayscale images by transferring color from a segmented example image. Rather than relying on a series of independent pixel-level decisions, we develop a new strategy that attempts to account for the higher-level context of each pixel. Given a grayscale image to colorize, first determine for each pixel which example segment it should learn its color from. This is done automatically using a robust supervised classification scheme that analyzes the low-level feature space defined by small neighborhoods of pixels in the example image. Next, each pixel is assigned a color from the appropriate region using a neighborhood matching metric, combined with spatial filtering for improved spatial coherence. Each color assignment is associated with a confidence value, and pixels with a sufficiently high confidence level are provided as “micro-scribbles” to the optimization-based colorization algorithm, which produces the final complete colorization of the image.

Colorization, the process of adding color to monochrome images and video, has long been recognized as highly laborious and tedious. Despite several recent important advances in the automation of the process, a considerable amount of manual effort is still required in many cases in order to achieve satisfactory results. For example, recently proposed a simple yet effective user-guided colorization method. In this method the user is required to scribble the desired colors in the interiors of the various regions. These constraints are formulated as a least-squares optimization problem that automatically propagates the scribbled colors to produce a completely colorized image. Other algorithms based on color scribbles have subsequently been proposed. While this approach has produced some impressive colorizations from a small amount of user input, sufficiently complex images may still require dozens, or more, carefully placed scribbles.

In addition to the manual effort involved in placing the scribbles, the pallet of colors must also be chosen carefully in order to achieve a convincing result, requiring both experience and a good sense of aesthetics. This difficulty may be alleviated by choosing the colors from a similar reference color image. In fact, proposed an automatic colorization technique that colorizes an image by matching small pixel neighborhoods in the image to those in the reference image, and transferring colors accordingly. This approach is a special case of the more general image analogies framework, where a general filter is learned from the relationship between two images A and A’ and then applied to an input image B to produce a filtered result B’. However, image analogies and its derivatives typically make local (pixel level) decisions and thus do not explicitly enforce a contiguous assignment of colors. The method promotes contiguity by formulating and solving a global optimization problem. Introduce a new color transfer method, which leverages the advantages of these two previous colorization approaches, while largely avoiding their shortcomings. Similarly to the method of colorizes one or more grayscale images, based on a user provided reference  a partially segmented example color image. This requires considerably less input from the user than scribbling-based interfaces, and the user is relieved from the task of selecting appropriate colors (beyond supplying the reference image). On the other hand, our method explicitly enforces spatial consistency, producing more robust colorizations than by using a spatial voting scheme followed by a final global optimization step.