Utilizing a colorization neural network to generate colorized images based on interactive color edges
Methods, systems, and non-transitory computer readable storage media are disclosed for utilizing an edge prediction neural network and edge-guided colorization neural network to transform grayscale digital images into colorized digital images. In one or more embodiments, the disclosed systems apply a color edge prediction neural network to a grayscale image to generate a color edge map indicating predicted chrominance edges. The disclosed systems can present the color edge map to a user via a colorization graphical user interface and receive user color points and color edge modifications. The disclosed systems can apply a second neural network, an edge-guided colorization neural network, to the color edge map or a modified edge map, user color points, and the grayscale image to generate an edge-constrained colorized digital image.
1. A non-transitory computer readable medium for generating colorized images, the non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause a computing device to:
generate a color edge map for a grayscale digital image using a color edge prediction neural network, wherein the color edge map comprises color edges;
provide, for display at a client device, a colorization graphical user interface comprising the color edge map, a color point input element, and the grayscale digital image;
generate a colorized image using an edge-guided colorization neural network based on the grayscale digital image, the color edge map, and user interaction with the color point input element; and
provide the colorized image for display via the colorization graphical user interface.
2. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
provide, for display as part of the colorization graphical user interface, the color edge map together with color edge modification elements;
generate a modified edge map based on user interaction with the color edge modification elements; and
generate a modified colorized image using the edge-guided colorization neural network based on the grayscale digital image, the modified edge map, and the user interaction with the color point input element.
3. The non-transitory computer readable medium of claim 2 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the color edge map by:
generating an initial color edge map comprising initial edges utilizing the color edge prediction neural network;
utilizing the color edge prediction neural network to determine a confidence score for each of the initial edges; and
based on determining that a confidence score associated with an initial edge of the initial edges meets a color edge threshold, including the initial edge in the color edge map as a color edge of the color edges.
4. The non-transitory computer readable medium of claim 3 , wherein the color edge modification elements comprise a color edge threshold element, and further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the modified edge map by:
determining a modified edge threshold based on user interaction with the color edge threshold element; and
including, in the modified edge map, a modified set of color edges based on the modified edge threshold.
5. The non-transitory computer readable medium of claim 2 , wherein the color edge modification elements comprise an edge suggestion element, and further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the modified edge map by:
based on user interaction with the edge suggestion element, generating, by utilizing a canny-edge detector, luminance edges of the grayscale digital image;
presenting the luminance edges via the colorization graphical user interface; and
based on user selection of one or more edges of the luminance edges, including the one or more edges in the modified edge map.
6. The non-transitory computer readable medium of claim 4 , wherein the color edge modification elements comprise a user-drawn edge element, and further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the modified edge map by:
generating, based on user interaction with the user-drawn edge element, a user-drawn edge; and
including the user-drawn edge in the modified edge map.
7. The non-transitory computer readable medium of claim 3 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the color edge map by utilizing non-maximum suppression to generate the color edge map from the initial color edge map.
8. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to train the edge-guided colorization neural network by:
applying a canny-edge detector to ground truth color corresponding to a ground truth color image to generate canny edges;
applying the edge-guided colorization neural network to a training grayscale image corresponding to the ground truth color image, training color points, and the canny edges;
utilizing the edge-guided colorization neural network to generate predicted color; and
modifying parameters of the edge-guided colorization neural network by comparing the predicted color with the ground truth color.
9. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the colorized image by:
generating a grayscale channel reflecting the grayscale digital image, a color edge map channel reflecting the color edge map, and a color point channel reflecting the user input;
processing the grayscale channel and the color point channel utilizing the edge-guided colorization neural network to generate a first feature vector via a first layer of the edge-guided colorization neural network;
concatenating the color edge map channel to the first feature vector to generate an edge-map modified feature vector; and
processing the edge-map modified feature vector utilizing a second layer of the edge-guided colorization neural network.
10. A system comprising:
one or more memory devices storing a grayscale digital image, a color edge prediction neural network, and an edge-guided colorization neural network;
at least one server configured to cause the system to:
use the color edge prediction neural network to generate a color edge map comprising color edges for the grayscale digital image;
provide, for display at a client device, a colorization graphical user interface comprising the color edge map, a color point input element, and a color edge modification element for modifying the color edges;
identifying user interaction with the color edge modification element via the colorization graphical user interface;
generate a modified edge map by modifying color edges based on the user interaction with the color edge modification element; and
generate, using the edge-guided colorization neural network, a colorized image based on the modified edge map and user interaction with the color point input element.
11. The system of claim 10 , further comprising instructions that, when executed by the at least one server, cause the system to generate the color edge map by:
generating an initial color edge map comprising initial edges utilizing the color edge prediction neural network;
utilizing the color edge prediction neural network to determine a confidence score for each of the initial edges; and
based on determining that a confidence score associated with an initial edge of the initial edges meets a color edge threshold, including the initial edge in the color edge map as a color edge of the color edges.
12. The system of claim 11 , wherein the color edge modification element comprises a color edge threshold element, and further comprising instructions that, when executed by the at least one server, cause the system to generate the modified edge map by:
determining a modified edge threshold based on user interaction with the color edge threshold element; and
including, in the modified edge map, a modified set of color edges based on the modified edge threshold.
13. The system of claim 10 , wherein the color edge modification element comprises an edge suggestion element, and further comprising instructions that, when executed by the at least one server, cause the system to generate the modified edge map by:
based on user interaction with the edge suggestion element, generating, by utilizing a canny-edge detector, luminance edges of the grayscale digital image;
presenting the luminance edges via the colorization graphical user interface; and
based on user selection of one or more edges of the luminance edges, including the one or more edges in the modified edge map.
14. The system of claim 12 , wherein the color edge modification element comprises a user-drawn edge element, and further comprising instructions that, when executed by the at least one server, cause the system to generate the modified edge map by:
generating, based on user interaction with the user-drawn edge element, a user-drawn edge; and
including the user-drawn edge in the modified edge map.
15. The system of claim 11 , further comprising instructions that, when executed by the at least one server, cause the system to generate the color edge map by utilizing non-maximum suppression to generate the color edge map from the initial color edge map.
16. The system of claim 10 , further comprising instructions that, when executed by the at least one server, cause the system to train the edge-guided colorization neural network by:
applying a canny-edge detector to ground truth color corresponding to a ground truth color image to generate canny edges;
applying the edge-guided colorization neural network to a training grayscale image corresponding to the ground truth color image, training color points, and the canny edges;
utilizing the edge-guided colorization neural network to generate predicted color; and
modifying parameters of the edge-guided colorization neural network by comparing the predicted color with the ground truth color.
17. The system of claim 10 , further comprising instructions that, when executed by the at least one server, cause the system to generate the colorized image by concatenating the color edge map into a plurality of layers of the edge-guided colorization neural network.
18. In a digital medium environment for editing and generating digital images, a computer-implemented method for generating enhanced-color digital images comprising:
generating a color edge map for a grayscale digital image using a color edge prediction neural network, wherein the color edge map comprises color edges;
providing, for display at a client device, a colorization graphical user interface comprising the color edge map, a color point input element, and the grayscale digital image;
generating a colorized image using an edge-guided colorization neural network based on the grayscale digital image, the color edge map, and user interaction with the color point input element; and
providing the colorized image for display via the colorization graphical user interface.
19. The method of claim 18 , further comprising:
providing, for display as part of the colorization graphical user interface, the color edge map together with color edge modification elements;
generating a modified edge map based on user interaction with the color edge modification elements; and
generating a modified colorized image using the edge-guided colorization neural network based on the grayscale digital image, the modified edge map, and the user interaction with the color point input element.
20. The method of claim 19 , further comprising generating the color edge map by:
generating an initial color edge map comprising initial edges utilizing the color edge prediction neural network;
utilizing the color edge prediction neural network to determine a confidence score for each of the initial edges; and
based on determining that a confidence score associated with an initial edge of the initial edges meets a color edge threshold, including the initial edge in the color edge map as a color edge of the color edges.