Albedo reconstruction based on inverse shading prediction
Various disclosed embodiments are directed to deriving an albedo output image from an input image based on deriving an inverse shading map. For example, an input image can be a photograph of a human face (i.e., the geometric features) with RGB values representing the color values of the face as well as pixels representing shadows (i.e., the shadow features) underneath the chin of the human face. The inverse shading map may be a black and white pixel value image that contains pixels representing the same human face without the RGB values and the shadows underneath the chin. The inverse shading map thus relies on the geometric space, rather than RGB space. Geometric space, for example, allows embodiments to capture the geometric features of a face, as opposed to those geometric features' RGB or shadow details.
1 . A system comprising:
at least one computer processor; and
one or more computer storage media storing computer-useable instructions that, when used by the at least one computer processor, cause the at least one computer processor to perform operations comprising:
receiving an input image, the input image including: a set of geometric features that define one or more portions of one or more real-world objects, a set of shadow features associated with the one or more real-world objects, and a set of color features that indicate one or more colors of the one or more real-world objects;
generating a segmentation map and a normal map, the normal map indicating a texture of the one or more real-world objects, the segmentation map indicating, via a unique pixel mask, different features of the one or more real-world objects;
deriving an inverse shading map based on the segmentation map and the normal map, the inverse shading map indicating the set of geometric features without the set of shadow features; and
based on the inverse shading map, deriving an albedo output image, the albedo output image indicating the set of geometric features and the set of color features but not the set of shadow features.
2 . The system of claim 1 , wherein the deriving of the inverse shading map is further based on generating, via a second model, a normal map and generating, via a third model, a segmentation map, the normal map indicating a microstructure texture of the one or more real-world objects, the segmentation map indicating, via a unique pixel mask, different features of the one or more real-world objects, wherein the normal map and the segmentation map are provided as input to the first model to produce the inverse shading map.
3 . The system of claim 1 , wherein the deriving of the albedo output image is further based on generating the albedo output image via a second model, and wherein the second model is a Generative Adversarial Network (GAN).
4 . The system of claim 1 , wherein the deriving of the albedo output image is further based on multiplying the input image by the inverse shading map.
5 . The system of claim 1 , wherein the operations further comprising training the first model by learning an inverse shading prediction function based on minimizing a perceptual loss between a ground truth image and an albedo training image and minimizing a discrimination loss between the ground truth image and a normal training image or a segmentation training image.
6 . The system of claim 1 , wherein the set of geometric features are features of a human face and hair, and wherein the one or more real-world objects include the human face and hair, and wherein the set of shadow features include shadows on the human face, and wherein the set of color features include a skin color of the human face, or hair color of the hair.
7 . The system of claim 1 , wherein the input image further includes a set of lighting features that represent highlights or lighting on the one or more real-world objects, and wherein the inverse shading map is further without the lighting features, and wherein the albedo output image does not include the lighting features.
8 . The system of claim 7 , wherein the inverse shading map is an image that includes negative lighting and shading relative to the input image.
9 . The system of claim 1 , wherein the albedo output image includes a second set of color features in a same position as the set of shadow features, and wherein the second set of color features are indicative of shadow removal from the input image.
10 . A computer-implemented method comprising:
receiving an input image, the input image including: a set of geometric features that define one or more portions of one or more real-world objects, a set of shadow features associated with the one or more real-world objects, and a set of color features that indicate one or more colors of the one or more real-world objects;
receiving a segmentation map and a normal map, the normal map indicating a texture of the one or more real-world objects, the segmentation map indicating, via a unique pixel mask, different features of the one or more real-world objects;
deriving an inverse shading map based the segmentation map and the normal map, the inverse shading map being a visual representation of the set of geometric features of the input image without the set of shadow features; and
based on the inverse shading map and the input image, deriving an albedo output image, the albedo output image includes a second set of color features in a same position as the set of shadow features, and wherein second set of color features is indicative of shadow removal from the input image.
11 . The computer-implemented method of claim 10 , wherein the deriving of the inverse shading map is based on generating, via a first model, the inverse shading map, and wherein the deriving of the inverse shading map is further based on generating, via a second model, a normal map and generating, via a third model, a segmentation map, the normal map indicating a microstructure texture of the one or more real-world objects, the segmentation map indicating, via a unique pixel mask, different features of the one or more real-world objects, wherein the normal map and the segmentation map are provided as input to the first model to produce the inverse shading map.
12 . The computer-implemented method of claim 10 , wherein the deriving of the albedo output image is further based on generating or modifying the albedo output image via a Generative Adversarial Network (GAN).
13 . The computer-implemented method of claim 10 , wherein the deriving of the albedo output image is further based on multiplying the input image by the inverse shading map.
14 . The computer-implemented method of claim 10 , wherein the deriving of the inverse shading map is based on a first model generating the inverse shading map, and wherein the method further comprising training the first model by learning an inverse shading prediction function based on minimizing a perceptual loss between a ground truth image and an albedo training image and minimizing a discrimination loss between the ground truth image and a normal training image or a segmentation training image.
15 . The computer-implemented method of claim 10 , wherein the set of geometric features are features of a human face and hair, and wherein the one or more real-world objects include the human face and the hair, and wherein the set of shadow features include shadows on the human face, and wherein the set of color features include a skin color of the human face, or hair color of the hair.
16 . The computer-implemented method of claim 10 , wherein the input image further includes a set of lighting features that represent highlights or lighting on the one or more real-world objects, and wherein the inverse shading map is further without the lighting features, and wherein the albedo output image does not include the lighting features.
17 . The computer-implemented method of claim 10 , wherein the inverse shading map is an image that includes negative lighting and shading relative to the input image.
18 . The computer-implemented method of claim 10 , wherein the albedo output image indicates the set of geometric features and the set of color features but not the set of shadow features.
19 . A computerized system, the system comprising:
an inverse shading map means for receiving an input image, the input image including: a set of geometric features that define one or more portions of one or more real-world objects, a set of shadow features associated with the one or more real-world objects, and a set of color features that indicate one or more colors of the one or more real-world objects;
at least one of a segmentation map means and a normal map means for generating at least one of a segmentation map and a normal map respectively, the normal map indicating a texture of the one or more real-world objects, the segmentation map indicating, via a unique pixel mask, different features of the one or more real-world objects;
and wherein the inverse shading map means is further for deriving an inverse shading map based on at least one of the segmentation map and the normal map, the inverse shading map indicating the set of geometric features without the set of shadow features; and
an albedo means for deriving an albedo output image based on the inverse shading map, the albedo output image indicating the set of geometric features and the set of color features but not the set of shadow features.
20 . The system of claim 19 , wherein the inverse shading map is generated by a first model, the normal map is generated by a second model, and the segmentation map is generated by a third model, and wherein the normal map and the segmentation map are provided as input to the first model to produce the inverse shading map.