Extracting material properties from a single image
Systems, methods, and non-transitory computer-readable media are disclosed for extracting material properties from a single digital image portraying one or more materials by utilizing a neural network encoder, a neural network material classifier, and one or more neural network material property decoders. In particular, in one or more embodiments, the disclosed systems and methods train the neural network encoder, the neural network material classifier, and one or more neural network material property decoders to accurately extract material properties from a single digital image portraying one or more materials. Furthermore, in one or more embodiments, the disclosed systems and methods train and utilize a rendering layer to generate model images from the extracted material properties.
1. A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause a computer system to:
generate a feature map by processing a digital image using a neural network encoder;
generate a material classification vector by processing the feature map using a neural network material classifier;
generate a plurality of bidirectional reflectance distribution function parameter vectors by processing the feature map using each of a plurality of neural network material property decoders; and
generate a material property set for the digital image based on a combination of the material classification vector and the plurality of bidirectional reflectance distribution function parameter vectors.
2. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computer system to generate the material classification vector by generating a plurality of material classification probabilities for a plurality of material types using a soft max classifier.
3. The non-transitory computer readable medium of claim 2 , further comprising instructions that, when executed by the at least one processor, cause the computer system to:
determine material type weights based on the plurality of material classification probabilities for the plurality of material types; and
generate the material property set for the digital image by utilizing a combination of the material type weights and the plurality of bidirectional reflectance distribution function parameter vectors.
4. The non-transitory computer readable medium of claim 3 , wherein the instructions, when executed by the at least one processor, cause the computer system to generate the plurality of bidirectional reflectance distribution function parameter vectors by generating a bidirectional reflectance distribution function parameter vector for each material type of the plurality of material types.
5. The non-transitory computer readable medium of claim 3 , wherein the instructions, when executed by the at least one processor, cause the computer system to generate the material property set for the digital image by:
multiplying each bidirectional reflectance distribution function parameter vector for each material type by a corresponding material type weight to generated weighted bidirectional reflectance distribution function parameter vectors; and
summing the weighted bidirectional reflectance distribution function parameter vectors.
6. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computer system to:
generate an intensity distance map comprising light intensity values corresponding to pixel coordinates of the digital image; and
generate the feature map for the digital image using the neural network encoder and the intensity distance map, wherein the digital image is captured with flash illumination comprising a single point light source using a concentrated light intensity and environment illumination.
7. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by at least one processor, cause the computer system to:
generate a model image using a rendering layer, wherein the model image comprises the generated material property set and a lighting environment;
generate a ground truth model image, wherein the ground truth model image comprises the digital image rendered based on a ground truth material property set and the lighting environment;
compare the model image to the ground truth model image to generate a material classifier loss, a plurality of material property decoder losses, and a rendering layer loss; and
adjust neural network parameters of the neural network encoder, the neural network material classifier, the plurality of neural network material property decoders, and the rendering layer based on a combination of the material classifier loss, the plurality of material property decoder losses, and the rendering layer loss.
8. The non-transitory computer readable medium of claim 1 , wherein the plurality of bidirectional reflectance distribution function parameter vectors comprises a diffuse color vector, a normal vector, and a roughness vector.
9. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by at least one processor, cause the computer system to generate a modified visual object by applying the material property set to a visual object.
10. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by at least one processor, cause the computer system to utilize densely connected conditional random fields to refine one or more bidirectional reflectance distribution function parameters from the plurality of bidirectional reflectance distribution function parameter vectors.
11. A system for extracting material properties from input digital images comprising:
one or more memories, comprising:
a digital image;
a neural network encoder;
a neural network material classifier; and
a plurality of neural network material property decoders; and
at least one computing device storing instructions thereon, that, when executed by the at least one computing device, cause the system to:
generate a feature map for the digital image using the neural network encoder;
generate a material classification vector that indicates a probability the digital image comprises each of a plurality of materials by processing the feature map using the neural network material classifier;
generate a plurality of bidirectional reflectance distribution function parameter vectors by processing the feature map using each of the plurality of neural network material property decoders; and
generate a material property set for the digital image based on a combination of the material classification vector and the plurality of bidirectional reflectance distribution function parameter vectors.
12. The system of claim 11 , further comprising instructions that, when executed by the at least one computing device, cause the system to:
generate the material classification vector by generating a plurality of material classification probabilities for a plurality of material types;
determine material type weights based on the plurality of material classification probabilities for the plurality of material types; and
generate the material property set for the digital image by utilizing a combination of the material type weights and the plurality of bidirectional reflectance distribution function parameter vectors.
13. The system of claim 11 , further comprising instructions that, when executed by the at least one computing device, cause the system to:
generate an intensity distance map comprising light intensity values corresponding to pixel coordinates of the digital image; and
generate the feature map for the digital image using the neural network encoder and intensity distance map, wherein the digital image is captured with flash illumination comprising a single point light source using a concentrated light intensity.
14. The system of claim 11 , further comprising instructions that, when executed by the at least one computing device, cause the system to:
generate a model image using a rendering layer, wherein the model image comprises the generated material property set and a lighting environment;
generate a ground truth model image, wherein the ground truth model image comprises the digital image rendered based on a ground truth material property set and the lighting environment;
compare the model image to the ground truth model image to generate a material classifier loss, a plurality of material property decoder losses, and a rendering layer loss; and
adjust neural network parameters of the neural network encoder, the neural network material classifier, the plurality of neural network material property decoders, and the rendering layer based on a combination of the material classifier loss, the plurality of material property decoder losses, and the rendering layer loss.
15. The system of claim 11 , wherein the plurality of neural network material property decoders comprises a neural network diffuse color decoder, a neural network normal decoder, and a neural network roughness decoder; and
further comprising instructions that, when executed by the at least one computing device, cause the system to:
generate a diffuse color vector by processing the feature map using the neural network diffuse color decoder, wherein the diffuse color vector comprises a diffuse color probability for each of a plurality of material types;
generate a normal vector by processing the feature map using the neural network normal decoder, wherein the normal vector comprises a normal probability for each of the plurality of material types; and
generate a roughness vector by processing the feature map using the neural network roughness decoder, wherein the roughness vector comprises a roughness probability for each of the plurality of material types.
16. The system of claim 11 , further comprising instructions that, when executed by the at least one computing device, cause the system to generate a modified visual object by applying the material property set to a visual object.
17. The system of claim 11 , further comprising instructions that, when executed by the at least one computing device, cause the system to:
generate a plurality of groupings, wherein each of the plurality of groupings comprises an angle range;
assign a plurality of predicted normal angles from the normal vector to each of the plurality of groupings based on the angle range of each of the plurality of groupings;
generate a normal distribution percentage for each of the plurality of groupings based on the number of predicted normal angles from the normal vector associated with each of the plurality of groupings compared to the total number of predicted normal angles from the normal vector; and
assign a normal weight to each of the plurality of groupings based on the normal distribution percentage for each of the plurality of groupings, wherein the normal weight increases as the normal distribution percentage decreases.
18. The system of claim 11 , further comprising instructions that, when executed by the at least one computing device, cause the system to utilize densely connected conditional random fields to refine one or more bidirectional reflectance distribution function parameters from the material property set.
19. In a digital medium environment for extracting material properties from input digital images, a method comprising:
performing a step for training a neural network encoder, a neural network material classifier, and a plurality of neural network decoders to generate a material property set from a single digital image comprising flash illumination;
performing a step for utilizing the neural network encoder, the neural network material classifier, and the plurality of neural network decoders to generate the material property set from the single digital image; and
generating a modified visual object by applying the material property set to a visual object.
20. The method of claim 19 , wherein generating the modified visual object comprising the material property set from the digital image by applying the material property set to a visual object comprises utilizing densely connected conditional random fields to refine one or more bidirectional reflectance distribution function parameters from the material property set.