Neural network-based image lighting
Apparatuses, systems, and techniques are presented to generate image data. In at least one embodiment, one or more neural networks are used to cause a lighting effect to be applied to one or more objects within one or more images based, at least in part, on synthetically generated images of the one or more objects.
1 . A processor, comprising:
one or more circuits to use one or more neural networks to cause a lighting effect to be applied to one or more objects within one or more images, wherein a first portion of the one or more neural networks is updated during a first stage based, at least in part, on synthetically generated images of the one or more objects, and a second portion of the one more neural networks that corresponds to predicting surface reflectivity is updated during a second stage based, at least in part, on captured images of one or more physical objects while keeping the first portion of the one or more neural networks fixed.
2 . The processor of claim 1 , wherein the one or more circuits are to train the one or more neural networks using the synthetically generated images, and further train only the second portion of the one or more neural networks using the captured images of one or more physical objects.
3 . The processor of claim 2 , wherein the one or more circuits are further to cause the one or more neural networks to be trained to predict a geometry and an initial reflectivity component of the one or more objects using the synthetically generated images.
4 . The processor of claim 3 , wherein the one or more circuits are further to cause the second portion of the one or more neural networks to be further trained to predict an additional reflectivity component of the lighting effect to be applied to the one or more objects using the captured images of the one or more physical objects.
5 . The processor of claim 1 , wherein the one or more circuits are further to train the one or more neural networks using a loss function including at least a one light at a time (OLAT) consistency loss term and a relative consistency loss term.
6 . The processor of claim 1 , wherein the one or more images are frames of a video sequence, and wherein the one or more neural networks include one or more residual neural networks to provide temporal stability of the lighting effect between the frames.
7 . A system, comprising:
one or more processors to use one or more neural networks to cause a lighting effect to be applied to one or more objects within one or more images, wherein a first portion of the one or more neural networks is updated during a first stage based, at least in part, on synthetically generated images of the one or more objects, and a second portion of the one more neural networks that corresponds to predicting surface reflectivity is updated during a second stage based, at least in part, on captured images of one or more physical objects while keeping the first portion of the one or more neural networks fixed.
8 . The system of claim 7 , wherein the one or more processors are to train the one or more neural networks using the synthetically generated images, and further train only the second portion of the one or more neural networks using the captured images of one or more physical objects.
9 . The system of claim 8 , wherein the one or more processors are further to cause the one or more neural networks to be trained to predict a geometry and an initial reflectivity component of the one or more objects using the synthetically generated images.
10 . The system of claim 9 , wherein the one or more processors are further to cause the second portion of the one or more neural networks to be further trained to predict an additional reflectivity component of the lighting effect to be applied to the one or more objects using the captured images of the one or more physical objects.
11 . The system of claim 7 , wherein the one or more processors are further to train the one or more neural networks using a loss function including at least a one light at a time (OLAT) consistency loss term and a relative consistency loss term.
12 . The system of claim 7 , wherein the one or more images are frames of a video sequence, and wherein the one or more neural networks include one or more residual neural networks to provide temporal stability of the lighting effect between the frames.
13 . A method, comprising:
using one or more neural networks to cause a lighting effect to be applied to one or more objects within one or more images, wherein a first portion of the one or more neural networks is updated during a first stage based, at least in part, on synthetically generated images of the one or more objects, and a second portion of the one more neural networks that corresponds to predicting surface reflectivity is updated during a second stage based, at least in part, on captured images of one or more physical objects while keeping the first portion of the one or more neural networks fixed.
14 . The method of claim 13 , further comprising:
training the one or more neural networks using the synthetically generated images, and further train only the second portion of the one or more neural networks using the captured images of one or more physical objects.
15 . The method of claim 14 , further comprising:
causing the one or more neural networks to be trained to predict a geometry and an initial reflectivity component of the one or more objects using the synthetically generated images.
16 . The method of claim 15 , further comprising:
causing the second portion of the one or more neural networks to be further trained to predict an additional reflectivity component of the lighting effect to be applied to the one or more objects using the captured images of the one or more physical objects.
17 . The method of claim 13 , further comprising:
training the one or more neural networks using a loss function including at least a one light at a time (OLAT) consistency loss term and a relative consistency loss term.
18 . The method of claim 13 , wherein the one or more images are frames of a video sequence, and wherein the one or more neural networks include one or more residual neural networks to provide temporal stability of the lighting effect between the frames.
19 . A non-transitory machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:
use one or more neural networks to cause a lighting effect to be applied to one or more objects within one or more images, wherein a first portion of the one or more neural networks is updated during a first stage based, at least in part, on synthetically generated images of the one or more objects, and a second portion of the one more neural networks that corresponds to predicting surface reflectivity is updated during a second stage based, at least in part, on captured images of one or more physical objects while keeping the first portion of the one or more neural networks fixed.
20 . The non-transitory machine-readable medium of claim 19 , wherein the instructions if performed further cause the one or more processors to:
train the one or more neural networks using the synthetically generated images, and further train only the second portion of the one or more neural networks using the captured images of one or more physical objects.
21 . The non-transitory machine-readable medium of claim 20 , wherein the instructions if performed further cause the one or more processors to:
cause the one or more neural networks to be trained to predict a geometry and an initial reflectivity component of the one or more objects using the synthetically generated images.
22 . The non-transitory machine-readable medium of claim 21 , wherein the instructions if performed further cause the one or more processors to:
cause the second portion of the one or more neural networks to be further trained to predict an additional reflectivity component of the lighting effect to be applied to the one or more objects using the captured images of the one or more physical objects.
23 . The non-transitory machine-readable medium of claim 19 , wherein the one or more processors are further to:
train the one or more neural networks using a loss function including at least a one light at a time (OLAT) consistency loss term and a relative consistency loss term.
24 . The non-transitory machine-readable medium of claim 19 , wherein the one or more images are frames of a video sequence, and wherein the one or more neural networks include one or more residual neural networks to provide temporal stability of the lighting effect between the frames.
25 . An image generation system, comprising:
one or more processors to use one or more neural networks to cause a lighting effect to be applied to one or more objects within one or more images, wherein a first portion of the one or more neural networks is updated during a first stage based, at least in part, on synthetically generated images of the one or more objects, and a second portion of the one more neural networks that corresponds to predicting surface reflectivity is updated during a second stage based, at least in part, on captured images of one or more physical objects while keeping the first portion of the one or more neural networks fixed; and
memory for storing network parameters for the one or more neural networks.
26 . The image generation system of claim 25 , wherein the one or more processors are to train the one or more neural networks using the synthetically generated images, and further train only the second portion of the one or more neural networks using the captured images of one or more physical objects.
27 . The image generation system of claim 26 , wherein the one or more processors are further to cause the one or more neural networks to be trained to predict a geometry and an initial reflectivity component of the one or more objects using the synthetically generated images.
28 . The image generation system of claim 27 , wherein the one or more processors are further to cause the second portion of the one or more neural networks to be further trained to predict an additional reflectivity component of the lighting effect to be applied to the one or more objects using the captured images of the one or more physical objects.
29 . The image generation system of claim 25 , wherein the one or more processors are further to train the one or more neural networks using a loss function including at least a one light at a time (OLAT) consistency loss term and a relative consistency loss term.
30 . The image generation system of claim 25 , wherein the one or more processors are further to wherein the one or more images are frames of a video sequence, and wherein the one or more neural networks include one or more residual neural networks to provide temporal stability of the lighting effect between the frames.