Light estimation method for three-dimensional (3D) rendered objects
A method for applying lighting conditions to a virtual object in an augmented reality (AR) device is described. In one aspect, the method includes generating, using a camera of a mobile device, an image, accessing a virtual object corresponding to an object in the image, identifying lighting parameters of the virtual object based on a machine learning model that is pre-trained with a paired dataset, the paired dataset includes synthetic source data and synthetic target data, the synthetic source data includes environment maps and 3D scans of items depicted in the environment map, the synthetic target data includes a synthetic sphere image rendered in the same environment map, applying the lighting parameters to the virtual object, and displaying, in a display of the mobile device, the shaded virtual object as a layer to the image.
1. A method comprising:
generating, using a camera of a mobile device, an image;
accessing a virtual object corresponding to an object in the image;
identifying shading parameters of the virtual object based on the object captured in the image and a machine learning model that is pre-trained with a paired dataset, the paired dataset comprising synthetic source data and synthetic target data, the synthetic source data comprising environment maps and three-dimensional (3D) scans of objects depicted in the environment maps, the synthetic target data comprising a synthetic sphere image rendered in a same environment map, wherein the environment maps include a set of HDR (High Dynamic Range) environment maps, wherein the 3D scans of objects include a set of 3D facial scans of people depicted in a corresponding HDR environment map of the set of HDR environment maps;
training the machine learning model by:
generating, using a first renderer, a synthetic face image based on the set of HDR (High Dynamic Range) environment maps and the set of 3D facial scans of people;
generating, using a neural network, predicted lighting parameters based on the synthetic face image;
generating, using a differential renderer, a predicted sphere image based on the predicted lighting parameters and a sphere asset that comprises synthetic sphere 3D models;
generating, using a second renderer, the synthetic sphere image based on the set of HDR environment maps and the sphere asset;
comparing the predicted sphere image with the synthetic sphere image using a L2 loss function; and
training the neural network using a result of the L2 loss function via back-propagation;
applying the shading parameters to the virtual object to generate a shaded virtual object; and
displaying, in a display of the mobile device, the shaded virtual object as a layer to the image.
2. The method of claim 1 , further comprising:
predicting, using the neural network, spherical Gaussians and ambient light based on the synthetic face image; and
generating, using the differential renderer, the predicted sphere image based on the sphere asset, the spherical Gaussians, and the ambient light.
3. The method of claim 1 , wherein applying the shading parameters to the virtual object comprises:
providing the shading parameters to a physically based rendering (PBR) shader; and
applying, using the PBR shader, estimated lighting conditions to the virtual object.
4. The method of claim 1 , wherein the image includes a self-portrait image of a user of the mobile device.
5. A computing apparatus comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the computing apparatus to:
generate, using a camera of a mobile device, an image;
access a virtual object corresponding to an object in the image;
identify shading parameters of the virtual object based on the object captured in the image and a machine learning model that is pre-trained with a paired dataset, the paired dataset comprising synthetic source data and synthetic target data, the synthetic source data comprising environment maps and three-dimensional (3D) scans of objects depicted in the environment maps, the synthetic target data comprising a synthetic sphere image rendered in a same environment map, wherein the environment maps include a set of HDR (High Dynamic Range) environment maps, wherein the 3D scans of objects include a set of 3D facial scans of people depicted in a corresponding HDR environment map of the set of HDR environment maps;
training the machine learning model by:
generating, using a first renderer, a synthetic face image based on the set of HDR (High Dynamic Range) environment maps and the set of 3D facial scans of people;
generating, using a neural network, predicted lighting parameters based on the synthetic face image;
generating, using a differential renderer, a predicted sphere image based on the predicted lighting parameters and a sphere asset that comprises synthetic sphere 3D models;
generating, using a second renderer, the synthetic sphere image based on the set of HDR environment maps and the sphere asset;
comparing the predicted sphere image with the synthetic sphere image using a L2 loss function; and
training the neural network using a result of the L2 loss function via back-propagation;
apply the shading parameters to the virtual object; and
display, in a display of the mobile device, the shaded virtual object as a layer to the image.
6. The computing apparatus of claim 5 , wherein the instructions further configure the computing apparatus to:
predict, using the neural network, spherical Gaussians and ambient light based on the synthetic face image; and
generate, using the differential renderer, the predicted sphere image based on the sphere asset, the spherical Gaussians, and the ambient light.
7. The computing apparatus of claim 5 , wherein applying the shading parameters to the virtual object comprises:
provide the shading parameters to a physically based rendering (PBR) shader; and
apply, using the PBR shader, estimated lighting conditions to the virtual object.
8. The computing apparatus of claim 5 , wherein the image includes a self-portrait image of a user of the mobile device.
9. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to:
generate, using a camera of a mobile device, an image;
access a virtual object corresponding to an object in the image;
identify shading parameters of the virtual object based on the object captured in the image and a machine learning model that is pre-trained with a paired dataset, the paired dataset comprising synthetic source data and synthetic target data, the synthetic source data comprising environment maps and three-dimensional (3D) scans of objects depicted in the environment maps, the synthetic target data comprising a synthetic sphere image rendered in a same environment map, wherein the environment maps include a set of HDR (High Dynamic Range) environment maps, wherein the 3D scans of objects include a set of 3D facial scans of people depicted in a corresponding HDR environment map of the set of HDR environment maps;
training the machine learning model by:
generating, using a first renderer, a synthetic face image based on the set of HDR (High Dynamic Range) environment maps and the set of 3D facial scans of people;
generating, using a neural network, predicted lighting parameters based on the synthetic face image;
generating, using a differential renderer, a predicted sphere image based on the predicted lighting parameters and a sphere asset that comprises synthetic sphere 3D models;
generating, using a second renderer, the synthetic sphere image based on the set of HDR environment maps and the sphere asset;
comparing the predicted sphere image with the synthetic sphere image using a L2 loss function; and
training the neural network using a result of the L2 loss function via back-propagation;
apply the shading parameters to the virtual object; and
display, in a display of the mobile device, the shaded virtual object as a layer to the image.
10. The computer-readable storage medium of claim 9 , wherein the instructions further cause the computer to:
predict, using the neural network, spherical Gaussians and ambient light based on the synthetic face image; and
generate, using the differential renderer, the predicted sphere image based on the sphere asset, the spherical Gaussians, and the ambient light.
11. The computer-readable storage medium of claim 9 , wherein applying the shading parameters to the virtual object comprises:
provide the shading parameters to a physically based rendering (PBR) shader; and
apply, using the PBR shader, estimated lighting conditions to the virtual object,
wherein the image includes a self-portrait image of a user of the mobile device.