Inverse rendering of a scene from a single image
Inverse rendering estimates physical scene attributes (e.g., reflectance, geometry, and lighting) from image(s) and is used for gaming, virtual reality, augmented reality, and robotics. An inverse rendering network (IRN) receives a single input image of a 3D scene and generates the physical scene attributes for the image. The IRN is trained by using the estimated physical scene attributes generated by the IRN to reproduce the input image and updating parameters of the IRN to reduce differences between the reproduced input image and the input image. A direct renderer and a residual appearance renderer (RAR) reproduce the input image. The RAR predicts a residual image representing complex appearance effects of the real (not synthetic) image based on features extracted from the image and the reflectance and geometry properties. The residual image represents near-field illumination, cast shadows, inter-reflections, and realistic shading that are not provided by the direct renderer.
1. A computer-implemented method, comprising:
processing an image of a three-dimensional (3D) scene by an inverse rendering neural network to estimate reflectance properties or geometry properties of the 3D scene;
computing, by a first encoder neural network, intrinsic features based on the reflectance properties or geometry properties of the 3D scene;
processing the image by a second encoder neural network to produce image features;
computing, by a decoder neural network, a residual image representing complex appearance effects of the image based on the image features and the intrinsic features; and
combining a rendered image corresponding to the image with the residual image to produce a reconstructed image corresponding to the image.
2. The computer-implemented method of claim 1 , wherein the image provided to the second encoder neural network and the inverse rendering neural network is associated with at least one of for gaming, virtual reality, augmented reality, navigation, or robotics.
3. The computer-implemented method of claim 1 , wherein the image corresponds to an indoor scene.
4. The computer-implemented method of claim 1 , further comprising:
processing an illumination map corresponding to the image, the reflectance properties, and the geometry properties by a closed-form direct renderer to generate the rendered image corresponding to the image;
and
reducing differences between the reconstructed image and the image.
5. The computer-implemented method of claim 4 , wherein parameters of the inverse rendering neural network are adjusted to reduce the differences.
6. The computer-implemented method of claim 4 , wherein the illumination map is generated by the inverse rendering neural network.
7. The computer-implemented method of claim 1 , wherein parameters of the first encoder neural network, the second encoder neural network, and the decoder neural network are maintained while the parameters of the inverse rendering neural network are adjusted.
8. The computer-implemented method of claim 1 , further comprising, before processing the image, training the inverse rendering neural network using a labeled synthetic training dataset.
9. The computer-implemented method of claim 1 , further comprising before processing the image, training the inverse rendering neural network using a labeled training dataset including sparse relative reflectance values determined by humans.
10. The computer-implemented method of claim 1 , wherein the geometry properties are defined by surface normal vectors for each pixel in the image estimated by the inverse rendering neural network.
11. The computer-implemented method of claim 1 , wherein the reflectance properties are defined by albedo data for each pixel in the image estimated by the inverse rendering neural network.
12. The computer-implemented method of claim 1 , wherein the complex appearance effects include one or more of near-field lighting effects, cast shadows, and inter-reflections.
13. The computer-implemented method of claim 1 , further comprising training the first encoder neural network, the second encoder neural network, and the decoder neural network using a labeled synthetic training dataset.
14. An inverse rendering training system, comprising:
an inverse rendering neural network configured to process an image of a three-dimensional (3D) scene to estimate reflectance properties or geometry properties of the 3D scene;
a first encoder neural network configured to compute intrinsic features based on the reflectance properties or geometry properties of the 3D scene;
a second encoder neural network configured to process the image to produce image features; and
a decoder neural network configured to compute a residual image representing complex appearance effects of the image based on the image features and the intrinsic features, wherein a rendered image is combined with the residual image to produce a reconstructed image corresponding to the image.
15. The system of claim 14 , wherein the image provided to the second encoder neural network and the inverse rendering neural network is associated with at least one of for gaming, virtual reality, augmented reality, navigation, or robotics.
16. The system of claim 14 , further comprising:
a closed-form direct renderer configured to process an illumination map corresponding to the image, the reflectance properties, and the geometry properties to generate the rendered image corresponding to the image; and
a loss function unit configured to reduce differences between the reconstructed image and the image.
17. The system of claim 14 , wherein the complex appearance effects include one or more of near-field lighting effects, cast shadows, and inter-reflections.
18. A non-transitory computer-readable media storing computer instructions for inverse rendering that, when executed by one or more processors, cause the one or more processors to perform the steps of:
processing an image of a three-dimensional (3D) scene by an inverse rendering neural network to estimate reflectance properties or geometry properties of the 3D scene;
computing, by a first encoder neural network, intrinsic features based on the reflectance properties or geometry properties of the 3D scene;
processing the image by a second encoder neural network to produce image features;
computing, by a decoder neural network, a residual image representing complex appearance effects of the image based on the image features and the intrinsic features; and
combining the rendered image with the residual image to produce a reconstructed image corresponding to the image.
19. The non-transitory computer-readable media of claim 18 , further comprising:
processing an illumination map corresponding to the image, the reflectance properties, and the geometry properties by a closed-form direct renderer to generate a rendered image corresponding to the image; and
reducing differences between the reconstructed image and the image.
20. The non-transitory computer-readable media of claim 18 , wherein the complex appearance effects include one or more of near-field lighting effects, cast shadows, and inter-reflections.