Method and apparatus with rendering
A method includes generating a first rendering based on input elements of an input scene, generating a second rendering by inputting a result of the first rendering to a generative machine learning model that is based on an artificial neural network (ANN), and generating a rendered output image based on the result of the first rendering and a result of the second rendering.
1 . A processor-implemented method, comprising:
generating a first rendered complete image of a scene from a viewpoint and view direction by applying input elements representing the scene to a computation-based rendering algorithm that generates the first rendered image of the scene from the viewpoint and view direction to include direct illumination of the scene according to the input elements;
generating a second rendered image of the scene from the viewpoint and the view direction by inputting the first rendered image to a generative machine learning model that is based on an artificial neural network (ANN), the generative machine learning model generating the second rendered image to include estimated indirect illumination of the first rendered image by performing inference on the first rendered complete image of the scene; and
generating a rendered output image of the scene based on combining the first rendered image that includes the direct lighting and the second rendered image that includes the estimated indirect illumination.
2 . The method of claim 1 , wherein the input elements comprise any one or any combination of any two or more of lighting information, geometric information, and texture information of the scene.
3 . The method of claim 1 , wherein the input elements comprise at least texture information, and
wherein the generating of the second rendered image comprises:
generating feature embedding on the texture information for each segment;
adding the feature embedding as a condition for the generative machine learning model; and
generating the second rendered image by inputting the first rendered image to the generative machine learning model including the condition.
4 . The method of claim 1 , wherein the generating of the first rendered image comprises generating the first rendered image based on a predetermined rendering equation.
5 . The method of claim 1 , wherein the generating of the first rendered image comprises generating the direct illumination and generating indirect illumination based on the input elements.
6 . The method of claim 1 , wherein the combining the first rendered image and the second rendered image comprises adding the first rendered image to the second rendered image.
7 . The method of claim 1 , wherein the generative machine learning model includes plural layers and an attention mechanism.
8 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .
9 . The method of claim 1 , wherein the combining comprises adding the second rendered image, as a residual, to the first rendered image.
10 . The method of claim 1 , wherein the generative machine learning model is a trained generative machine learning model.
11 . The method of claim 1 , wherein the first rendered image is generated by only one iteration of the computation-based rendering.
12 . A processor-implemented method, comprising:
generating a first rendered complete image of a scene from a viewpoint and view direction by applying input elements representing the scene to a computation-based rendering algorithm that generates the first rendered image of the scene to include direct illumination of the scene according to the input elements, the generated first rendered image not including indirect illumination of the scene;
generating a second rendered image of the scene from the viewpoint and view direction by inputting the first rendered image to a generative machine learning model that generates the second rendered image to include estimated indirect illumination of the first rendered image by performing inference on the first rendered complete image of the scene;
generating a rendered output image of the scene based on a combination of the first rendered image and the second rendered image;
determining a difference, between the rendered output image and a ground truth image corresponding to the first rendered image, by inputting the rendered output image and the ground truth image to a discriminator model; and
training the generative machine learning model to minimize the difference.
13 . The method of claim 12 , further comprising: training the discriminator model to discriminate between the rendered output image and the ground truth image.
14 . The method of claim 12 , wherein the ground truth image comprises either one or both of a full rendering image corresponding to the first rendered image and a natural image.
15 . An electronic device, comprising:
one or more processors configured to:
generate a first rendered complete image of a scene from a viewpoint and view direction by applying input elements representing the scene to a computation-based rendering algorithm, the first rendered image from the viewpoint and view direction including direct illumination of the scene;
generate a second rendered image of the scene from the viewpoint and the view direction by a generative machine learning model, the second rendered image including indirect illumination of the scene inferred by the generative machine learning model based on the first rendered image by performing inference on the first rendered complete image of the scene; and
generate a rendered output image of the first scene based on the first rendered image and the second rendered image.
16 . The electronic device of claim 15 , further comprising a memory configured to store the generative machine learning model and instructions that when executed by the processor configure the one or more processors to perform:
the generation of the first rendered image;
the generation of the second rendered image; and
the generation of the rendered output image.
17 . The electronic device of claim 15 ,
wherein the input elements comprise texture information of the input scene, or the texture information and any one or both of lighting information and geometric information of the input scene, and
wherein the one or more processors are further configured to:
generate feature embedding on the texture information for each segment;
include the feature embedding as a condition for the generative machine learning model; and
generate the second rendered image by inputting the first rendered image to the generative machine learning model including the condition.
18 . The electronic device of claim 15 , wherein the one or more processors are further configured to:
generate the first rendered image to include indirect illumination rendering based on the input elements.
19 . An apparatus, comprising:
one or more processors configured to:
generate a second rendered complete image of a scene from a viewpoint and view direction using a generative neural network model that infers the second rendered image from a first rendered image of the scene that has been generated by computation-based rendering based on input elements representing the scene and based on the viewpoint and view direction, wherein the first image includes direct illumination of the scene according to the input elements representing the scene, and wherein the second rendered image is generated to include estimated indirect illumination of the first rendered image of the scene;
determine a rendered output image of the scene from the viewpoint and the view direction by combining the first rendered image that includes the direct illumination and the second rendered image that includes the indirect illumination;
discriminate a difference between the rendered output image and a ground truth image corresponding to the first rendered image; and
train the generative neural network model toward reducing the difference.
20 . The apparatus of claim 19 ,
wherein the training of the generative neural network model, to minimize the difference, is based on an output of a discriminator model that performs the difference, and
wherein the one or more processors are further configured to train the discriminator model to discriminate between the rendered output image and the ground truth image.