Granted Patent B2

US 12,505,583 · App. 16/890,941 · Granted Dec 23, 2025

Inventors: Carl Jacob Munkberg (Skåne, SE); Jon Niklas Theodor Hasselgren (Skåne, SE)

Assignee: NVIDIA Corporation

G06T11/00G06F18/2148G06N3/045G06N3/08G06V20/647

Abstract

Claims (141)

1 . A processor, comprising:

one or more circuits to use one or more neural networks to render one or more images of a scene using a plurality of path-traced samples to simulate a transport of lighting in the scene by at least:

partitioning the plurality of path-traced samples into a plurality of layers, each layer including information for two or more samples of the plurality of path-traced samples;

applying a different filter for each layer of the plurality layers to generate a plurality of filter outputs; and

compositing the plurality of filter outputs to determine one or more properties of the pixel.

2 . The processor of claim 1 , wherein the one or more circuits are to use the one or more neural networks to:

obtain the plurality of path-traced samples for a pixel of the one or more images;

wherein to apply the different filter to each layer and composite the plurality of filter outputs is to denoise the one or more images.

3 . The processor of claim 2 , wherein the one or more neural networks to partition the plurality of path-traced samples are to use a convolutional neural network to determine respective weights of each of the plurality of layers.

4 . The processor of claim 2 , wherein the one or more neural networks are to further determine a plurality of sample embeddings from the plurality of path-traced samples using a fully connected network, wherein the plurality of sample embeddings are to be used at least to partition the plurality of path-traced samples into the plurality of layers.

5 . The processor of claim 4 , wherein the one or more neural networks are to further:

determine one or more context features from the plurality of sample embeddings using a U-net;

generate a first weight for a first layer of the plurality of layers as a result of the plurality of path-traced samples being partitioned into the plurality of layers; and

generate a first filter for the first layer of the plurality of layers based at least in part on the plurality of sample embeddings and the first weight.

6 . The processor of claim 2 , wherein:

the plurality of filter outputs comprise, for each layer of the plurality of layers: a radiance, a weight, and an occupancy; and

the one or more neural networks are to use alpha compositing on respective layers, respective radiances, and respective occupancies of the plurality of layers to determine the one or more properties of the pixel.

7 . The processor of claim 2 , wherein the one or more properties includes a color of the pixel.

8 . A non-transitory machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to use one or more neural networks to at least;

partition a set of samples into a plurality of layers, each layer including information for two or more samples of the set of samples, wherein the samples are path-traced samples to simulate lighting according to a three-dimensional model to render a pixel for a two-dimensional representation;

apply a different filter to each layer of the plurality layers to generate a plurality of filter outputs; and

composite the plurality of filter outputs to determine one or more properties of the pixel.

9 . The non-transitory machine-readable medium of claim 8 , wherein the one or more processors are to use the one or more neural networks to:

obtain the set of samples for the pixel of the two-dimensional representation;

generate a set of embeddings from the set of samples;

process the set of embeddings to generate a set of context data;

wherein to partition the set of samples into the plurality of layers is based at least in part on the set of embeddings and the set of context data; and

generate, for the plurality of layers, the different filters based at least in part on a result of partitioning the set of samples;

wherein to apply the different filter to each layer and composite the plurality of filter outputs is to denoise the two-dimensional representation.

10 . The non-transitory machine-readable medium of claim 9 , wherein:

the one or more neural networks to partition the set of samples into the plurality of layers are to determine a set of weights for the plurality of layers; and

the one or more neural networks to generate the plurality of filter outputs based at least in part on the set of weights.

11 . The non-transitory machine-readable medium of claim 9 , wherein:

a sample of the set of samples comprises radiance data and a set of feature guides; and

the set of embeddings are to be generated by applying a fully connected network on the set of samples.

12 . The non-transitory machine-readable medium of claim 11 , wherein the set of feature guides includes at least one of:

normal;

depth;

albedo;

specular color; or

specular roughness.

13 . The non-transitory machine-readable medium of claim 9 , wherein the plurality of filter outputs comprise:

a layer radiance;

a layer weight; and

a layer occupancy.

14 . A system, comprising:

one or more processors to use one or more neural networks to;

partition a set of samples into a plurality of layers, each layer including information for two or more samples of the set of samples, wherein the samples are path-traced samples to simulate lighting according to a three-dimensional model to render a pixel for a two-dimensional representation;

apply a different filter to each layer of the plurality layers to generate a plurality of filter outputs; and

composite the plurality of filter outputs to determine one or more properties of the pixel; and

one or more memories to store the one or more neural networks.

15 . The system of claim 14 , wherein the one or more processors are to use the one or more neural networks to:

obtain the set of samples for the pixel of the two-dimensional representation;

transform the set of samples to generate a set of embeddings;

process the set of embeddings to generate a set of context data;

wherein to partition the set of samples into the plurality of layers is based at least in part on the set of embeddings and the set of context data; and

generate, for the plurality of layers, the different filters based at least in part on a result of partitioning the set of samples;

wherein to apply the different filter to each layer and composite the plurality of filter outputs is to denoise the two-dimensional representation.

16 . The system of claim 15 , wherein:

the set of samples are to be transformed through a first neural network of the one or more neural networks;

the set of embeddings is to be processed with a second neural network of the one or more neural networks; and

the set of samples is to be partitioned with a third neural network of the one or more neural networks.

17 . The system of claim 16 , wherein:

the first neural network is a first fully connected network;

the second neural network is a U-net; and

the third neural network is a second fully connected network.

18 . The system of claim 17 , wherein the U-net includes one or more skip connections.

19 . The system of claim 15 , wherein the one or more processors are to generate respective filters of the different filters using respective neural networks of the one or more neural networks.

20 . The system of claim 14 , wherein a hyperparameter defines how many layers are in the plurality of layers.

21 . A processor, comprising:

one or more circuits to train one or more neural networks to;

partition a plurality of samples into a plurality of layers, each layer including information for two or more samples of the plurality of samples, wherein the samples are path-traced samples to simulate lighting according to a three-dimensional model to render a pixel for a two-dimensional representation;

apply a different filter to each layer of the plurality layers to generate a plurality of filter outputs; and

composite the plurality of filter outputs to determine one or more properties of the pixel.

22 . The processor of claim 21 , wherein the one or more circuits are to use the one or more neural networks to:

obtain the plurality of samples for the pixel of the two-dimensional representation;

wherein to apply the different filter to each layer and composite the plurality of filter outputs is to denoise the two-dimensional representation.

23 . The processor of claim 22 , wherein the one or more neural networks to partition the plurality of samples are to use a convolutional neural network to determine respective weights of each of the plurality of layers.

24 . The processor of claim 22 , wherein the one or more neural networks are to further determine a plurality of sample embeddings from the plurality of samples using a fully connected network, wherein the plurality of sample embeddings are to be used at least to partition the plurality of samples into the plurality of layers.

25 . The processor of claim 24 , wherein the one or more neural networks are to further:

determine one or more context features from the plurality of sample embeddings using a U-net;

generate a first weight for a first layer of the plurality of layers as a result of the plurality of samples being partitioned into the plurality of layers; and

generate a first filter for the first layer of the plurality of layers based at least in part on the plurality of sample embeddings and the first weight.

26 . The processor of claim 22 , wherein:

the plurality of filter outputs comprise, for each layer of the plurality of layers: a radiance, a weight, and an occupancy; and

the one or more neural networks are to use alpha compositing on respective layers, respective radiances, and respective occupancies of the plurality of layers to determine the one or more properties of the pixel.

27 . The processor of claim 22 , wherein the one or more properties includes an average radiance of the pixel.

28 . A non-transitory machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:

cause one or more neural networks to be trained to:

partition a set of samples into a plurality of layers, each layer including information for two or more samples of the set of samples, wherein the samples are path-traced samples to simulate lighting according to a three-dimensional model to render a pixel for a two-dimensional representation;

apply a different filter to each layer of the plurality layers to generate a plurality of filter outputs; and

composite the plurality of filter outputs to determine one or more properties of the pixel.

29 . The non-transitory machine-readable medium of claim 28 , wherein the one or more processors are to use the one or more neural networks to:

obtain the set of samples for the pixel of the two-dimensional representation;

generate a set of embeddings from the set of samples;

process the set of embeddings to generate a set of context data;

wherein to partition the set of samples into the plurality of layers is based at least in part on the set of embeddings and the set of context data; and

generate, for the plurality of layers, the different filters based at least in part on a result of partitioning the set of samples;

wherein to apply the different filter to each layer and composite the plurality of filter outputs is to denoise the two-dimensional representation.

30 . The non-transitory machine-readable medium of claim 29 , wherein:

the one or more neural networks to partition the set of samples into the plurality of layers are to determine a set of weights for the plurality of layers; and

the one or more neural networks to generate the plurality of filter outputs based at least in part on the set of weights.

31 . The non-transitory machine-readable medium of claim 29 , wherein:

a sample of the set of samples comprises radiance data and a set of feature guides; and

the set of embeddings are to be generated by applying a fully connected network on the set of samples.

32 . The non-transitory machine-readable medium of claim 31 , wherein the set of feature guides includes at least one of:

motion vector;

circle of confusion;

lens position; or

time.

33 . The non-transitory machine-readable medium of claim 29 , wherein the plurality of filter outputs comprise:

a layer radiance;

a layer weight; and

a layer occupancy.

34 . A system, comprising:

one or more processors to calculate parameters corresponding to one or more neural networks to:

partition a set of samples into a plurality of layers, each layer including information for two or more samples of the set of samples, wherein the samples are path-traced samples to simulate lighting according to a three-dimensional model to render a pixel for a two-dimensional representation;

apply a different filter to each layer of the plurality layers to generate a plurality of filter outputs; and

composite the plurality of filter outputs to determine one or more properties of the pixel; and

one or more memories to store the parameters.

35 . The system of claim 34 , wherein the one or more processors are to use the one or more neural networks to:

obtain the set of samples for the pixel of the two-dimensional representation;

transform the set of samples to generate a set of embeddings;

process the set of embeddings to generate a set of context data;

wherein to partition the set of samples into the plurality of layers is based at least in part on the set of embeddings and the set of context data; and

generate, for the plurality of layers, the different filters based at least in part on a result of partitioning the set of samples;

wherein to apply the different filter to each layer and composite the plurality of filter outputs is to denoise the two-dimensional representation.

36 . The system of claim 35 , wherein:

the set of samples are to be transformed through a first neural network of the one or more neural networks;

the set of embeddings is to be processed with a second neural network of the one or more neural networks; and

the set of samples is to be partitioned with a third neural network of the one or more neural networks.

37 . The system of claim 36 , wherein:

the first neural network is a first fully connected network;

the second neural network is a U-net; and

the third neural network is a second fully connected network.

38 . The system of claim 37 , wherein the U-net includes one or more skip connections.

39 . The system of claim 35 , wherein the one or more processors are to generate respective filters of the different filters using respective neural networks of the one or more neural networks.

40 . A processor comprising one or more circuits to use one or more neural networks to denoise a two-dimensional representation of a three-dimensional model, wherein to denoise the two-dimensional representation the one or more circuits are to use the one or more neural networks to:

partition a plurality of samples, to render a pixel of the two-dimensional representation, into a plurality of layers, each layer including information for two or more of the plurality of samples;

apply a different filter to each layer of the plurality layers to generate a plurality of filter outputs; and

composite the plurality of filter outputs to determine one or more properties of the pixel.

Assignments (1)

Continuity (1)

References Cited (65)