Characteristic-based acceleration for efficient scene rendering
In various examples, images (e.g., novel views) of an object may be rendered using an optimized number of samples of a 3D representation of the object. The optimized number of the samples may be determined based at least on casting rays into a scene that includes the 3D representation of the object and/or an acceleration data structure corresponding to the object. The acceleration data structure may include features corresponding to characteristics of the object, and the features may be indicative of the number of samples to be obtained from various portions of the 3D representation of the object to render the images. In some examples, the 3D representation may be a neural radiance field that includes, as a neural output, a spatially varying kernel size predicting the characteristics of the object, and the features of the acceleration data structure may be related to the spatially varying kernel size.
1 . A method comprising:
applying, to one or more machine learning models (MLMs), image data capturing one or more views of an object;
determining, based at least on the applying, one or more predictions indicating one or more characteristics of one or more portions of a three-dimensional (3D) representation of the object;
generating, using the one or more predictions and based at least on the one or more characteristics, a pair of surfaces including a first surface defining an inner boundary associated with the 3D representation of the object and a second surface defining an outer boundary associated with the 3D representation of the object;
casting one or more rays into a scene that includes the 3D representation of the object to determine one or more intersections between the one or more rays and the pair of surfaces;
generating one or more samples of the 3D representation of the object using one or more locations in the scene that correspond to the one or more intersections; and
rendering, using the one or more samples, one or more images of the object.
2 . The method of claim 1 , wherein the one or more locations include at least a first location where a ray of the one or more rays intersects the first surface and a second location where the ray intersects the second surface, the method further comprising:
determining a number of samples to generate along the ray based at least on a distance between the first location and the second location; and
generating the number of samples of the 3D representation along the ray between the first location and the second location.
3 . The method of claim 1 , wherein a first number of samples of the one or more samples is greater along a first ray intersecting a first location in the scene corresponding to a first portion of the 3D representation than a second number of samples of the one or more samples along a second ray intersecting a second location of the one or more locations in the scene corresponding to a second portion of the 3D representation.
4 . The method of claim 3 , wherein the first portion corresponds to a first characteristic of the one or more characteristics and the second portion corresponds to a second characteristic of the one or more characteristics that is different from the first characteristic.
5 . The method of claim 1 , wherein the pair of surfaces is a pair of meshes including at least a first mesh corresponding to the inner boundary associated with the 3D representation of the object and a second mesh corresponding to the outer boundary associated with the 3D representation of the object.
6 . The method of claim 1 , wherein:
the one or more portions includes at least a first portion and a second portion, and
a distance between the first surface and the second surface varies between the first portion and the second portion based at least on a variation in the one or more predictions between the first portion and the second portion.
7 . The method of claim 1 , wherein:
the one or more portions includes at least a first portion and a second portion,
a first distance between the first surface and the second surface at the first portion is related to a first characteristic of the one or more characteristics, and
a second distance between the first surface and the second surface at the second portion is related to a second characteristic of the one or more characteristics.
8 . The method of claim 1 , wherein the one or more characteristics correspond to at least one of one or more surface compositions of the object or one or more volumetric features of the object.
9 . The method of claim 1 , wherein the one or more predictions are represented by one or more sizes of a kernel in the 3D representation, the one or more sizes of the kernel indicating the one or more characteristics.
10 . A system comprising:
one or more processors to:
determine, using one or more machine learning models (MLMs) and based at least on image data representing one or more views of an object, one or more predictions indicating one or more characteristics of one or more portions of a three-dimensional (3D) representation of the object;
generate, using the one or more predictions and based at least on the one or more characteristics, a pair of surfaces defining a boundary region associated with the 3D representation of the object;
obtain one or more samples of the 3D representation of the object based at least on one or more intersections between one or more rays and one or more locations disposed within the boundary region; and
render one or more images of the object using the one or more samples.
11 . The system of claim 10 , wherein the one or more processors are further to cast the one or more rays into a scene including the pair of surfaces, wherein the obtaining of the one or more samples is based at least on the casting of the one or more rays.
12 . The system of claim 10 , wherein:
the one or more locations includes at least a first location and a second location within the boundary region,
the one or more samples include one or more first samples corresponding to the first location and one or more second samples corresponding to the second location, and
a first number of the one or more first samples is greater than a second number of the one or more second samples based at least on a thickness of the boundary region being greater at the first location than at the second location.
13 . The system of claim 10 , wherein a distance between the pair of surfaces varies between the one or more portions of the 3D representation based at least on the one or more characteristics.
14 . The system of claim 10 , wherein the one or more characteristics correspond to at least one of one or more surface compositions of the object or one or more volumetric features of the object.
15 . The system of claim 10 , wherein:
the one or more MLMs are to generate, based at least on the image data, data indicating the 3D representation of the object, and
the one or more samples is of the data indicating the 3D representation of the object.
16 . The system of claim 10 , wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing one or more simulation operations;
a system for performing one or more digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing one or more deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing one or more generative AI operations;
a system for performing operations using a large language model;
a system for performing one or more conversational AI operations;
a system for generating synthetic data;
a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
17 . The system of claim 10 , the one or more processors further to:
apply a surface-based rendering technique at a first location within the boundary region based at least on a thickness of the boundary region being less than a threshold at the first location; and
apply a volumetric rendering technique at a second location within the boundary region based at least on the thickness of the boundary region being greater than the threshold at the second location.
18 . The system of claim 10 , wherein the boundary region defined by the pair of surfaces corresponds to an acceleration structure for ray traversal during the rendering of the one or more images.
19 . At least one processor comprising:
one or more circuits to render one or more images of an object, the one or more images rendered using one or more samples of a three-dimensional (3D) representation of the object, the 3D representation of the object including a pair of surfaces defining a boundary region of the object, the boundary region having a varying thickness that is indicative of a number of samples to be obtained from locations within the boundary region to render the one or more images.
20 . The processor of claim 19 , wherein the processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing one or more simulation operations;
a system for performing one or more digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing one or more deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing one or more generative AI operations;
a system for performing operations using a large language model;
a system for performing one or more conversational AI operations;
a system for generating synthetic data;
a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.