Generating complete three-dimensional scene geometries using machine learning

1 . A method, comprising:

determining an initial representation of a three-dimensional (3D) scene including a first set of cells;

computing a transition kernel for a second set of cells within a neighborhood of at least one cell included in the first set of cells;

generating, using a machine learning model, a first update to the initial representation at a previous time step and a second update to the initial representation at a current time step, the second update generated based on sampling the transition kernel and on at least on a set of predictions associated with the first set of cells; and

generating a 3D model of the 3D scene based at least on the second update to the initial representation.

2 . The method of claim 1 , wherein the machine learning model was trained using a loss that is computed based at least on a set of visible cells included in one or more training 3D scenes.

3 . The method of claim 1 , wherein the machine learning model was trained to generate a target geometry derived from a first set of scans of a training 3D scene based at least on an input geometry derived from a subset of the first set of scans of the training 3D scene.

4 . The method of claim 1 , wherein the machine learning model was trained to generate a series of intermediate states between a first representation of a training 3D scene and a second representation of the training 3D scene.

5 . The method of claim 1 , wherein generating the first update and the second update comprises:

applying a threshold to the set of predictions to determine the first set of cells corresponding to the first update; and

generating, via the machine learning model based at least on the first set of cells, the second update that includes a second set of predictions for the second set of cells located within a neighborhood of the first set of cells.

6 . The method of claim 5 , wherein generating the second set of predictions for the second set of cells comprises aggregating a plurality of probabilities associated with a cell included in the second set of cells into an overall probability associated with the cell.

7 . The method of claim 1 , wherein determining the initial representation comprises converting one or more scans of the 3D scene into the initial representation.

8 . The method of claim 1 , wherein the initial representation comprises a set of voxel occupancies.

9 . The method of claim 1 , wherein the machine learning model comprises a sparse convolutional neural network.

10 . A processor comprising:

one or more circuits to perform operations comprising:

determining an initial representation of a three-dimensional (3D) scene including a first set of cells;

computing a transition kernel for a second set of cells within a neighborhood of at least one cell included in the first set of cells;

generating, using a machine learning model, a first update to the initial representation at a previous time step and a second update to the initial representation at a current time step, the second update generated based on sampling the transition kernel and on at least on a set of predictions associated with the first set of cells; and

generating a 3D model of the 3D scene based at least on the second update to the initial representation.

11 . The processor of claim 10 , wherein the machine learning model was trained to generate a target geometry derived from a first set of scans of a second 3D shape based at least on an input geometry derived from a second set of scans of the second 3D shape, and wherein the second set of scans corresponds to a subset of the first set of scans.

12 . The processor of claim 10 , wherein the machine learning model was further trained using at least one of:

a binary cross-entropy loss associated with a set of visible cells included in one or more scans of a set of training 3D shapes; or

one or more sets of simulated scans of a set of synthetic 3D shapes.

13 . The processor of claim 10 , wherein the operations further comprise generating a simulation based at least on the model.

14 . The processor of claim 13 , wherein the simulation comprises at least one of a collision simulation, a driving simulation, or a structural load simulation.

15 . The processor of claim 10 , wherein generating the model comprises:

determining one or more 3D cells associated with the second update that are located between a sensor used to capture the initial representation and a set of 3D cells included in the initial representation; and

omitting the one or more 3D cells from the model.

16 . The processor of claim 10 , wherein the processor is comprised in at least one of:

a control system for an autonomous or semi-autonomous machine;

a perception system for an autonomous or semi-autonomous machine;

a system for performing simulation operations;

a system for performing digital twin operations;

a system for performing light transport simulation;

a system for performing collaborative content creation for 3D assets;

a system for performing deep learning operations;

a system implemented using an edge device;

a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content;

a system implemented using a robot;

a system for performing conversational Al operations;

a system for generating synthetic data;

a system incorporating one or more virtual machines (VMs);

a system implemented at least partially in a data center; or

a system implemented at least partially using cloud computing resources.

17 . A system comprising:

one or more processing units to execute operations comprising:

generating an input geometry based at least on a first set of scans of a shape;

generating a target geometry based at least on a second set of scans of the shape, wherein the first set of scans corresponds to a subset of the second set of scans;

executing a machine learning model to generate a first update to the input geometry at a previous time step and a second update to the input geometry at a current time step, including, for at least one cell included in the first set of cells, computing a transition kernel for a third set of cells within a neighborhood of the cell, wherein the second update is generated based at least on a set of predictions associated with a first set of cells and corresponding to the first update and a set of probabilities indicating occupancy within a 3D scene for a second set of cells located within a neighborhood of the first set of set of cells, and including sampling from the transition kernel to generate the second update; and

updating the machine learning model using one or more losses that are computed based at least on a set of visible cells associated with at least one of the first update or the second update.

18 . The system of claim 17 , wherein the system is comprised in at least one of:

a control system for an autonomous or semi-autonomous machine;

a perception system for an autonomous or semi-autonomous machine;

a system for performing simulation operations;

a system for performing digital twin operations;

a system for performing light transport simulation;

a system for performing collaborative content creation for 3D assets;

a system for performing deep learning operations;

a system implemented using an edge device;

a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content;

a system implemented using a robot;

a system for performing conversational Al operations;

a system for generating synthetic data;

a system incorporating one or more virtual machines (VMs);

a system implemented at least partially in a data center; or

a system implemented at least partially using cloud computing resources.