Systems and methods for simulating dynamic objects based on real world data

1. A computing system comprising:

one or more processors; and

one or more computer-readable medium storing instructions that when executed by the one or more processors cause the computing system to perform operations, the operations comprising:

obtaining sensor data indicative of a dynamic object, wherein;

the sensor data comprises three-dimensional data corresponding to a first time and two-dimensional data corresponding to the first time, and

the three-dimensional data and two-dimensional data are associated with the dynamic object;

providing input data descriptive of the sensor data indicative of the dynamic object as an input to a machine-learned object parameter estimation model;

receiving as an output of the machine-learned object parameter estimation model, in response to receipt of the input data, a plurality of object model parameters indicative of at least one of a pose or a shape modification with respect to a template object model, wherein:

the plurality of object model parameters correspond to the first time, and

the plurality of object model parameters represent a three-dimensional dynamic object model indicative of an object pose and an object shape for the dynamic object;

obtaining sequential sensor data indicative of the dynamic object at one or more second times;

determining a sequence of object model parameters descriptive of a respective object pose for the dynamic object at the one or more second times, wherein the sequence of object model parameters is indicative of a trajectory of the dynamic object, wherein the trajectory is indicative of at least one or more velocities of the dynamic object; and

generating at least a portion of simulation data based, at least in part, on the sequence of object model parameters.

2. The computing system of claim 1 , wherein the operations further comprise:

generating a three-dimensional mesh representation of the dynamic object based, at least in part, on the plurality of object model parameters;

determining an initial simulated three-dimensional point cloud for the dynamic object based, at least in part, on the three-dimensional mesh representation; and

generating a plurality of enhanced object model parameters based, at least in part, on the initial simulated three-dimensional point cloud, the three-dimensional data, and an objective function.

3. The computing system of claim 2 , wherein the operations further comprise:

generating a three-dimensional set of interconnected joints representative of the dynamic object based, at least in part, on the plurality of object model parameters;

determining a two-dimensional set of interconnected joints for the dynamic object based, at least in part, on the three-dimensional set of interconnected joints; and

generating the plurality of enhanced object model parameters based, at least in part, on the two-dimensional set of interconnected joints, the two-dimensional data, and the objective function.

4. The computing system of claim 3 , wherein generating the plurality of enhanced object model parameters based, at least in part, on the objective function comprises:

applying the objective function to the initial simulated three-dimensional point cloud and the three-dimensional data to determine a three-dimensional consistency measure for the plurality of object model parameters, wherein the three-dimensional consistency measure is indicative of a consistency between the three-dimensional mesh representation of the dynamic object and the three-dimensional data;

applying the objective function to the two-dimensional set of interconnected joints and the two-dimensional data to determine a joint consistency measure for the plurality of object model parameters, wherein the joint consistency measure is indicative of a consistency between the three-dimensional set of interconnected joints representative of the dynamic object and the two-dimensional data; and

modifying the plurality of object model parameters based, at least in part, on the three-dimensional consistency measure and the joint consistency measure to generate the plurality of enhanced object model parameters, wherein the plurality of object model parameters are modified to increase (i) the consistency between the three-dimensional mesh representation of the dynamic object and the three-dimensional data and (ii) the consistency between the three-dimensional set of interconnected joints representative of the dynamic object and the two-dimensional data.

5. The computing system of claim 4 , wherein the machine-learned object parameter estimation model is trained based, at least in part, on the three-dimensional consistency measure and the joint consistency measure.

6. The computing system of claim 1 , wherein the two dimensional data comprises an image frame corresponding to the first time, and wherein the three dimensional data comprises a three dimensional point cloud corresponding to the first time.

7. The computing system of claim 1 , wherein the simulation data comprises a plurality of object-specific single action sequences descriptive of a plurality of dynamic objects, and wherein generating at least the portion of the simulation data based, at least in part, on the plurality of object model parameters comprises:

determining one or more object actions corresponding to the sequence of object model parameters based, at least in part, on the one or more velocities of the dynamic object;

generating one or more single action sequences from the sequence of object model parameters based, at least in part, on the sequence of object model parameters and the one or more object actions corresponding to the sequence of object model parameters; and

storing the one or more single action sequences in a dynamic object database comprising the plurality of object-specific single action sequences descriptive of the plurality of dynamic objects.

8. The computing system of claim 7 , wherein the simulation data further comprises a simulated object scene indicative of one or more movements of a respective dynamic object within a simulated environment, and wherein generating at least the portion of the simulation data based, at least in part, on the plurality of object model parameters further comprises:

obtaining simulation scene data indicative of the simulated environment;

obtaining at least one of the plurality of object-specific single action sequences from the dynamic object database, wherein the at least one object-specific single action sequence is indicative of the one or more movements of the respective dynamic object; and

generating the simulated object scene based, at least in part, on the simulation scene data and the at least one object-specific single action sequence.

9. The computing system of claim 1 , wherein the template object model is a linear blend skinning model comprising a template mesh representation and a template hierarchical set of interconnected bones.

10. The computing system of claim 9 , wherein the plurality of model parameters comprise at least one of (i) a plurality of joint angles corresponding to the template hierarchical set of interconnected bones, (ii) a plurality of offsets corresponding to the template hierarchical set of interconnected bones, (iii) a plurality of scale factors corresponding to the template hierarchical set of interconnected bones, or (iv) a plurality of deformations corresponding to the template mesh representation.

11. The computing system of claim 1 , wherein the dynamic object is a pedestrian.

12. A computer-implemented method comprising:

obtaining sensor data indicative of a dynamic object, wherein:

the sensor data comprises three-dimensional data corresponding to a first time and two-dimensional data corresponding to the first time, and

the three-dimensional data and two-dimensional data are associated with the dynamic object;

providing input data descriptive of the sensor data indicative of the dynamic object as an input to a machine-learned object parameter estimation model;

receiving as an output of the machine-learned object parameter estimation model, in response to receipt of the input data, a plurality of object model parameters indicative of at least one of a pose or a shape modification with respect to a template object model, wherein:

the plurality of object model parameters correspond to the first time, and

the plurality of object model parameters represent a three-dimensional dynamic object model indicative of an object pose and an object shape for the dynamic object;

obtaining sequential sensor data indicative of the dynamic object at one or more second times;

determining a sequence of object model parameters descriptive of a respective object pose for the dynamic object at the one or more second times, wherein the sequence of object model parameters is indicative of a trajectory of the dynamic object, wherein the trajectory is indicative of at least one or more velocities of the dynamic object; and

generating at least a portion of simulation data based, at least in part, on the sequence of object model parameters.

13. The computer-implemented method of claim 12 , further comprising:

generating a three-dimensional mesh representation of the dynamic object based, at least in part, on the plurality of object model parameters;

determining an initial simulated three-dimensional point cloud for the dynamic object based, at least in part, on the three-dimensional mesh representation; and

generating a plurality of enhanced object model parameters based, at least in part, on the initial simulated three-dimensional point cloud, the three-dimensional data, and an objective function.

14. The computer-implemented method of claim 13 , further comprising:

generating a three-dimensional set of interconnected joints representative of the dynamic object based, at least in part, on the plurality of object model parameters;

determining a two-dimensional set of interconnected joints for the dynamic object based, at least in part, on the three-dimensional set of interconnected joints; and

generating the plurality of enhanced object model parameters based, at least in part, on the two-dimensional set of interconnected joints, the two-dimensional data, and the objective function.

15. The computer-implemented method of claim 14 , wherein generating the plurality of enhanced object model parameters based, at least in part, on the objective function comprises:

applying the objective function to the initial simulated three-dimensional point cloud and the three-dimensional data to determine a three-dimensional consistency measure for the plurality of object model parameters, wherein the three-dimensional consistency measure is indicative of a consistency between the three-dimensional mesh representation of the dynamic object and the three-dimensional data;

applying the objective function to the two-dimensional set of interconnected joints and the two-dimensional data to determine a joint consistency measure for the plurality of object model parameters, wherein the joint consistency measure is indicative of a consistency between the three-dimensional set of interconnected joints representative of the dynamic object and the two-dimensional data; and

modifying the plurality of object model parameters based, at least in part, on the three-dimensional consistency measure and the joint consistency measure to generate the plurality of enhanced object model parameters, wherein the plurality of object model parameters are modified to increase (i) the consistency between the three-dimensional mesh representation of the dynamic object and the three-dimensional data and (ii) the consistency between the three-dimensional set of interconnected joints representative of the dynamic object and the two-dimensional data.

16. The computer-implemented method of claim 15 , wherein modifying the plurality of object model parameters based, at least in part, on the three-dimensional consistency measure and the joint consistency measure to generate the plurality of enhanced object model parameters, comprises:

performing energy minimization over the plurality of object model parameters to generate the plurality of enhanced object model parameters.

17. One or more tangible, non-transitory, computer readable media that store instructions that when executed by one or more processors cause the one or more processors to perform operations comprising:

obtaining sensor data indicative of a dynamic object, wherein:

the sensor data comprises three-dimensional data corresponding to a first time and two-dimensional data corresponding to the first time, and

the three-dimensional data and two-dimensional data are associated with the dynamic object;

providing input data descriptive of the sensor data indicative of the dynamic object as an input to a machine-learned object parameter estimation model;

receiving as an output of the machine-learned object parameter estimation model, in response to receipt of the input data, a plurality of object model parameters indicative of at least one of a pose or a shape modification with respect to a template object model, wherein:

the plurality of object model parameters correspond to the first time, and

the plurality of object model parameters represent a three-dimensional dynamic object model indicative of an object pose and an object shape for the dynamic object;

obtaining sequential sensor data indicative of the dynamic object at one or more second times;

determining a sequence of object model parameters descriptive of a respective object pose for the dynamic object at the one or more second times, wherein the sequence of object model parameters is indicative of a trajectory of the dynamic object, wherein the trajectory is indicative of at least one or more velocities of the dynamic object; and

generating at least a portion of simulation data based, at least in part, on the sequence of object model parameters.

18. The one or more tangible, non-transitory, computer readable media of claim 17 , wherein the simulation data comprises a plurality of object-specific single action sequences descriptive of a plurality of dynamic objects, and wherein generating at least the portion of the simulation data based, at least in part, on the plurality of object model parameters comprises:

determining one or more object actions corresponding to the sequence of object model parameters based, at least in part, on the one or more velocities of the dynamic object;

generating one or more single action sequences from the sequence of object model parameters based, at least in part, on the sequence of object model parameters and the one or more object actions corresponding to the sequence of object model parameters; and

storing the one or more single action sequences in a dynamic object database comprising the plurality of object-specific single action sequences descriptive of the plurality of dynamic objects.