System and method of capturing three-dimensional human motion capture with LiDAR
Described herein are systems and methods for training machine learning models to generate three-dimensional (3D) motions based on light detection and ranging (LiDAR) point clouds. In various embodiments, a computing system can encode a machine learning model representing an object in a scene. The computing system can train the machine learning model using a dataset comprising synchronous LiDAR point clouds captured by monocular LiDAR sensors and ground-truth three-dimensional motions obtained from IMU devices. The machine learning model can be configured to generate a three-dimensional motion of the object based on an input of a plurality of point cloud frames captured by a monocular LiDAR sensor.
1. A computer-implemented method for training machine learning models to generate three-dimensional motions based on LiDAR point clouds, the method comprising:
encoding, by a computing system, a machine learning model representing an object in a scene; and
training, by the computing system, the machine learning model using a dataset comprising synchronous LiDAR point clouds captured by monocular LiDAR sensors and ground-truth three-dimensional motions obtained from Inertial Measurement Units (IMU) devices;
wherein the machine learning model is configured to generate a three-dimensional motion of the object based on an input of a plurality of point cloud frames captured by a monocular LiDAR sensor, and
the machine learning model comprises a feature learning network, a Gated Recurring Unit (GRU), a multiplayer perceptron (MLP) decoder, wherein the feature learning network is configured to extract a global descriptor from each point cloud frame, the GRU is configured to generate a plurality of hidden variables for the global descriptor, and the MLP decoder is configured to predict a plurality of joint locations based on the plurality of hidden variables.
2. The method of claim 1 , wherein the object is a human.
3. The method of claim 2 , wherein the synchronous LiDAR point clouds comprises a plurality of point cloud frames captured by the monocular LiDAR sensors situated at a particular distance away from the human, and each point of the plurality of point cloud frames comprises at least a timestamp, and wherein the distance ranges from at least 10 to 50 meters.
4. The method of claim 2 , wherein the ground-truth three-dimensional motions are associated with the human and each ground-truth three-dimensional motion of the human comprises a timestamp, spatial coordinates, rotations of a plurality of joints of the human, and three-dimensional poses of the human.
5. The method of claim 2 , wherein the dataset further comprises synchronous images of the human.
6. The method of claim 5 , wherein the dataset further comprises labels for the synchronous images of the human, wherein each label comprises a two-dimension enclosure enclosing the human depicted in the synchronous images.
7. The method of claim 6 , further comprising
training a second machine learning model using the synchronous images of the human in the dataset;
outputting, by the trained second machine learning model, a second three-dimensional motion of the human; and
evaluating the three-dimensional motion of the human based on the second three-dimensional motion of the human.
8. The method of claim 1 , wherein the feature learning network comprises a PointNet++ network.
9. The method of claim 8 , wherein the machine learning model further comprises a kinematics solver module for concatenating a frame-wise global feature with each joint to generate completed joint features and outputting the completed joint features to compute a plurality of joint rotations, wherein the kinematics solver comprises a ST-GCN configured to concatenate the frame-wise global features with each joint.
10. The method of claim 9 , wherein the machine learning model further comprises a joint optimizer module for optimizing rotations of the plurality of joint rotations, wherein the joint optimizer module comprises a SMPL model configured to optimize the rotations of the plurality of joint rotations.
11. A system for training machine learning models to generate three-dimensional motions based on LiDAR point clouds, the system comprises comprising:
a processor;
a storage device for storing a machine learning model representing an object in a scene, and a dataset comprising synchronous LiDAR point clouds captured by monocular LiDAR sensors and ground-truth three-dimensional motions obtained from IMU devices, wherein the machine learning model is configured to generate a three-dimensional motion of the object based on an input of a plurality of point cloud frames captured by a monocular LiDAR sensor, and the machine learning model comprises a feature learning network, a Gated Recurring Unit (GRU), a multiplayer perceptron (MLP) decoder, wherein the feature learning network is configured to extract a global descriptor from each point cloud frame, the GRU is configured to generate a plurality of hidden variables for the global descriptor, and the MLP decoder is configured to predict a plurality of joint locations based on the plurality of hidden variables; and
a memory for storing instructions that, when executed by the processor, cause the system to perform training of the machine learning model using the dataset.
12. The system of claim 11 , wherein the object is a human.
13. The system of claim 12 , wherein the synchronous LiDAR point clouds comprises a plurality of point cloud frames captured by the monocular LiDAR sensors situated at a distance away from the human, and each point of the plurality of point cloud frames comprises a timestamp, and wherein the distance ranges from at least 10 to 50 meters.
14. The system of claim 12 , wherein the ground-truth three-dimensional motions are associated with the human and each ground-truth three-dimensional motion of the human comprises a timestamp, spatial coordinates, rotations of a plurality of joints of the human, and three-dimensional poses of the human.
15. The system of claim 12 , wherein the dataset further comprises synchronous images of the human.
16. The system of claim 15 , wherein the dataset further comprises labels for the synchronous images of the human, wherein each label comprises a two-dimension enclosure enclosing the human depicted in the synchronous images.
17. The system of claim 16 , wherein the instructions, when executed by the processor, further causes the system to perform:
training a second machine learning model using the synchronous images of the human in the dataset;
outputting, by the trained second machine learning model, a second three-dimensional motion of the human; and
evaluating the three-dimensional motion of the human based on the second three-dimensional motion of the human.
18. The system of claim 11 , wherein the feature learning network comprises a PointNet++ network.
19. The system of claim 18 , wherein the machine learning model further comprises a kinematics solver module for concatenating a frame-wise global feature with each joint to generate completed joint features and outputting the completed joint features to compute a plurality of joint rotations, wherein the kinematics solver comprises a ST-GCN configured to concatenate the frame-wise global features with each joint.
20. The system of claim 19 , wherein the machine learning model further comprises a joint optimizer module for optimizing the plurality of joint rotations, wherein the joint optimizer module comprises a SMPL model configured to optimize the rotations of the plurality of joint rotations.
21. A computer-implemented method for generating three-dimensional motions based on LiDAR point clouds, the method comprising:
inputting a plurality of point cloud frames to a machine learning model, wherein each point cloud frame comprises a plurality of points captured by a monocular LiDAR sensor, and the machine learning model comprises a temporal encoder module comprising a feature learning network, a two-way GRU, and an MLP decoder;
extracting, by the feature learning network, a global descriptor for each point cloud frame;
feeding the global descriptor into the two-way GRU to generate a plurality of hidden variables;
inputting the hidden variables to the MLP decoder to predict locations and rotations of a plurality of joints of an object;
outputting, by the machine learning model, a three-dimensional motion of the object based on the predicted locations and rotations of the plurality of joints.
22. The method of claim 21 , wherein the object is a human.
23. The method of claim 21 , further comprising:
training the machine learning model using a dataset comprising synchronous LiDAR point clouds captured by monocular LiDAR sensors and ground-truth three-dimensional motions obtained from IMU devices.
24. The method of claim 23 , wherein the synchronous LiDAR point clouds comprises a plurality of point cloud frames captured by the monocular LiDAR sensors situated at a distance away from a human, and each point of the plurality of point cloud frames comprises a timestamp and an intensity value, wherein the distance ranges from at least 10 to 50 meters.
25. The method of claim 21 , wherein the feature learning network is a PointNet++ network.
26. The method of claim 21 , wherein the feature learning network is a Point 4D Transformer.
27. The method of claim 21 , wherein the two-way GRU comprises a hidden layer configured to output the hidden variables.
28. The method of claim 21 , further comprising:
estimating the locations of the plurality of joints by the temporal encoder by minimizing a loss formulated as:
ℒ
𝒥
=
∑
t
J
GT
(
t
)
-
J
^
(
t
)
2
2
(
1
)
where Ĵ (t) is a predicted joint location of the t-th frame and J GT (t) is a ground truth joint (t) location of the t-th frame.
29. The method of claim 21 , wherein the machine learning model further comprises a kinematics solver module, wherein the kinematics solver module comprises a ST-CGN, and the method further comprises:
concatenating by the kinematics solver module the global descriptor with each joint to generate joint features; and
outputting the joint features to compute the rotations of the plurality of joints.
30. The method of claim 29 , wherein the machine learning model further comprises a joint optimizer module, wherein the joint optimizer module comprises an SMPL model, and the method further comprises:
inputting the rotations of the plurality of joints to the joint optimizer module to obtain optimized rotations of the plurality of joints.
31. The method of claim 30 , further comprising:
estimating the rotations of the plurality of joints by the temporal encoder module, the kinematics solver module, and the joint optimizer module.