System for training neural networks that predict the parameters of a human mesh model using dense depth and part-based UV map annotations
A system for training neural networks that predict the parameters of a human mesh model is disclosed herein. The system includes at least one camera and a data processor configured to execute computer executable instructions for: receiving a first frame and a second frame of a video from the at least one camera; extracting first and second image data from the first and second frames of the video; inputting the sequence of frames of the video into a human mesh estimator module, the human mesh estimator module estimating mesh parameters from the sequence of frames of the video so as to determine a predicted mesh; and generating a training signal for input into the human mesh estimator module by using a two-dimensional keypoint loss module that compares a first set of two-dimensional image-based keypoints to a second set of two-dimensional model-based keypoints.
1. A system for training neural networks that predict the parameters of a human mesh model, the system comprising:
at least one camera, the at least one camera configured to capture a sequence of frames of a video, the sequence of frames of the video including at least a first frame and a second frame; and
a data processor including at least one hardware component, the data processor configured to execute computer executable instructions, the computer executable instructions comprising instructions for:
receiving the first frame and the second frame of the video from the at least one camera;
extracting first image data from the first frame of the video;
extracting second image data from the second frame of the video;
inputting the sequence of frames of the video into a human mesh estimator module, the human mesh estimator module estimating mesh parameters from the sequence of frames of the video so as to determine a human mesh model;
generating a training signal for input into the human mesh estimator module by using a rigid transform loss module that calculates rigid transformations between the first and second frames of the sequence of frames for one or more body parts, warps a predicted mesh for the first frame from the human mesh model using the rigid transformations, and compares the warped mesh with a predicted mesh for the second frame from the human mesh model; and
generating another training signal for input into the human mesh estimator module by using a two-dimensional keypoint loss module that compares a first set of two-dimensional image-based keypoints to a second set of two-dimensional model-based keypoints.
2. The system according to claim 1 , wherein the data processor is further configured to estimate the first set of two-dimensional image-based keypoints from at least one of the first image data or the second image data using a two-dimensional keypoint estimator.
3. The system according to claim 1 , wherein the data processor is further configured to estimate the first set of two-dimensional image-based keypoints from a training dataset that comprises two-dimensional ground truth keypoints.
4. The system according to claim 1 , wherein the data processor is further configured to estimate the second set of two-dimensional model-based keypoints by projecting the second set of two-dimensional model-based keypoints from the human mesh model.
5. The system according to claim 1 , wherein, when generating the training signal for input into the human mesh estimator module using the two-dimensional keypoint loss module, the data processor is further configured to minimize numerical differences between corresponding keypoints of the first set of two-dimensional image-based keypoints and the second set of two-dimensional model-based keypoints so as to ensure a precise alignment between the human mesh model and real-world images in the video.
6. The system according to claim 1 , wherein the data processor is further configured to generate depth maps and IUV maps for the first frame and the second frame of the video.
7. The system according to claim 6 , wherein the data processor is configured to generate the depth maps using a human depth estimator module.
8. The system according to claim 6 , wherein the at least one camera comprises an RGB-D camera that outputs both image data and depth data, and the data processor is configured to generate the depth maps using the depth data from the RGB-D camera.
9. The system according to claim 1 , wherein the data processor is configured to estimate the second set of two-dimensional model-based keypoints by additionally considering a movement of the human mesh model over a period of time.