Performing occlusion-aware global 3D pose and shape estimation of articulated objects
In order to determine accurate three-dimensional (3D) models for objects within a video, the objects are first identified and tracked within the video, and a pose and shape are estimated for these tracked objects. A translation and global orientation are removed from the tracked objects to determine local motion for the objects, and motion infilling is performed to fill in any missing portions for the object within the video. A global trajectory is then determined for the objects within the video, and the infilled motion and global trajectory are then used to determine infilled global motion for the object within the video. This enables the accurate depiction of each object as a 3D pose sequence for that model that accounts for occlusions and global factors within the video.
1. A method comprising, at a device:
performing motion infilling for one or more tracked objects within a video, wherein a translation and global orientation are removed from the one or more tracked objects within the video prior to performing the motion infilling;
predicting a global trajectory for each of the one or more tracked objects within the video; and
for the one or more tracked objects within the video, combining the infilled motion for the tracked object with the global trajectory for the tracked object to determine infilled global motion for the tracked object.
2. The method of claim 1 , wherein the video is monocular.
3. The method of claim 1 , wherein the one or more tracked objects each have one or more missing portions resulting from occlusion of the tracked objects in one or more frames of the video.
4. The method of claim 1 , wherein the one or more tracked objects each have one or more missing portions resulting from truncation of the tracked objects in one or more frames of the video.
5. The method of claim 1 , wherein the one or more tracked objects each have one or more missing portions resulting from the object moving out of a scene in one or more frames of the video.
6. The method of claim 1 , wherein in response to identifying one or more missing portions for one of the tracked objects within a predetermined frame of the video, previous pose and shape data for the tracked object is identified within previous frames of the video.
7. The method of claim 6 , wherein the previous pose and shape data for the tracked object is input into a trained neural network architecture, where the trained neural network architecture predicts pose and shape data for the object within the predetermined frame of the video, and uses the predicted pose and shape data to fill in the missing portions for the tracked object within the predetermined frame of the video.
8. The method of claim 1 , wherein after the motion infilling is performed for each of the one or more tracked objects within the video, the global trajectory is predicted for each of the one or more tracked object, using infilled local motion for each of the one or more tracked objects.
9. The method of claim 1 , wherein the infilled global motion is determined with respect to a global coordinate system.
10. The method of claim 1 , wherein one or more camera parameters are accounted for during the determination of the infilled global motion.
11. A system comprising:
a hardware processor of a device that is configured to:
perform motion infilling for one or more tracked objects within a video, wherein a translation and global orientation are removed from the one or more tracked objects within the video prior to performing the motion infilling;
predict a global trajectory for each of the one or more tracked objects within the video; and
for the one or more tracked objects within the video, combine the infilled motion for the tracked object with the global trajectory for the tracked object to determine infilled global motion for the tracked object.
12. The system of claim 11 , wherein the video is monocular.
13. The system of claim 11 , wherein the one or more tracked objects each have one or more missing portions resulting from occlusion of the tracked objects in one or more frames of the video.
14. The system of claim 11 , wherein the one or more tracked objects each have one or more missing portions resulting from truncation of the tracked objects in one or more frames of the video.
15. The system of claim 11 , wherein the one or more tracked objects each have one or more missing portions resulting from the object moving out of a scene in one or more frames of the video.
16. The system of claim 11 , wherein in response to identifying one or more missing portions for one of the tracked objects within a predetermined frame of the video, previous pose and shape data for the tracked object is identified within previous frames of the video.
17. The system of claim 16 , wherein the previous pose and shape data for the tracked object is input into a trained neural network architecture, where the trained neural network architecture predicts pose and shape data for the object within the predetermined frame of the video, and uses the predicted pose and shape data to fill in the missing portions for the tracked object within the predetermined frame of the video.
18. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor of a device, causes the processor to cause the device to:
perform motion infilling for one or more tracked objects within a video, wherein a translation and global orientation are removed from the one or more tracked objects within the video prior to performing the motion infilling;
predict a global trajectory for each of the one or more tracked objects within the video; and
for the one or more tracked objects within the video, combine the infilled motion for the tracked object with the global trajectory for the tracked object to determine infilled global motion for the tracked object.
19. The computer-readable storage medium of claim 18 , wherein:
in response to identifying one or more missing portions for one of the tracked objects within a predetermined frame of the video, previous pose and shape data for the tracked object is identified within previous frames of the video, and
the previous pose and shape data for the tracked object is input into a trained neural network architecture, where the trained neural network architecture predicts pose and shape data for the object within the predetermined frame of the video, and uses the predicted pose and shape data to fill in the missing portions for the tracked object within the predetermined frame of the video.