Device and method of tracking poses of multiple objects based on single-object pose estimator

1. A method of tracking poses of multiple objects, comprising:

detecting, by one or more processors, objects from each of a plurality of consecutive frames in a video sequence, comprising: determining a bounding box for each of the objects from each of the plurality of consecutive frames; and determining a confidence score of the bounding box for the one of the objects within the one of the plurality of consecutive frames;

estimating, by the one or more processors, a pose of each of the objects within each of the plurality of consecutive frames, comprising: determining from a region defined by the bounding box for the one of the objects within the one of the plurality of consecutive frames, keypoints of the one of the objects within the one of the plurality of consecutive frames; determining confidence scores of the keypoints of the one of the objects within the one of the plurality of consecutive frames; and applying a model ensemble mode combing two or more pose estimation models to determine the keypoints of the one of the objects within the one of the plurality of consecutive frames; and

tracking, by the one or more processors, the poses of each of the objects across the plurality of consecutive frames.

2. The method of claim 1 , wherein the two or more pose estimation models comprise at least a first model and a second model, the model ensemble mode comprises an expert mode in which, for each of the keypoints of each of the objects within each of the plurality of consecutive frames, a weight of one is assigned to one of pose estimation result for the first model and pose estimation result for the second model that has a higher average precision (AP) for the one of the keypoints, and a weight of zero is assigned to the other of the pose estimation result for the first model and the pose estimation result for the second model.

3. The method of claim 1 , wherein the two or more pose estimation models comprise at least a first model and a second model, the model ensemble mode comprises an average mode in which, for all of the keypoints of each of the objects within each of the plurality of consecutive frames, a weight of 0.5 is assigned to pose estimation result for the first model and a weight of 0.5 is assigned to pose estimation result for the second model.

4. The method of claim 1 , wherein the two or more pose estimation models comprise at least a first model and a second model, one of the first model and the second model is epoch 291, and the other of the first model and the second model is epoch 293.

5. The method of claim 1 , wherein the step of tracking the poses of each of the objects across the plurality of consecutive frames comprises:

assigning, by the one or more processors, a same identifier to the bounding boxes indicating a same one of the objects across the plurality of consecutive frames; and

associating, by the one or more processors, the keypoints within the bounding boxes having the same identifier across the plurality of consecutive frames, so as to build a pose flow for each of the objects across the plurality of consecutive frames.

6. The method of claim 1 , wherein the step of tracking the poses of each of the objects across the plurality of consecutive frames further comprises:

performing, by the one or more processors, adaptive keypoint pruning on the keypoints of each of the objects within each of the plurality of consecutive frames with an adaptive keypoint pruner.

7. The method of claim 6 , wherein the step of performing the adaptive keypoint pruning on the keypoints of each of the objects within each of the plurality of consecutive frames with the adaptive keypoint pruner comprises:

determining, by the one or more processors, a keypoint drop threshold for each of predefined categories of keypoint, wherein each of the keypoints of each of the objects within each of the plurality of consecutive frames belongs to one of the predefined categories of keypoint;

dropping, by the one or more processors, a keypoint having a confidence score lower than a keypoint drop threshold for one of the predefined categories of keypoint to which the keypoint belongs; and

retaining, by the one or more processors, a keypoint having a confidence score greater than or equal to a keypoint drop threshold for one of the predefined categories of keypoint to which the keypoint belongs.

8. The method of claim 1 , wherein the step of detecting the objects from each of the plurality of consecutive frames in a video sequence is implemented by a deformable feature pyramid network (FPN) that is determined based on an object detector selecting mechanism.

9. The method of claim 1 , wherein the step of detecting the objects from each of the plurality of consecutive frames in a video sequence further comprises:

converting, by the one or more processors, the result of the detection for each of the objects within each of the plurality of consecutive frames into an openSVAI standardized data format, so as to generate a standardized detection result.

10. The method of claim 1 , wherein the step of estimating the pose of each of the objects within each of the plurality of consecutive frames further comprises:

converting, by the one or more processors, the result of the estimation for each of the objects within each of the plurality of consecutive frames into an openSVAI standardized data format, so as to generate a standardized estimation result.

11. A method of tracking poses of multiple objects, comprising:

detecting, by one or more processors, objects from each of a plurality of consecutive frames in a video sequence, comprising: determining a bounding box for each of the objects from each of the plurality of consecutive frames, and determining a confidence score of the bounding box for the one of the objects within the one of the plurality of consecutive frames;

estimating, by the one or more processors, a pose of each of the objects within each of the plurality of consecutive frames; and

tracking, by the one or more processors, the poses of each of the objects across the plurality of consecutive frames,

wherein the step of estimating the pose of each of the objects within each of the plurality of consecutive frames comprises:

regressing, by the one or more processors, a region defined by the bounding box for the one of the objects within the one of the plurality of consecutive frames into heatmaps of the one of the objects within the one of the plurality of consecutive frames;

suppressing, by the one or more processors, the heatmaps of the one of the objects within the one of the plurality of consecutive frames into keypoints of the one of the objects within the one of the plurality of consecutive frames with cross-heatmap pose non-maximum suppression (NMS); and

determining, by the one or more processors, confidence scores of the keypoints of the one of the objects within the one of the plurality of consecutive frames.

12. A device for tracking poses of multiple objects, comprising:

a processor; and

a memory storing instructions which, when executed by the processor, cause the processor to:

detect objects from each of a plurality of consecutive frames in a video sequence by: determining a bounding box for each of the objects from each of the plurality of consecutive frames; and determining a confidence score of the bounding box for the one of the objects within the one of the plurality of consecutive frames;

estimate a pose of each of the objects within each of the plurality of consecutive frames by: determining from a region defined by the bounding box for the one of the objects within the one of the plurality of consecutive frames, keypoints of the one of the objects within the one of the plurality of consecutive frames; determining confidence scores of the keypoints of the one of the objects within the one of the plurality of consecutive frames; and applying a model ensemble mode combing two or more pose estimation models to determine the keypoints of the one of the objects within the one of the plurality of consecutive frames; and

track the poses of each of the objects across the plurality of consecutive frames.

13. The device of claim 12 , wherein the instructions which, when executed by the processor, further cause the processor to:

assign a same identifier to the bounding boxes indicating a same one of the objects across the plurality of consecutive frames; and

associate the keypoints within the bounding boxes having the same identifier across the plurality of consecutive frames, so as to build a pose flow for each of the objects across the plurality of consecutive frames.

14. The device of claim 12 , wherein the instructions which, when executed by the processor, further cause the processor to:

convert the result of the detection for each of the objects within each of the plurality of consecutive frames into an openSVAI standardized data format, so as to generate a standardized detection result; and

convert the result of the estimation for each of the objects within each of the plurality of consecutive frames into an openSVAI standardized data format, so as to generate a standardized estimation result.