Keypoint based pose-tracking using entailment
Aspects of the present disclosure describe systems, methods and structures for an efficient multi-person posetracking method that advantageously achieves state-of-the-art performance on PoseTrack datasets by only using keypoint information in a tracking step without optical flow or convolution routines. As a consequence, our method has fewer parameters and FLOPs and achieves faster FPS. Our method benefits from our parameter-free tracking method that outperforms commonly used bounding box propagation in top-down methods. Finally, we disclose tokenization and embedding multi-person pose keypoint information in the transformer architecture that can be re-used for other pose tasks such as pose-based action recognition.
1. A pose tracking method comprising:
keypoint estimation of a person identified in multiple frames of a video;
temporal matching of poses identified by the keypoints; and
ID assignment of a current pose to other ones;
wherein temporal matching is performed by a transformer matching network and uses only keypoint information and no RGB information, and
wherein the transformer matching network is a multi-layer, multi-head network.
2. The method of claim 1 , wherein the keypoint estimation uses parameter-free bounding box propagation.
3. A keypoint based pose tracking method using entailment comprising:
extracting information from a sentence of tokens, each of which correspond to a keypoint on a pose;
embedding the keypoints of each pose and providing the embeddings to a transformer network;
matching a current pose to ones from multiple previous timesteps; and
outputting an indicia of that matched pose.
4. The method of claim 3 wherein the transformer network is a multi-layer, multi-head network.