Granted Patent B2

US 12,205,397 · App. 17/463,166 · Granted Jan 21, 2025

Inventors: Xiaomin Liu (Sunnyvale, CA); Lei Zhang (Campbell, CA); Qun Gu (San Jose, CA)

Assignee: Black Sesame Technologies Inc.

G06V40/103G06T7/73G06V10/50

Abstract

Claims (56)

1. A method of human pose estimation, comprising:

receiving an image frame containing at least one image feature wherein the image feature includes human body part locations;

extracting via a neural network in at least an initial stage an initial stage image feature based on the image frame;

extracting in at least a refinement stage a refinement stage image feature based on the first stage image feature;

extracting in at least a subsequent stage a subsequent stage image feature based on the refinement stage image feature;

up-sampling the subsequent stage image feature;

up-sampling the refinement stage image feature;

after the upsampling, concatenating the first stage image feature, the up-sampled second stage image feature and the up-sampled subsequent stage image feature;

outputting a feature map based on an output of the concatenation; and

determining a human pose estimation.

2. The method of human pose estimation of claim 1 , wherein the feature map is based on a bottom-up determination.

3. The method of human pose estimation of claim 1 , wherein the image frame is at least one of red-blue-green and infrared.

4. The method of human pose estimation of claim 1 , wherein the extracting in at least the first stage utilizes a resnet-v1-50.

5. The method of human pose estimation of claim 1 , wherein the up-sampling of the subsequent stage image feature is by four.

6. The method of human pose estimation of claim 1 , wherein the up-sampling of the second stage image feature is by two.

7. The method of human pose estimation of claim 1 , further comprising a third stage to extract a third stage image feature.

8. The method of human pose estimation of claim 7 , wherein the feature map is based on a bottom-up determination.

9. The method of human pose estimation of claim 7 , wherein the heat map predicts a location of a body part and the part affinity map predicts an association of the body part.

10. The method of human pose estimation of claim 7 , wherein the convolution if the feature map utilizes a resnet-v1-50.

11. The method of human pose estimation of claim 7 , further comprising determining a body part confidence map based on the refinement part affinity map.

12. The method of human pose estimation of claim 7 , wherein the concatenating utilizes a residual connection of the feature map.

13. The method of human pose estimation of claim 7 , wherein the convolution of the feature map includes at least one 3×3 convolution and at least one 1×1 convolution.

14. The method of human pose estimation of claim 13 , wherein the feature map is based on a bottom-up determination.

15. The method of human pose estimation of claim 13 , wherein the heat map predicts a location of a body part and the part affinity map predicts an association of the body part.

16. The method of human pose estimation of claim 13 , wherein the convolution of the feature map utilizes a resnet-v1-50.

17. The method of human pose estimation of claim 13 , further comprising determining a body part confidence map based on the refinement part affinity map.

18. The method of human pose estimation of claim 13 , wherein the concatenating of the first stage, The second stage and the third stage utilities a residual connection of the feature map.

19. The method of human pose estimation of claim 13 , wherein the convolution of the feature map includes at least one 3×3 convolution and at least one 1×1 convolution.

20. A method of human pose estimation, comprising:

receiving a feature map based on an image frame containing at least one image feature wherein the image feature includes human body part locations by an initial stage;

convoluting the feature map based on an initial part affinity map branch of the initial stage;

determining an initial part affinity map based on an output of the initial part affinity map branch;

convoluting the feature map based on an initial heat map branch of the initial stage;

determining an initial heat map based on an output of the initial heat map branch;

concatenating the initial part affinity map branch, the initial heat map branch and the feature map to output an initial stage concatenation;

receiving the initial stage concatenation by a refinement stage;

convoluting the initial stage concatenation via refinement part affinity map branch of the refinement stage;

determining a refinement part affinity map based on an output of the refinement part affinity map branch;

convoluting the initial stage concatenation via refinement heat map branch of the refinement stage;

determining a refinement heat map based on an output of the refinement heat map branch; and

determining a human pose estimation.

21. A method of human pose estimation, comprising:

receiving a feature map based on an image frame containing human body parts by an initial stage;

convoluting the feature map based on a first part affinity map branch of the initial stage;

determining a first stage part affinity map based on an output of the first part affinity map branch;

concatenating the first stage part affinity map and the feature map to output a first stage part affinity map concatenation;

convoluting the first stage part affinity map concatenation to output a second stage part affinity map convolution;

determining second stage part affinity map based on an output of the second stage part affinity map convolution;

concatenating the second stage part affinity map and the feature map to output a second stage part affinity map concatenation;

receiving the second stage part affinity map concatenation by a refinement stage;

convoluting the second stage part affinity map concatenation to output a third stage part affinity map convolution;

determining a third stage part affinity map based on an output of the third stage part affinity map convolution;

concatenating the third stage part affinity map and the feature map to output a concatenated third stage part affinity map;

convoluting the concatenated third stage part affinity map to output a refinement heatmap prediction;

determining a refinement heat map based on an output of the refinement heatmap prediction; and

determining a human pose estimation.

Assignments (2)

Continuity (1)

References Cited (11)