Method and apparatus with heat map-based pose estimation

1 . A processor-implemented method with pose estimation, the method comprising:

receiving an image, instance queries for object identification, and joint queries for joint identification;

generating first heat maps each associated with a plurality of joints included in a respective one of a plurality of objects and second heat maps each associated with a single type of joint included in each of the plurality of objects in parallel based on the image, the instance queries, and the joint queries; and

estimating respective joint information of the plurality of objects included in the image by combining the first heat maps and the second heat maps.

2 . The method of claim 1 , wherein

each of the first heat maps comprises information on positions of all joints of any one of the plurality of objects, and

each of the second heat maps comprises information on a position of the single type of joint included in any one of the plurality of objects.

3 . The method of claim 1 , wherein the first heat maps and the second heat maps correspond to two-dimensional images representing a probability that a joint is present at each pixel location.

4 . The method of claim 1 , wherein the generating in parallel comprises:

generating an image feature based on the image; and

generating output instance queries and output joint queries by inputting the image feature, the instance queries, and the joint queries to a transformer decoder.

5 . The method of claim 4 , wherein

each of the output instance queries comprises information on a single object identified in the image, and

each of the output joint queries comprises information on the single type of joint identified in the image.

6 . The method of claim 4 , wherein multi-head self-attention is performed on the instance queries and the joint queries through the transformer decoder.

7 . The method of claim 4 , wherein

the transformer decoder comprises a plurality of layers, and

the instance queries and the joint queries are gradually updated by sequentially passing through the plurality of layers to generate the output instance queries and the output joint queries.

8 . The method of claim 4 , wherein the generating in parallel further comprises:

generating a high-resolution image feature based on the image feature;

generating the first heat maps based on the high-resolution image feature and the output instance queries;

generating the second heat maps based on the high-resolution image feature and the output joint queries; and

generating an object score by scoring each of the output instance queries.

9 . The method of claim 1 , wherein the estimating comprises estimating the respective joint information of the plurality of objects by multiplying first heat maps having an object score greater than or equal to a threshold value by the second heat maps.

10 . The method of claim 1 , wherein

the joint information comprises coordinates of a joint and a probability that a joint is present at the coordinates, and

the pose estimation method further comprises estimating poses of the plurality of objects included in the image based on coordinates of a joint having the probability greater than a threshold value.

11 . A processor-implemented method with pose estimation, the method comprising:

receiving an image feature generated based on an image, instance queries for object identification, and joint queries for joint identification;

generating output instance queries and output joint queries in parallel by inputting the image feature, the instance queries, and the joint queries to a transformer decoder trained based on first heat maps each associated with a plurality of joints included in a respective one of a plurality of objects and second heat maps each associated with a single type of joint included in each of the plurality of objects; and

estimating respective poses of a plurality of objects included in the image based on either one or both of the output instance queries and the output joint queries.

12 . The method of claim 11 , wherein the transformer decoder is trained based on the first heat maps associated with the objects and the second heat maps associated with the joints.

13 . The method of claim 12 , wherein the first heat maps and the second heat maps correspond to two-dimensional images representing a probability that a joint is present at each pixel location.

14 . The method of claim 12 , wherein the estimating of the respective poses comprises generating a box regression result and a pose regression result based on the output instance queries.

15 . The method of claim 12 , wherein the estimating of the respective poses comprises:

generating a box regression result based on the output instance queries; and

generating a pose regression result based on a combination of the output instance queries and the output joint queries.

16 . The method of claim 15 , wherein

the first heat maps are generated based on the box regression result, the pose regression result, and a high-resolution image feature, and

the second heat maps are generated based on the high-resolution image feature and the output instance queries.

17 . The method of claim 11 , wherein

each of the output instance queries comprises information on a single object identified in the image, and

each of the output joint queries comprises information on the single type of joint identified in the image.

18 . The method of claim 11 , wherein the estimating of the respective poses comprises:

generating a high-resolution image feature based on the image feature;

generating first heat maps associated with objects based on the high-resolution image feature and the output instance queries; and

generating second heat maps associated with joints based on the high-resolution image feature and the output joint queries.

19 . The method of claim 18 , wherein the estimating of the respective poses further comprises:

estimating reference points based on the first heat maps; and

generating a pose regression result based on the reference points, the output instance queries, and the output joint queries.

20 . An apparatus with pose estimation, the apparatus comprising:

one or more processors configured to:

receive an image, instance queries for object identification, and joint queries for joint identification;

generate first heat maps each associated with a plurality of joints included in a respective one of a plurality of objects and second heat maps each associated with a single type of joint included in each of the plurality of objects in parallel based on the image, the instance queries, and the joint queries; and

estimate respective joint information of the plurality of objects included in the image by combining the first heat maps and the second heat maps.