Granted Patent B2

US 12,266,147 · App. 17/747,837 · Granted Apr 1, 2025

Inventor: Yang Zhou (Palo Alto, CA)

Assignee: GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP., LTD.

G06V10/44G06T7/73G06V10/82G06V40/11G06T2207/20084G06T2207/30196

Abstract

Claims (91)

1. A hand posture estimation method, comprising:

obtaining an initial feature map corresponding to a hand region in a candidate image;

obtaining a fused feature map by performing feature fusion processing on the initial feature map; wherein the feature fusion processing is configured to fuse features around a plurality of key points;

obtaining a target feature map by performing deconvolution processing on the fused feature map; wherein the deconvolution processing is configured to adjust a resolution of the fused feature map; and

obtaining coordinate information of the plurality of key points based on the target feature map to determine a posture estimation result of the hand region in the candidate image;

wherein the obtaining a fused feature map by performing feature fusion processing on the initial feature map comprises:

obtaining a first feature map by performing a first convolution processing on the initial feature map through a first convolutional network; wherein the first convolution processing is configured to extract local detail information of the plurality of key points; and

wherein before the performing a first convolution processing on the initial feature map through a first convolutional network, the method further comprises:

obtaining a dimensionality-reduced feature map by performing dimensionality reduction processing on the initial feature map; wherein the dimensionality reduction processing is configured to reduce the number of channels of the initial feature map; and

obtaining the first feature map by performing the first convolution processing on the dimensionality-reduced feature map through the first convolutional network.

2. The method according to claim 1 , wherein the initial feature map is a region-of-interest alignment (RoIAlign) feature map.

3. The method according to claim 2 , wherein the obtaining an initial feature map corresponding to a hand region in a candidate image comprises:

performing identification processing on an image content of the candidate image, and determining the hand region in the candidate image; and

performing RoIAlign feature extraction on the hand region in the candidate image, and obtaining the initial feature map.

4. The method according to claim 3 , wherein,

the performing identification processing on an image content of the candidate image, and determining the hand region in the candidate image comprise:

obtaining the candidate image containing a hand, and identifying and positioning the hand region of the candidate image by means of a bounding box detection; and

the performing RoIAlign feature extraction on the hand region in the candidate image, and obtaining the initial feature map comprise:

performing shallow feature extraction on the hand region with a RoIAlign feature extractor to obtain a RoIAlign feature map, the hand region comprising an approximate outline and an edge position of the hand; and

taking the RoIAlign feature map as the initial feature map.

5. The method according to claim 1 , wherein the obtaining a fused feature map by performing feature fusion processing on the initial feature map further comprises:

obtaining a first down-sampled feature map by performing a first down-sampling processing on the first feature map;

obtaining a second feature map by performing a second convolution processing on the first down-sampled feature map through a second convolutional network; wherein the second convolution processing is configured to extract context information of the plurality of key points; and

obtaining the fused feature map by performing a second down-sampling processing on the second feature map.

6. The method according to claim 5 , wherein the first convolutional network comprises N sub-convolutional networks, where N is an integer greater than 1;

the obtaining a first feature map by performing a first convolution processing on the initial feature map through a first convolutional network comprises:

in condition of i being equal to 1, performing an i-th convolution processing on the initial feature map through an i-th sub-convolutional network, outputting an i-th feature map, and obtaining an i-th weighted sum feature map by performing weighted summing processing on the initial feature map and the i-th feature map, where i is an integer greater than or equal to 1 and less than N;

in condition of i being not equal to 1, performing the i-th convolution processing on an i-1th weighted sum feature map through an i-th sub-convolutional network, outputting an i-th feature map, and obtaining an i-th weighted sum feature map by performing weighted summing processing on the i-1th weighted sum feature map and the i-th feature map;

performing an i+1th convolution processing on the i-th weighted sum feature map through an i+1th sub-convolutional network, and so on until performing an N-th convolution processing on an N-1th weighted sum feature map through an N-th sub-convolutional network; and outputting an N-th weighted sum feature map; and

obtaining the first feature map by performing weighted summing processing on the N-th weighted sum feature map and an N-1th feature map.

7. The method according to claim 5 , wherein the obtaining a second feature map by performing a second convolution processing on the first down-sampled feature map through a second convolutional network comprises:

performing the second convolution processing on the first down-sampled feature map through the second convolutional network, and outputting a second convolutional feature map; and

obtaining the second feature map by performing weighted summing processing on the second convolutional feature map and the first down-sampled feature map.

8. The method according to claim 5 , wherein the obtaining a target feature map by performing deconvolution processing on the fused feature map comprises:

obtaining a first up-sampled feature map by performing a first up-sampling processing on the fused feature map;

obtaining a third feature map by performing a third convolution processing on the first up-sampled feature map through a third convolutional network;

obtaining a second up-sampled feature map by performing a second up-sampling processing on the third feature map;

obtaining a fourth feature map by performing a fourth convolution processing on the second up-sampled feature map through a fourth convolutional network; and

obtaining the target feature map by performing a third up-sampling processing on the fourth feature map.

9. The method according to claim 8 , wherein the obtaining a third feature map by performing a third convolution processing on the first up-sampled feature map through a third convolutional network comprises:

performing the third convolution processing on the first up-sampled feature map through the third convolutional network, and outputting a third convolutional feature map; and

obtaining the third feature map by performing weighted summing processing on the third convolutional feature map and the second feature map.

10. The method according to claim 8 , wherein the obtaining a fourth feature map by performing a fourth convolution processing on the second up-sampled feature map through a fourth convolutional network comprises:

performing a fourth convolution processing on the second up-sampled feature map through the fourth convolutional network, and outputting a fourth convolutional feature map; and

obtaining the fourth feature map by performing weighted summing processing on the fourth convolutional feature map and the first feature map.

11. An electronic device, comprising a memory and a processor; wherein,

the memory is configured to store executable instructions executable on the processor;

when running the executable instructions, the processor is configured to perform:

obtaining an initial feature map corresponding to a hand region in a candidate image;

obtaining a fused feature map by performing feature fusion processing on the initial feature map; wherein the feature fusion processing is configured to fuse features around a plurality of key points; the plurality of key points represent skeleton key nodes of the hand region;

obtaining a target feature map by performing deconvolution processing on the fused feature map; wherein the deconvolution processing is configured to adjust a resolution of the fused feature map; and

obtaining coordinate information of the plurality of key points based on the target feature map to determine a posture estimation result of the hand region in the candidate image;

wherein the obtaining a fused feature map by performing feature fusion processing on the initial feature map comprises:

obtaining a first feature map by performing a first convolution processing on the initial feature map through a first convolutional network; wherein the first convolution processing is configured to extract local detail information of the plurality of key points; and

wherein before the performing a first convolution processing on the initial feature map through a first convolutional network, the method further comprises:

obtaining a dimensionality-reduced feature map by performing dimensionality reduction processing on the initial feature map; wherein the dimensionality reduction processing is configured to reduce the number of channels of the initial feature map; and

obtaining the first feature map by performing the first convolution processing on the dimensionality-reduced feature map through the first convolutional network.

12. The electronic device according to claim 11 , wherein the initial feature map is a region-of-interest alignment (RoIAlign) feature map.

13. The electronic device according to claim 12 , wherein in the obtaining an initial feature map corresponding to a hand region in a candidate image, the processor is further configured to perform:

performing identification processing on an image content of the candidate image, and determining the hand region in the candidate image; and

performing RoIAlign feature extraction on the hand region in the candidate image, and obtaining the initial feature map.

14. The electronic device according to claim 11 , wherein in the obtaining a fused feature map by performing feature fusion processing on the initial feature map, the processor is further configured to perform:

obtaining a first down-sampled feature map by performing a first down-sampling processing on the first feature map;

obtaining a second feature map by performing a second convolution processing on the first down-sampled feature map through a second convolutional network, wherein the second convolution processing is configured to extract context information of the plurality of key points; and

obtaining the fused feature map by performing a second down-sampling processing on the second feature map.

15. The electronic device according to claim 14 , wherein the first convolutional network comprises N sub-convolutional networks, where N is an integer greater than 1;

in the obtaining a first feature map by performing a first convolution processing on the initial feature map through a first convolutional network, the processor is further configured to perform:

in condition of i being equal to 1, performing an i-th convolution processing on the initial feature map through an i-th sub-convolutional network, outputting an i-th feature map, and obtaining an i-th weighted sum feature map by performing weighted summing processing on the initial feature map and the i-th feature map, where i is an integer greater than or equal to 1 and less than N;

in condition of i being not equal to 1, performing the i-th convolution processing on an i-1th weighted sum feature map through an i-th sub-convolutional network, outputting an i-th feature map, and obtaining an i-th weighted sum feature map by performing weighted summing processing on the i-1th weighted sum feature map and the i-th feature map;

performing an i+1th convolution processing on the i-th weighted sum feature map through an i+1th sub-convolutional network, and so on until performing an N-th convolution processing on an N-1th weighted sum feature map through an N-th sub-convolutional network; and outputting an N-th weighted sum feature map; and

obtaining the first feature map by performing weighted summing processing on the N-th weighted sum feature map and an N-1th feature map.

16. The electronic device according to claim 14 , wherein in the obtaining a second feature map by performing a second convolution processing on the first down-sampled feature map through a second convolutional network, the processor is further configured to perform:

performing the second convolution processing on the first down-sampled feature map through the second convolutional network, and outputting a second convolutional feature map; and

obtaining the second feature map by performing weighted summing processing on the second convolutional feature map and the first down-sampled feature map.

17. The electronic device according to claim 14 , wherein in the obtaining a target feature map by performing deconvolution processing on the fused feature map, the processor is further configured to perform:

obtaining a first up-sampled feature map by performing a first up-sampling processing on the fused feature map;

obtaining a third feature map by performing a third convolution processing on the first up-sampled feature map through a third convolutional network;

obtaining a second up-sampled feature map by performing a second up-sampling processing on the third feature map;

obtaining a fourth feature map by performing a fourth convolution processing on the second up-sampled feature map through a fourth convolutional network; and

obtaining the target feature map by performing a third up-sampling processing on the fourth feature map.

18. A non-transitory computer-readable storage medium, storing a computer program;

wherein the computer program is executed by a first processor to perform:

obtaining an initial feature map corresponding to a hand region in a candidate image;

obtaining a fused feature map by performing feature fusion processing on the initial feature map; wherein the feature fusion processing is configured to fuse features around a plurality of key points; the plurality of key points represent skeleton key nodes of the hand region;

obtaining a target feature map by performing deconvolution processing on the fused feature map; wherein the deconvolution processing is configured to adjust a resolution of the fused feature map; and

obtaining coordinate information of the plurality of key points based on the target feature map to determine a posture estimation result of the hand region in the candidate image;

wherein the obtaining a fused feature map by performing feature fusion processing on the initial feature map comprises:

obtaining a first feature map by performing a first convolution processing on the initial feature map through a first convolutional network; wherein the first convolution processing is configured to extract local detail information of the plurality of key points; and

wherein before the performing a first convolution processing on the initial feature map through a first convolutional network, the method further comprises:

obtaining a dimensionality-reduced feature map by performing dimensionality reduction processing on the initial feature map; wherein the dimensionality reduction processing is configured to reduce the number of channels of the initial feature map; and

obtaining the first feature map by performing the first convolution processing on the dimensionality-reduced feature map through the first convolutional network.

Assignments (1)

Continuity (3)

References Cited (17)