Hand key point recognition model training method, hand key point recognition method and device

1. A hand key-point recognition model training method for a model training device, comprising:

converting a sample virtual image into an emulation image by using a Cycle-GAN model, the sample virtual image being an image generated through three-dimensional modeling, the sample virtual image comprising key-point coordinates corresponding to hand key-points, and the emulation image being used for emulating an image acquired in a real scenario;

extracting a hand image in the emulation image; and

training a hand key-point recognition model according to the hand image in the emulation image and the key-point coordinates, the hand key-point recognition model being used for outputting hand key-point coordinates of a hand in a real image according to the inputted real image;

wherein the training the hand key-point recognition model according to the hand image in the emulation image and the key-point coordinates comprises:

constructing the hand key-point recognition model, the hand key-point recognition model comprising a two-dimensional recognition branch and a three-dimensional recognition branch, the two-dimensional recognition branch comprising a plurality of two-dimensional residual layers and a convolution layer, and the three-dimensional recognition branch comprising a plurality of three-dimensional residual layers and a fully connected layer;

calculating a two-dimensional recognition loss and a three-dimensional recognition loss of the hand key-point recognition model according to the hand image and the key-point coordinates; and

reversely training the hand key-point recognition model according to the two-dimensional recognition loss and the three-dimensional recognition loss.

2. The method according to claim 1 , wherein before converting the sample virtual image, the method further comprises:

training the Cycle-GAN model according to a sample real image and the sample virtual image, the sample real image being an image acquired in the real scenario.

3. The method according to claim 2 , wherein the training the Cycle-GAN model according to a sample real image and the sample virtual image comprises:

constructing a first generator and a second generator, the first generator being configured to generate the emulation image, and the second generator being configured to generate a pseudo-virtual image, the pseudo-virtual image being used for emulating a style of the sample virtual image;

constructing a first discriminator and a second discriminator, the first discriminator being configured to discriminate the sample virtual image from the pseudo-virtual image, and the second discriminator being configured to discriminate the sample real image from the emulation image;

constructing the Cycle-GAN model according to the first generator, the second generator, the first discriminator, and the second discriminator;

calculating a loss of the Cycle-GAN model according to the sample real image and the sample virtual image, the loss of the Cycle-GAN model comprising a generator loss, a discriminator loss, and a cycle loss; and

reversely training the Cycle-GAN model according to the loss of the Cycle-GAN model.

4. The method according to claim 3 , wherein the converting a sample virtual image into an emulation image by using a Cycle-GAN model comprises:

converting the sample virtual image into the emulation image by using the first generator in the Cycle-GAN model.

5. The method according to claim 1 , wherein the extracting a hand image in the emulation image comprises:

inputting the emulation image into a gesture segmentation neural network, to output a hand confidence region, the gesture segmentation neural network being obtained by training according to images labeled with hand regions; and

extracting the hand image from the emulation image according to the hand confidence region.

6. The method according to claim 5 , wherein:

the gesture segmentation neural network comprises n convolution layers and n deconvolution layers, n being an integer greater than or equal to 2; and

the inputting the emulation image into a gesture segmentation neural network to output a hand confidence region comprises:

processing the emulation image through the n convolution layers, to obtain an intermediate feature map corresponding to the emulation image;

processing the intermediate feature map through the n deconvolution layers, to obtain a target feature map corresponding to the emulation image;

generating a probabilistic matrix according to the target feature map, the probabilistic matrix being used for representing a probability of each pixel point in the emulation image being the hand; and

performing binarization processing on the probabilistic matrix, to output a heatmap corresponding to the hand region, the heatmap being used for indicating the hand confidence region in the emulation image.

7. The method according to claim 6 , wherein the processing the intermediate feature map through the n deconvolution layers comprises:

splicing a feature map outputted by an m th deconvolution layer and a feature map outputted by an (n−m) th convolution layer, 1≤m≤n−1; and

processing a spliced feature map through an (m+1) th deconvolution layer.

8. The method according to claim 1 , wherein the plurality of two-dimensional residual layers comprises i two-dimensional residual layers, the plurality of three-dimensional residual layers comprises i three-dimensional residual layers, and first j two-dimensional residual layers being coupled to first j three-dimensional residual layers, 2≤j≤i−1, i and j being integers; and wherein the calculating a two-dimensional recognition loss and a three-dimensional recognition loss of the hand key-point recognition model according to the hand image and the key-point coordinates comprises:

inputting the hand image into the two-dimensional recognition branch and the three-dimensional recognition branch respectively;

adding a feature map outputted by a k th two-dimensional residual layer with a feature map outputted by a k th three-dimensional residual layer, 1≤k≤j−1;

inputting a feature map obtained after the adding into a (k+1) th two-dimensional residual layer and a (k+1) th three-dimensional residual layer;

processing a feature map outputted by the j th two-dimensional residual layer through (j+1) th to i th two-dimensional residual layers and the convolution layer, to obtain a two-dimensional recognition result;

processing a feature map outputted by the j th three-dimensional residual layer through (j+1) th to i th three-dimensional residual layers and the fully connected layer, to obtain a three-dimensional recognition result;

calculating the two-dimensional recognition loss according to the two-dimensional recognition result and the key-point coordinates; and

calculating the three-dimensional recognition loss according to the three-dimensional recognition result and the key-point coordinates.

9. A hand key-point recognition method for a hand key-point recognition device, comprising:

acquiring a real image, the real image comprising a hand image;

extracting the hand image in the real image; and

inputting the hand image into a hand key-point recognition model, to obtain hand key-point coordinates,

wherein the hand key-point recognition model is obtained by training according to an emulation image and three-dimensional label data of a hand image in the emulation image, the emulation image being generated by a Cycle-GAN model according to a sample virtual image, the Cycle-GAN model being generated according to a sample real image and the sample virtual image, the sample virtual image being an image generated through three-dimensional modeling, and the sample virtual image comprising key-point coordinates corresponding to hand key-points, comprising:

constructing the hand key-point recognition model, the hand key-point recognition model comprising a two-dimensional recognition branch and a three-dimensional recognition branch, the two-dimensional recognition branch comprising a plurality of two-dimensional residual layers and a convolution layer, and the three-dimensional recognition branch comprising a plurality of three-dimensional residual layers and a fully connected layer;

calculating a two-dimensional recognition loss and a three-dimensional recognition loss of the hand key-point recognition model according to the hand image and the key-point coordinates; and

reversely training the hand key-point recognition model according to the two-dimensional recognition loss and the three-dimensional recognition loss.

10. The method according to claim 9 , wherein:

the real image is acquired by the hand key-point recognition device using an image acquisition device; and

the hand image is extracted using a gesture segmentation neural network.

11. A model training device, comprising:

a memory storing computer program instructions; and

a processor coupled to the memory and, when executing the computer program instructions, configured to perform:

converting a sample virtual image into an emulation image by using a Cycle-GAN model, the sample virtual image being an image generated through three-dimensional modeling, the sample virtual image comprising key-point coordinates corresponding to hand key-points, and the emulation image being used for emulating an image acquired in a real scenario;

extracting a hand image in the emulation image; and

training a hand key-point recognition model according to the hand image in the emulation image and the key-point coordinates, the hand key-point recognition model being used for outputting hand key-point coordinates of a hand in a real image according to the inputted real image;

wherein the training the hand key-point recognition model according to the hand image in the emulation image and the key-point coordinates comprises:

constructing the hand key-point recognition model, the hand key-point recognition model comprising a two-dimensional recognition branch and a three-dimensional recognition branch, the two-dimensional recognition branch comprising a plurality of two-dimensional residual layers and a convolution layer, and the three-dimensional recognition branch comprising a plurality of three-dimensional residual layers and a fully connected layer;

calculating a two-dimensional recognition loss and a three-dimensional recognition loss of the hand key-point recognition model according to the hand image and the key-point coordinates; and

reversely training the hand key-point recognition model according to the two-dimensional recognition loss and the three-dimensional recognition loss.

12. The model training device according to claim 11 , wherein, before converting the sample virtual image, the processor is further configured to perform:

training the Cycle-GAN model according to a sample real image and the sample virtual image, the sample real image being an image acquired in the real scenario.

13. The model training device according to claim 12 , wherein the training the Cycle-GAN model according to a sample real image and the sample virtual image comprises:

constructing a first generator and a second generator, the first generator being configured to generate the emulation image, and the second generator being configured to generate a pseudo-virtual image, the pseudo-virtual image being used for emulating a style of the sample virtual image;

constructing a first discriminator and a second discriminator, the first discriminator being configured to discriminate the sample virtual image from the pseudo-virtual image, and the second discriminator being configured to discriminate the sample real image from the emulation image;

constructing the Cycle-GAN model according to the first generator, the second generator, the first discriminator, and the second discriminator;

calculating a loss of the Cycle-GAN model according to the sample real image and the sample virtual image, the loss of the Cycle-GAN model comprising a generator loss, a discriminator loss, and a cycle loss; and

reversely training the Cycle-GAN model according to the loss of the Cycle-GAN model.

14. The model training device according to claim 13 , wherein the converting a sample virtual image into an emulation image by using a Cycle-GAN model comprises:

converting the sample virtual image into the emulation image by using the first generator in the Cycle-GAN model.

15. The model training device according to claim 11 , wherein the extracting a hand image in the emulation image comprises:

inputting the emulation image into a gesture segmentation neural network, to output a hand confidence region, the gesture segmentation neural network being obtained by training according to images labeled with hand regions; and

extracting the hand image from the emulation image according to the hand confidence region.

16. The model training device according to claim 15 , wherein:

the gesture segmentation neural network comprises n convolution layers and n deconvolution layers, n being an integer greater than or equal to 2; and

the inputting the emulation image into a gesture segmentation neural network to output a hand confidence region comprises:

processing the emulation image through the n convolution layers, to obtain an intermediate feature map corresponding to the emulation image;

processing the intermediate feature map through the n deconvolution layers, to obtain a target feature map corresponding to the emulation image;

generating a probabilistic matrix according to the target feature map, the probabilistic matrix being used for representing a probability of each pixel point in the emulation image being the hand; and

performing binarization processing on the probabilistic matrix, to output a heatmap corresponding to the hand region, the heatmap being used for indicating the hand confidence region in the emulation image.

17. The model training device according to claim 16 , wherein the processing the intermediate feature map through the n deconvolution layers comprises:

splicing a feature map outputted by an m th deconvolution layer and a feature map outputted by an (n−m) th convolution layer, 1≤m≤n−1; and

processing a spliced feature map through an (m+1) th deconvolution layer.

18. The model training device according to claim 17 , wherein the plurality of two-dimensional residual layers comprises i two-dimensional residual layers, the plurality of three-dimensional residual layers comprises i three-dimensional residual layers, and first j two-dimensional residual layers being coupled to first j three-dimensional residual layers, 2≤j≤i−1, i and j being integers; and wherein the calculating a two-dimensional recognition loss and a three-dimensional recognition loss of the hand key-point recognition model according to the hand image and the key-point coordinates comprises:

inputting the hand image into the two-dimensional recognition branch and the three-dimensional recognition branch respectively;

adding a feature map outputted by a k th two-dimensional residual layer with a feature map outputted by a k th three-dimensional residual layer, 1≤k≤j−1;

inputting a feature map obtained after the adding into a (k+1) th two-dimensional residual layer and a (k+1) th three-dimensional residual layer;

processing a feature map outputted by the j th two-dimensional residual layer through (j+1) th to i th two-dimensional residual layers and the convolution layer, to obtain a two-dimensional recognition result;

processing a feature map outputted by the j th three-dimensional residual layer through (j+1) th to i th three-dimensional residual layers and the fully connected layer, to obtain a three-dimensional recognition result;

calculating the two-dimensional recognition loss according to the two-dimensional recognition result and the key-point coordinates; and

calculating the three-dimensional recognition loss according to the three-dimensional recognition result and the key-point coordinates.

19. The model training device according to claim 13 , further comprises constructing a loss function of the Cycle-GAN model according to the generator loss, the discriminator loss, and the cycle loss.

20. The model training device according to claim 19 , wherein in the loss function, a multiplication factor of the cycle loss is greater than a multiplication factor of the discriminator loss.