3D hand pose estimation based on depth-image guided adversarial network 1

1. A method of estimating three-dimensional (3D) hand poses in images, comprising:

receiving, by a computer, data corresponding to a hand image;

generating, by the computer, a first depth map corresponding to the received hand image data;

generating, by the computer, one or more hand joint heat maps based on the received hand image data;

estimating, by the computer, joint locations based on the generated one or more hand joint heat maps;

reconstructing, by the computer, a second depth map based on the estimated joint locations and the first depth map; and

estimating, by the computer, a hand pose from the received hand image data and the first and second depth maps.

2. The method of claim 1 , wherein the generating the first depth map is performed by a generative adversarial network.

3. The method of claim 2 , wherein the generative adversarial network comprises a generator and a discriminator.

4. The method of claim 3 , wherein the generative adversarial network utilizes a generative adversarial network (GAN) loss value and a task loss value as training values.

5. The method of claim 4 , wherein the generative adversarial network is trained by the generator minimizing the GAN loss value and the task loss value.

6. The method of claim 4 , wherein the generative adversarial network is trained by the discriminator maximizing the GAN loss value and the task loss value.

7. The method of claim 4 , wherein the generator generates the first depth map using solely the received hand image data.

8. The method of claim 7 , wherein the discriminator determines the GAN loss value using the first depth map and one or more unpaired depth maps.

9. The method of claim 1 , wherein the hand pose is estimated by using only the received hand image data and the first and second depth maps corresponding to the received hand image data.

10. A computer system for estimating three-dimensional (3D) hand poses in images, the computer system comprising:

one or more computer-readable non-transitory storage media configured to store computer program code; and

one or more computer processors configured to access said computer program code and operate as instructed by said computer program code, said computer program code including:

receiving code configured to cause the one or more computer processors to receive a data corresponding to a hand image;

first generating code configured to cause the one or more computer processors to generate a first depth map corresponding to the received hand image data;

second generating code configured to cause the one or more computer processors to generate one or more hand joint heat maps based on the received hand image data;

first estimating code configured to cause the one or more computer processors to estimate joint locations based on the generated one or more hand joint heat maps; and

reconstructing code configured to cause the one or more computer processors to reconstruct a second depth map based on the estimated joint locations and the first depth map; and

second estimating code configured to cause the one or more computer processors to estimate a hand pose from the received hand image data and the first and second depth maps.

11. The computer system of claim 10 , wherein the generating is performed by a generative adversarial network.

12. The computer system of claim 11 , wherein the generative adversarial network comprises a generator and a discriminator.

13. The computer system of claim 12 , wherein the generative adversarial network utilizes a generative adversarial network (GAN) loss value and a task loss value as training values.

14. The computer system of claim 13 , wherein the generative adversarial network is trained by the generator minimizing the GAN loss value and the task loss value.

15. The computer system of claim 13 , wherein the generative adversarial network is trained by the discriminator maximizing the GAN loss value and the task loss value.

16. The computer system of claim 13 , wherein the generator generates the first depth map using solely the received hand image data.

17. The computer system of claim 16 , wherein the discriminator determines the GAN loss value using the first depth map and one or more unpaired depth maps.

18. A non-transitory computer readable medium having stored thereon a computer program for estimating three-dimensional (3D) hand poses in images, the computer program configured to cause one or more computer processors to:

receive a hand image data;

generate a first depth map corresponding to the received hand image data;

generate one or more hand joint heat maps based on the received hand image data;

estimate joint locations based on the one or more hand joint heat maps;

reconstruct a second depth map based on the estimated joint locations and the first depth map; and

estimate a hand pose from the received hand image data and the first and second depth maps.

19. The computer system of claim 13 , wherein the hand pose is estimated by using only the received hand image data and the first and second depth maps corresponding to the received hand image data.

20. The non-transitory computer readable medium of claim 18 , wherein the first depth map is generated by a generative adversarial network.