Image based finger tracking plus controller tracking

1. An apparatus comprising:

at least one processor programmed with instructions to:

identify an image from at least one camera of a hand grasping a computer game controller;

crop the image to a region containing the controller and the hand; and

present on a computerized display a virtual representation of the hand generated at least in part based on image analysis of the region and at least one touch signal from the controller, wherein the instructions are executable by the processor to:

execute a machine learning (ML) module to generate the virtual representation by modifying keypoints of a template image based on the image in the region and the touch signal from the controller.

2. The apparatus of claim 1 , wherein the at least one camera is mounted on a head-mounted display (HMD).

3. The apparatus of claim 2 , comprising the HMD.

4. The apparatus of claim 1 , wherein the touch signal is from a control key element of the controller.

5. The apparatus of claim 1 , wherein the touch signal is from a sensor on the controller other than a control key element of the controller.

6. The apparatus of claim 1 , wherein the instructions are executable by the processor to generate, using the touch signal, a virtual representation of a portion of the hand blocked from the camera by the controller.

7. The apparatus of claim 1 , wherein the instructions are executable by the processor to generate the virtual representation without using hand recognition and using controller recognition responsive to identifying a touch signal from the controller.

8. The apparatus of claim 1 , wherein the ML model comprises at least one neural network (NN) and at least one heatmap.

9. A method, comprising:

identifying an image of a hand grasping a computer simulation controller;

receiving from the computer simulation controller at least one touch signal; and

based on both the image of the hand and the touch signal, generating an image of a virtual hand for display thereof.

10. The method of claim 9 , comprising presenting the image of the virtual hand on a computerized display.

11. The method of claim 10 , wherein the computerized display comprises a head-mounted display (HMD).

12. The method of claim 9 , comprising cropping the image of the hand grasping the computer simulation controller to a region containing the controller and the hand, and generating the image of the virtual hand using only the region and no other parts of the image of the hand grasping the computer simulation controller.

13. The method of claim 9 , wherein the touch signal is from a control key element of the controller.

14. The method of claim 9 , wherein the touch signal is from a sensor on the controller other than a control key element of the controller.

15. The method of claim 9 , comprising generating using the touch signal, a virtual representation of a portion of the hand blocked from a camera view by the controller.

16. The method of claim 9 , comprising generating the image of the virtual hand without using hand recognition and using controller recognition responsive to identifying a touch signal from the controller.

17. The method of claim 9 , comprising

executing a machine learning (ML) module to generate the virtual representation by modifying keypoints of a template image based on the image in the region and the touch signal from the controller.

18. The method of claim 17 , wherein the ML model comprises at least one key neural network (NN) and at least one heatmap.

19. A device comprising:

at least one computer storage that is not a transitory signal and that comprises instructions executable by at least one processor to:

receive at least one image of a human hand grasping a computer game controller;

receive at least one touch signal from the controller; and

generate, based on both the image and the touch signal, a virtual hand representing the human hand for presentation of the virtual hand on at least one computerized display.