HAND GESTURE INPUT FOR WEARABLE SYSTEM
Techniques are disclosed for allowing a user's hands to interact with virtual objects. An image of at least one hand may be received from an image capture devices. A plurality of keypoints associated with at least one hand may be detected. In response to determining that a hand is making or is transitioning into making a particular gesture, a subset of the plurality of keypoints may be selected. An interaction point may be registered to a particular location relative to the subset of the plurality of keypoints based on the particular gesture. A proximal point may be registered to a location along the user's body. A ray may be cast from the proximal point through the interaction point. A multi-DOF controller for interacting with the virtual object may be formed based on the ray.
1 . A method of interacting with a virtual object, the method comprising:
receiving one or more images of a first hand and a second hand of a user;
analyzing the one or more images to detect a plurality of keypoints associated with each of the first hand and the second hand;
determining an interaction point for each of the first hand and the second hand based on the plurality of keypoints associated with each of the first hand and the second hand;
generating one or more bimanual deltas based on the interaction point for each of the first hand and the second hand; and
interacting with the virtual object using the one or more bimanual deltas.
2 . The method of claim 1 , further comprising:
determining a bimanual interaction point based on the interaction point for each of the first hand and the second hand.
3 . The method of claim 1 , wherein the interaction point for the first hand is determined based on the plurality of keypoints associated with the first hand, and the interaction point for the second hand is determined based on the plurality of keypoints associated with the second hand.
4 . The method of claim 1 , wherein determining the interaction point for each of the first hand and the second hand includes:
determining, based on analyzing the one or more images, whether the first hand is making or is transitioning into making a first particular gesture from a plurality of gestures; and
in response to determining that the first hand is making or is transitioning into making the first particular gesture:
selecting a subset of the plurality of keypoints associated with the first hand that correspond to the first particular gesture;
determining a first particular location relative to the subset of the plurality of keypoints associated with the first hand, wherein the first particular location is determined based on the subset of the plurality of keypoints associated with the first hand and the first particular gesture; and
registering the interaction point for the first hand to the first particular location.
5 . The method of claim 4 , wherein determining the interaction point for each of the first hand and the second hand further includes:
determining, based on analyzing the one or more images, whether the second hand is making or is transitioning into making a second particular gesture from the plurality of gestures; and
in response to determining that the second hand is making or is transitioning into making the second particular gesture:
selecting a subset of the plurality of keypoints associated with the second hand that correspond to the second particular gesture;
determining a second particular location relative to the subset of the plurality of keypoints associated with the second hand, wherein the second particular location is determined based on the subset of the plurality of keypoints associated with the second hand and the second particular gesture; and
registering the interaction point for the second hand to the second particular location.
6 . The method of claim 5 , wherein the plurality of gestures includes at least one of a grasping gesture, a pointing gesture, or a pinching gesture.
7 . The method of claim 1 , wherein the one or more images include:
a first image of the first hand and a second image of the second hand; or
a single image of the first hand and the second hand.
8 . The method of claim 1 , wherein the one or more images include a series of time-sequenced imaged.
9 . The method of claim 1 , wherein the one or more bimanual deltas are determined based on a frame-to-frame movement of the interaction point for each of the first hand and the second hand.
10 . The method of claim 9 , wherein the one or more bimanual deltas include:
a translation delta corresponding to a frame-to-frame translational movement of the interaction point for each of the first hand and the second hand;
a rotation delta corresponding to a frame-to-frame rotational movement of the interaction point for each of the first hand and the second hand; or
a sliding delta corresponding to a frame-to-frame separation movement of the interaction point for each of the first hand and the second hand.
11 . A system comprising:
one or more processors; and
a machine-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for interacting with a virtual object, the operations comprising:
receiving one or more images of a first hand and a second hand of a user;
analyzing the one or more images to detect a plurality of keypoints associated with each of the first hand and the second hand;
determining an interaction point for each of the first hand and the second hand based on the plurality of keypoints associated with each of the first hand and the second hand;
generating one or more bimanual deltas based on the interaction point for each of the first hand and the second hand; and
interacting with the virtual object using the one or more bimanual deltas.
12 . The system of claim 11 , wherein the operations further comprise:
determining a bimanual interaction point based on the interaction point for each of the first hand and the second hand.
13 . The system of claim 11 , wherein the interaction point for the first hand is determined based on the plurality of keypoints associated with the first hand, and the interaction point for the second hand is determined based on the plurality of keypoints associated with the second hand.
14 . The system of claim 11 , wherein determining the interaction point for each of the first hand and the second hand includes:
determining, based on analyzing the one or more images, whether the first hand is making or is transitioning into making a first particular gesture from a plurality of gestures; and
in response to determining that the first hand is making or is transitioning into making the first particular gesture:
selecting a subset of the plurality of keypoints associated with the first hand that correspond to the first particular gesture;
determining a first particular location relative to the subset of the plurality of keypoints associated with the first hand, wherein the first particular location is determined based on the subset of the plurality of keypoints associated with the first hand and the first particular gesture; and
registering the interaction point for the first hand to the first particular location.
15 . The system of claim 14 , wherein determining the interaction point for each of the first hand and the second hand further includes:
determining, based on analyzing the one or more images, whether the second hand is making or is transitioning into making a second particular gesture from the plurality of gestures; and
in response to determining that the second hand is making or is transitioning into making the second particular gesture:
selecting a subset of the plurality of keypoints associated with the second hand that correspond to the second particular gesture;
determining a second particular location relative to the subset of the plurality of keypoints associated with the second hand, wherein the second particular location is determined based on the subset of the plurality of keypoints associated with the second hand and the second particular gesture; and
registering the interaction point for the second hand to the second particular location.
16 . A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations for interacting with a virtual object, the operations comprising:
receiving one or more images of a first hand and a second hand of a user;
analyzing the one or more images to detect a plurality of keypoints associated with each of the first hand and the second hand;
determining an interaction point for each of the first hand and the second hand based on the plurality of keypoints associated with each of the first hand and the second hand;
generating one or more bimanual deltas based on the interaction point for each of the first hand and the second hand; and
interacting with the virtual object using the one or more bimanual deltas.
17 . The non-transitory machine-readable medium of claim 16 , wherein the operations further comprise:
determining a bimanual interaction point based on the interaction point for each of the first hand and the second hand.
18 . The non-transitory machine-readable medium of claim 16 , wherein the interaction point for the first hand is determined based on the plurality of keypoints associated with the first hand, and the interaction point for the second hand is determined based on the plurality of keypoints associated with the second hand.
19 . The non-transitory machine-readable medium of claim 16 , wherein determining the interaction point for each of the first hand and the second hand includes:
determining, based on analyzing the one or more images, whether the first hand is making or is transitioning into making a first particular gesture from a plurality of gestures; and
in response to determining that the first hand is making or is transitioning into making the first particular gesture:
selecting a subset of the plurality of keypoints associated with the first hand that correspond to the first particular gesture;
determining a first particular location relative to the subset of the plurality of keypoints associated with the first hand, wherein the first particular location is determined based on the subset of the plurality of keypoints associated with the first hand and the first particular gesture; and
registering the interaction point for the first hand to the first particular location.
20 . The non-transitory machine-readable medium of claim 19 , wherein determining the interaction point for each of the first hand and the second hand further includes:
determining, based on analyzing the one or more images, whether the second hand is making or is transitioning into making a second particular gesture from the plurality of gestures; and
in response to determining that the second hand is making or is transitioning into making the second particular gesture:
selecting a subset of the plurality of keypoints associated with the second hand that correspond to the second particular gesture;
determining a second particular location relative to the subset of the plurality of keypoints associated with the second hand, wherein the second particular location is determined based on the subset of the plurality of keypoints associated with the second hand and the second particular gesture; and
registering the interaction point for the second hand to the second particular location.