Wrist rotation manipulation of virtual objects
An Augmented Reality (AR) system is provided. The AR system uses a combination of gesture and DMVO methodologies to provide for the user's selection and modification of virtual object of an AR experience. The user indicates that they want to interact with a virtual object of the AR experience by moving their hand to overlap the virtual object. While keeping their hand in an overlapping position, the user rotates their wrist and the virtual object is rotated as well. To end the interaction, the user moves their hand such that their hand is no longer overlapping the virtual object.
1 . A computer-implemented method comprising:
displaying a virtual object in an Augmented Reality (AR) experience provided to a user by an AR system, the virtual object including first 3D coordinate data of a location of the virtual object in a real-world scene;
determining a user wants to interact with the virtual object by performing Direct Manipulation of Virtual Object (DMVO) operations comprising:
determining, using one or more cameras of the AR system, second 3D coordinate data of one or more nodes of a skeletal model of a hand of a user; and
determining an overlap between the hand of the user and the virtual object from a perspective of the user using the first 3D coordinate data and the second 3D coordinate data;
in response to determining the overlap, rotating the virtual object by performing gesture recognition operations comprising:
determining a current rotation value of a wrist joint of the hand of the user based on recognizing a gesture of rotation using the one more cameras, the gesture of rotation not involving DMVO operations on the virtual object; and
generating an updated rotation value for the virtual object based on the current rotation value of the wrist joint, a default rotation value of the virtual object, and a difference between the current rotation value of the wrist joint and a default rotation value for the wrist joint; and
redisplaying the virtual object based on the updated rotation value.
2 . The computer-implemented method of claim 1 , further comprising:
determining that the hand of the user no longer overlaps the virtual object; and
based on determining that the hand of the user no longer overlaps the virtual object, ending interaction of the user with the virtual object.
3 . The computer-implemented method of claim 1 , wherein operations of determining an overlap between the hand of the user and the virtual object further comprise:
capturing, using the one or more cameras, hand-tracking video frame data of the hand of the user;
generating hand-tracking data based on the hand-tracking video frame data; and
determining the overlap between the hand of the user and the virtual object using the hand-tracking data.
4 . The computer-implemented method of claim 1 , wherein operations of determining an overlap between the hand of the user and the virtual object further comprise:
capturing, using the one or more cameras, hand-tracking video frame data of the hand of the user; and
determining the overlap between the hand of the user and the virtual object based on the hand-tracking video frame data using one or more computer vision methodologies.
5 . The computer-implemented method of claim 1 , wherein operations of determining the current rotation value of the wrist joint of the hand of the user further comprise:
capturing, using the one or more cameras, hand-tracking video frame data of the hand of the user; and
determining the current rotation value of the wrist joint of the hand of the user based on the hand-tracking video frame data and a wrist rotation model generated using machine learning methodologies.
6 . The computer-implemented method of claim 1 , wherein the AR system comprises a head-worn apparatus.
7 . A computing apparatus comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the computing apparatus to perform operations comprising:
displaying a virtual object in an AR experience provided to a user by an AR system, the virtual object including first 3D coordinate data of a location of the virtual object in a real-world scene;
determining a user wants to interact with the virtual object by performing Direct Manipulation of Virtual Object (DMVO) operations comprising:
determining, using one or more cameras of the AR system, second 3D coordinate data of one or more nodes of a skeletal model of a hand of a user; and
determining an overlap between the hand of the user and the virtual object from a perspective of the user using the first 3D coordinate data and the second 3D coordinate data;
in response to determining the overlap, rotating the virtual object by performing gesture recognition operations comprising:
determining a current rotation value of a wrist joint of the hand of the user based on recognizing a gesture of rotation using the one more cameras, the gesture of rotation not involving DMVO operations on the virtual object; and
generating an updated rotation value for the virtual object based on the current rotation value of the wrist joint, a default rotation value of the virtual object, and a difference between the current rotation value of the wrist joint and a default rotation value for the wrist joint; and
redisplaying the virtual object based on the updated rotation value.
8 . The computing apparatus of claim 7 , wherein the operations further comprise:
determining that the hand of the user no longer overlaps the virtual object; and
based on determining that the hand of the user no longer overlaps the virtual object, ending interaction of the user with the virtual object.
9 . The computing apparatus of claim 7 , wherein operations of determining an overlap between the hand of the user and the virtual object further comprise:
capturing, using the one or more cameras, hand-tracking video frame data of the hand of the user;
generating hand-tracking data based on the hand-tracking video frame data; and
determining the overlap between the hand of the user and the virtual object using the hand-tracking data.
10 . The computing apparatus of claim 7 , wherein operations of determining an overlap between the hand of the user and the virtual object further comprise:
capturing, using the one or more cameras, hand-tracking video frame data of the hand of the user; and
determining the overlap between the hand of the user and the virtual object based on the hand-tracking video frame data using one or more computer vision methodologies.
11 . The computing apparatus of claim 7 , wherein operations of determining the current rotation value of the wrist joint of the hand of the user further comprise:
capturing, using the one or more cameras, hand-tracking video frame data of the hand of the user; and
determining the current rotation value of the wrist joint of the hand of the user based on the hand-tracking video frame data and a wrist rotation model generated using machine learning methodologies.
12 . The computing apparatus of claim 7 , wherein the AR system comprises a head-worn apparatus.
13 . A non-transitory computer-readable storage medium including instructions that, when executed by a computer, cause the computer to perform operations comprising:
displaying a virtual object in an AR experience provided to a user by an AR system, the virtual object including first 3D coordinate data of a location of the virtual object in a real-world scene;
determining a user wants to interact with the virtual object by performing Direct Manipulation of Virtual Object (DMVO) operations comprising:
determining, using one or more cameras of the AR system, second 3D coordinate data of one or more nodes of a skeletal model of a hand of a user; and
determining an overlap between the hand of the user and the virtual object from a perspective of the user using the first 3D coordinate data and the second 3D coordinate data;
in response to determining the overlap, rotating the virtual object by performing gesture recognition operations comprising:
determining a current rotation value of a wrist joint of the hand of the user based on recognizing a gesture of rotation using the one more cameras, the gesture of rotation not involving DMVO operations on the virtual object; and
generating an updated rotation value for the virtual object based on the current rotation value of the wrist joint, a default rotation value of the virtual object, and a difference between the current rotation value of the wrist joint and a default rotation value for the wrist joint; and
redisplaying the virtual object based on the updated rotation value.
14 . The non-transitory computer-readable storage medium of claim 13 , wherein the operations further comprise:
determining that the hand of the user no longer overlaps the virtual object; and
based on determining that the hand of the user no longer overlaps the virtual object, ending interaction of the user with the virtual object.
15 . The non-transitory computer-readable storage medium of claim 13 , wherein operations of determining an overlap between the hand of the user and the virtual object further comprise:
capturing, using the one or more cameras, hand-tracking video frame data of the hand of the user;
generating hand-tracking data based on the hand-tracking video frame data; and
determining the overlap between the hand of the user and the virtual object using the hand-tracking data.
16 . The non-transitory computer-readable storage medium of claim 13 , wherein operations of determining an overlap between the hand of the user and the virtual object further comprise:
capturing, using the one or more cameras, hand-tracking video frame data of the hand of the user; and
determining the overlap between the hand of the user and the virtual object based on the hand-tracking video frame data using one or more computer vision methodologies.
17 . The non-transitory computer-readable storage medium of claim 13 , wherein the AR system comprises a head-worn apparatus.