Granted Patent B2

US 12,620,173 · App. 18/234,737 · Granted May 5, 2026

Inventors: Laura Rosalia Luidolt (Vienna, AT); Kai Zhou (Wiener Neudorf, AT); Adrian Schoisengeier (Vienna, AT)

Assignee: Snap Inc.

G06T17/20G06T7/70G06T13/40G06T15/20G06T19/006G06V40/11G06T2207/20081G06T2207/30196

Abstract

Claims (81)

1 . An augmented reality (AR) device configured to generate hand annotations for an image depicting a hand of a user, the AR device comprising:

a display;

a processor;

two or more image sensors;

a memory storing instructions thereon, which, when executed by the processor, cause the AR device to perform operations comprising:

performing a calibration operation to generate a three-dimensional (3-D) data model of the hand of the user based on measurements relating to the hand of the user, the calibration operation comprising:

presenting, in AR via the display, an animated hand gesture;

prompting the user to position and move a hand to correspond with the animated hand gesture;

while the animated hand gesture is being presented, capturing one or more images of the hand of the user; and

using the one or more images as input to a pre-trained hand pose estimation model to generate the 3-D data model of the hand of the user;

generating, based on the 3-D model of the hand of the user, a 3-D virtual representation of a hand in a hand pose corresponding with a hand gesture different from the animated hand gesture;

presenting, in AR via the display, the 3-D virtual representation of the hand in the hand pose corresponding with the hand gesture;

during presentation of the 3-D virtual representation of the hand in the hand pose corresponding with the hand gesture in AR, detecting an input;

responsive to detecting the input, invoking at least one image sensor to capture an image of the hand of the user positioned to correspond with the 3-D virtual representation of the hand in the hand pose corresponding with the hand gesture; and

storing the captured image of the hand of the user with corresponding hand annotations based on the 3-D data model of the hand of the user.

2 . The AR device of claim 1 , where the pre-trained hand pose estimation model generates the 3-D data model of the hand as a skeleton-based hand mesh representation by computing a fixed number of mesh vertices to represent the hand of the user, based on 3-D location and rotation data derived from one or more captured images depicting the hand of the user.

3 . The AR device of claim 1 , wherein the 3-D virtual representation of the hand comprises:

a synthetic hand mesh;

a point cloud; or

a skeletal model.

4 . The AR device of claim 1 , wherein generating the 3-D virtual representation of the hand in the hand pose corresponding with the hand gesture, based on the 3-D data model of the hand of the user, further comprises:

mapping motion data of an animation of a hand performing a gesture to corresponding positional data for joints or bones, as represented by the 3-D data model of the hand of the user;

adjusting joint angles of the 3-D data model of the hand according to the motion data of the animation; and

rendering a plurality of two-dimensional (2-D) images of the 3-D virtual representation of the hand in the hand pose corresponding with the hand gesture, each 2-D image in the plurality of 2-D images representing a viewpoint of the 3-D virtual representation of the hand.

5 . The AR device of claim 1 , wherein presenting, in AR via the display, the 3-D virtual representation of the hand in the hand pose corresponding with the hand gesture further comprises;

presenting the 3-D virtual representation of the hand fixed in AR space, thereby enabling the user to move around in physical space to observe the 3-D virtual representation of the hand from different angles and different perspectives.

6 . The AR device of claim 1 , wherein the corresponding hand annotations based on the 3-D data model of the hand of the user are hand annotations representing 3-D positional data for various points of the 3-D data model of the hand of the user, wherein the 3-D positional data of each point has been adjusted to reflect the hand pose corresponding with the hand gesture as depicted by the 3-D virtual representation of the hand when the image of the hand of the user was captured.

7 . The AR device of claim 1 , wherein the memory is storing additional instructions thereon, which, when executed by the processor, cause the AR device to perform additional operations comprising:

prior to storing the captured image of the hand of the user with corresponding hand annotations based on the 3-D data model of the hand of the user, performing a mesh fitting operation to improve accuracy of 3-D positional data of various points of the 3-D data model of the hand, based on analysis of the image of the hand of the user.

8 . The AR device of claim 1 , wherein the detected input results from a triggering event comprising:

an audible command;

a hand gesture;

a press of a button located on the AR device; or

a press of a button located on a device communicatively coupled with the AR device.

9 . The AR device of claim 1 , wherein generating the 3-D virtual representation of a hand in the hand pose corresponding with the hand gesture comprises:

selecting motion data for a particular hand gesture from a library of animation files, each animation file in the library containing pre-captured motion data for a different hand gesture; and

using the selected motion data to generate the 3-D virtual representation of the hand in the hand pose corresponding with the particular hand gesture.

10 . A computer-implemented method comprising:

performing a calibration operation to generate a three-dimensional (3-D) data model of a hand of a user based on measurements relating to the hand of the user, the calibration operation comprising:

presenting in AR via a display, an animated hand gesture;

prompting the user to position and move a hand to correspond with the animated hand gesture;

while the animated hand gesture is being presented, capturing one or more images of the hand of the user; and

using the one or more images as input to a pre-trained hand pose estimation model to generate the 3-D data model of the hand of the user;

generating, based on the 3-D model of the hand of the user, a 3-D virtual representation of a hand in a hand pose corresponding with a hand gesture different from the animated hand gesture;

presenting, in AR via a display, the 3-D virtual representation of the hand in the hand pose corresponding with the hand gesture;

during presentation of the 3-D virtual representation of the hand in the hand pose corresponding with the hand gesture in AR, detecting an input;

responsive to detecting the input, invoking at least one image sensor to capture an image of the hand of the user positioned to correspond with the 3-D virtual representation of the hand in the hand pose corresponding with the hand gesture; and

storing the captured image of the hand of the user with corresponding hand annotations based on the 3-D data model of the hand of the user.

11 . The computer-implemented method of claim 10 , where the pre-trained hand pose estimation model generates the 3-D data model of the hand as a skeleton-based hand mesh representation by computing a fixed number of mesh vertices to represent the hand of the user, based on 3-D location and rotation data derived from one or more captured images depicting the hand of the user.

12 . The computer-implemented method of claim 10 , wherein the 3-D virtual representation of the hand comprises:

a synthetic hand mesh;

a point cloud; or

a skeletal model.

13 . The computer-implemented method of claim 10 , wherein generating the 3-D virtual representation of the hand in the hand pose corresponding with the hand gesture, based on the 3-D data model of the hand of the user, further comprises:

mapping motion data of an animation of a hand performing a gesture to corresponding positional data for joints or bones, as represented by the 3-D data model of the hand of the user;

adjusting joint angles of the 3-D data model of the hand according to the motion data of the animation; and

rendering a plurality of two-dimensional (2-D) images of the 3-D virtual representation of the hand in the hand pose corresponding with the hand gesture, each 2-D image in the plurality of 2-D images representing a viewpoint of the 3-D virtual representation of the hand.

14 . The computer-implemented method of claim 10 , wherein presenting, in AR via the display, the 3-D virtual representation of the hand in the hand pose corresponding with the hand gesture further comprises;

presenting the 3-D virtual representation of the hand fixed in AR space, thereby enabling the user to move around in physical space to observe the 3-D virtual representation of the hand from different angles and different perspectives.

15 . The computer-implemented method of claim 10 , wherein the corresponding hand annotations based on the 3-D data model of the hand of the user are hand annotations representing 3-D positional data for various points of the 3-D data model of the hand of the user, wherein the 3-D positional data of each point has been adjusted to reflect the hand pose corresponding with the hand gesture as depicted by the 3-D virtual representation of the hand when the image of the hand of the user was captured.

16 . The computer-implemented method of claim 10 , further comprising:

prior to storing the captured image of the hand of the user with corresponding hand annotations based on the 3-D data model of the hand of the user, performing a mesh fitting operation to improve accuracy of 3-D positional data of various points of the 3-D data model of the hand, based on analysis of the image of the hand of the user.

17 . The computer-implemented method of claim 10 , wherein the detected input results from a triggering event comprising:

an audible command;

a hand gesture;

a press of a button located on an AR device; or

a press of a button located on a device communicatively coupled with the AR device.

18 . The computer-implemented method of claim 10 , wherein generating the 3-D virtual representation of a hand in the hand pose corresponding with the hand gesture comprises:

selecting motion data for a particular hand gesture from a library of animation files, each animation file in the library containing pre-captured motion data for a different hand gesture; and

using the selected motion data to generate the 3-D virtual representation of the hand in the hand pose corresponding with the particular hand gesture.

19 . An AR device configured to generate hand annotations for an image depicting a hand of a user, the AR device comprising:

means for performing a calibration operation to generate a 3-D data model of the hand of the user based on measurements relating to the hand of the user, the calibration operation comprising:

presenting, in AR via a display, an animated hand gesture;

prompting the user to position and move a hand to correspond with the animated hand gesture;

while the animated hand gesture is being presented, capturing one or more images of the hand of the user; and

using the one or more images as input to a pre-trained hand pose estimation model to generate the 3-D data model of the hand of the user;

means for generating, based on the 3-D model of the hand of the user, a 3-D virtual representation of a hand in a hand pose corresponding with a hand gesture different from the animated hand gesture;

means for presenting, in AR, the 3-D virtual representation of the hand in the hand pose corresponding with the hand gesture;

during presentation of the 3-D virtual representation of the hand in the hand pose corresponding with the hand gesture in AR, means for detecting an input;

responsive to detecting the input, means for invoking at least one image sensor to capture an image of the hand of the user positioned to correspond with the 3-D virtual representation of the hand in the hand pose corresponding with the hand gesture; and

means for storing the captured image of the hand of the user with corresponding hand annotations based on the 3-D data model of the hand of the user.

Assignments (1)

Continuity (1)

References Cited (10)