Granted Patent B2

US 12,423,910 · App. 18/061,752 · Granted Sep 23, 2025

Inventors: Roman Furko (Marina del Rey, CA); Vladyslav Horbatiuk (Kyiv, UA); Amir Iagudin (Moscow, RU); James Supancic, III (Irvine, CA)

Assignee: SNAP INC.

G06T15/205G06T7/246G06T19/006G06V20/20G06V20/46G06V40/107G06T2207/10016G06T2207/30196G06T2215/16

Abstract

Claims (62)

1. A computing system comprising:

one or more processors;

one or more cameras; and

a memory storing instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising:

capturing, using the one or more cameras, video frame tracking data of a wrist of a user;

generating 3D parameter data describing a 3D model of the wrist of the user based on the video frame tracking data;

correcting the 3D parameter data of the wrist of the user based on 2D loss data generated using a 2D projection of the 3D parameter data describing the 3D model of the wrist of the user onto an image of the wrist of the user;

generating 3D render data of a virtual item based on the 3D parameter data of the wrist of the user and 3D model data of a physical item represented by the virtual item;

generating Augmented Reality (AR) user interface video frame data based on the 3D render data and the video frame tracking data; and

providing an AR user interface based on the AR user interface video frame data.

2. The computing system of claim 1 , wherein the computing system further comprises a distance sensor, and wherein generating the 3D parameter data further comprises:

capturing, using one or more distance sensors of the computing system, distance data of the wrist of the user; and

generating the 3D parameter data based on the video frame tracking data and the distance data.

3. The computing system of claim 1 , wherein generating the 3D parameter data further comprises:

generating feature map data including 3D coordinate data of visual features of the wrist of the user based on the video frame tracking data; and

generating intermediate 3D parameter vector data based on the feature map data.

4. The computing system of claim 1 , wherein generating the 3D parameter data further comprises:

capturing, using one or more distance sensors of the computing system, distance data of the wrist of the user;

generating feature map data including 2D coordinate data of visual features of the wrist of the user based on the video frame tracking data; and

generating intermediate 3D parameter vector data based on the feature map data and the distance data.

5. The computing system of claim 1 , wherein generating the 3D parameter data further comprises:

generating the 3D parameter data of the wrist of the user based on the video frame tracking data and a 3D parameter model generated using a combination of synthetic video frame tracking data and actual video frame tracking data.

6. The computing system of claim 5 , wherein a loss function used during the generation of the 3D parameter model is applied to the actual video frame tracking data and not to the synthetic video frame tracking data.

7. A computer-implemented method comprising:

capturing, using one or more cameras of a computing system, video frame tracking data of a wrist of a user;

generating, by one or more processors, 3D parameter data describing a 3D model of the wrist of the user based on the video frame tracking data;

correcting, by the one or more processors, the 3D parameter data of the wrist of the user based on 2D loss data generated using a 2D projection of the 3D parameter data describing the 3D model of the wrist of the user onto an image of the wrist of the user;

generating, by the one or more processors, 3D render data of a virtual item based on the 3D parameter data of the wrist of the user and 3D model data of a physical item represented by the virtual item;

generating, by the one or more processors, AR user interface video frame data based on the 3D render data and the video frame tracking data; and

providing, by the one or more processors, using a display of a computing system, an AR user interface to the user based on the video frame AR user interface video frame data.

8. The computer-implemented method of claim 7 , wherein generating the 3D parameter data further comprises:

capturing, using one or more distance sensors of the computing system, distance data of the wrist of the user; and

generating the 3D parameter data based on the video frame tracking data and the distance data.

9. The computer-implemented method of claim 7 , wherein generating the 3D parameter data further comprises:

generating feature map data including 3D coordinate data of visual features of the wrist of the user based on the video frame tracking data; and

generating intermediate 3D parameter vector data based on the feature map data.

10. The computer-implemented method of claim 7 , wherein generating the 3D parameter data further comprises:

capturing, using one or more distance sensors of the computing system, distance data of the wrist of the user;

generating feature map data including 2D coordinate data of visual features of the wrist of the user based on the video frame tracking data; and

generating intermediate 3D parameter vector data based on the feature map data and the distance data.

11. The computer-implemented method of claim 7 , wherein generating the 3D parameter data further comprises:

generating the 3D parameter data of the wrist of the user based on the video frame tracking data and a 3D parameter model generated using a combination of synthetic video frame tracking data and actual video frame tracking data.

12. The computer-implemented method of claim 11 , wherein a loss function used during the generation of the 3D parameter model is applied to the actual video frame tracking data and not to the synthetic video frame tracking data.

13. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising:

capturing, using one or more cameras, video frame tracking data of a wrist of a user;

generating 3D parameter data describing a 3D model of the wrist of the user based on the video frame tracking data;

correcting the 3D parameter data of the wrist of the user based on 2D loss data generated using a 2D projection of the 3D parameter data describing the 3D model of the wrist of the user onto an image of the wrist of the user;

generating 3D render data of a virtual item based on the 3D parameter data of the wrist of the user and 3D model data of a physical item represented by the virtual item;

generating AR user interface video frame data based on the 3D render data and the video frame tracking data; and

providing, using a display of a computing system, an AR user interface to the user based on the AR user interface video frame data.

14. The computer-readable storage medium of claim 13 , wherein generating the 3D parameter data further comprises:

capturing, using one or more distance sensors of the computing system, distance data of the wrist of the user; and

generating the 3D parameter data based on the video frame tracking data and the distance data.

15. The computer-readable storage medium of claim 13 , wherein generating the 3D parameter data further comprises:

generating feature map data including 3D coordinate data of visual features of the wrist of the user based on the video frame tracking data; and

generating intermediate 3D parameter vector data based on the feature map data.

16. The computer-readable storage medium of claim 13 , wherein generating the 3D parameter data further comprises:

capturing, using one or more distance sensors of the computing system, distance data of the wrist of the user;

generating feature map data including 2D coordinate data of visual features of the wrist of the user based on the video frame tracking data; and

generating intermediate 3D parameter vector data based on the feature map data and the distance data.

17. The computer-readable storage medium of claim 13 , wherein generating the 3D parameter data further comprises:

generating the 3D parameter data of the wrist of the user based on the video frame tracking data and a 3D parameter model generated using a combination of synthetic video frame tracking data and actual video frame tracking data.

Assignments (1)

Continuity (1)

References Cited (89)