Granted Patent B2

US 12,288,298 · App. 17/845,626 · Granted Apr 29, 2025

Inventor: Sharon Moll (Lachen, CH)

Assignee: Snap Inc.

G06T19/006G06F3/011G06F3/04815G06F3/0482G06T3/40G06T7/20G06T7/70G06T2200/24

Abstract

Claims (86)

1. A computing device comprising:

a camera;

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the computing device to perform operations comprising:

capturing, by the camera, one or more images within a field of view of the camera;

activating an augmented reality (AR) content item that is executed within a client application;

in response to activating the augmented reality content item, providing a continuous stream of camera data that includes the one or more images to a camera service of a plurality of AR graphics services using one or more first application programming interface (API) calls;

generating, by the augmented reality content item, one or more second API calls for an object detection service to implement one or more object detection models to detect, in the camera data, one or more objects of interest;

obtaining, by the augmented reality content item and from the object detection service, first AR graphics display data indicating (i) an AR graphics display surface labeled as a first object of interest within the field of view, (ii) a graphics input tool labeled as a second object of interest, and (iii) a hand grasping the graphics input tool labeled as a third object of interest, wherein the AR graphics display surface includes a surface of one or more physical objects included in an environment in which the computing device is located;

generating, by the augmented reality content item, one or more third API calls for an object tracking service to implement one or more object tracking models to track, in the camera data, first motion of the graphics input tool and second motion of the hand grasping the graphics input tool;

obtaining, by the augmented reality content item and from the object tracking service, second AR graphics display data indicating a first path of motion of the graphics input tool based on differences in locations of the graphics input tool across a number of video frames;

obtaining, by the augmented reality content item and from the object tracking service, third AR graphics display data indicating a second path of motion of the hand grasping the graphics input tool based on differences in locations of the hand across the number of video frames;

obtaining, by the augmented reality content item and from the object tracking service, fourth AR graphics display data indicating an angle at which the graphics input tool is being held based on first images of the graphics input tool and second images of the hand grasping the graphics input tool within the number of video frames; and

causing, by the augmented reality content item and based on AR graphics display data obtained from the plurality of AR graphics services, display of a user interface overlaid on the AR graphics display surface, the user interface including an AR graphic having at least one of a shape or one or more contours that (i) follow the first path of motion of the graphics input tool and the second path of motion of the hand grasping the graphics input tool and (ii) have a width that is based on the angle at which the graphics input tool is being held.

2. The computing device of claim 1 , wherein the computing device is a head-worn device that includes at least one display device and the user interface is displayed by the at least one display device.

3. The computing device of claim 1 , wherein the memory stores additional instructions that, when executed by the one or more processors, cause the computing device to perform additional operations comprising:

in response to determining that the graphics input tool is within a threshold distance of the AR graphics display surface, activating one or more features of the augmented reality content item to at least one of create or modify the AR graphic;

determining that the graphics input tool is outside of the threshold distance of the AR graphics display surface; and

in response to determining that the graphics input tool is outside of the threshold distance of the AR graphics display surface, deactivating the one or more features of the augmented reality content item.

4. The computing device of claim 1 , wherein the memory stores additional instructions that, when executed by the one or more processors, causes the computing device to perform additional operations comprising:

causing an AR graphics menu to be displayed within the user interface, the AR graphics menu including a number of user interface elements, individual user interface elements of the number of user interface elements being selectable to activate one or more features of an augmented reality content item to at least one of create or modify at least a portion of the AR graphic.

5. The computing device of claim 4 , wherein the memory stores additional instructions that, when executed by the one or more processors, causes the computing device to perform additional operations comprising:

determining that the graphics input tool is within an additional threshold distance of a user interface element of the number of user interface elements of the AR graphics menu; and

causing a feature of the user interface element to be activated.

6. The computing device of claim 1 , wherein the memory stores additional instructions that, when executed by the one or more processors, causes the computing device to perform additional operations comprising:

obtaining, based on the camera data, coordinates in real world space for the AR graphics display surface; and

determining, based on the camera data, a distance from the camera to the AR graphics display surface.

7. The computing device of claim 6 , wherein the memory stores additional instructions that, when executed by the one or more processors, causes the computing device to perform additional operations comprising:

determining that the distance from the camera to the AR graphics display surface has changed from a first distance to a second distance; and

modifying a level of magnification of the AR graphic within the user interface based on the distance from the camera to the AR graphics display surface changing from the first distance to the second distance.

8. A computer-implemented method comprising:

capturing, by one or more cameras of one or more computing devices including one or more processors and memory, one or more images within a field of view of the one or more cameras;

activating, by at least one computing device of the one or more computing devices, an augmented reality (AR) content item that is executed within a client application;

in response to activating the augmented reality content item, providing, by at least one computing device of the one or more computing devices, a continuous stream of camera data that includes the one or more images to a camera service of a plurality of AR graphics services using one or more first application programming interface (API) calls;

generating, by the augmented reality content item, one or more second API calls for an object detection service to implement one or more object detection models to detect, in the camera data, one or more objects of interest;

obtaining, by the augmented reality content item and from the object detection service, first AR graphics display data indicating (i) an AR graphics display surface within the field of view, (ii) a graphics input tool labeled as a first object of interest, and (iii) a hand grasping the graphics input tool labeled as a second object of interest, wherein the AR graphics display surface includes a surface of one or more physical objects included in an environment in which the at least one computing device is located;

generating, by the augmented reality content item, one or more third API calls for an object tracking service to implement one or more object tracking models to track, in the camera data, first motion of the graphics input tool and second motion of the hand grasping the graphics input tool;

obtaining, by the augmented reality content item and from the object tracking service, second AR graphics display data indicating a first path of motion of the graphics input tool based on differences in locations of the graphics input tool across a number of video frames;

obtaining, by the augmented reality content item and from the object tracking service, third AR graphics display data indicating a second path of motion of the hand grasping the graphics input tool based on differences in locations of the hand across the number of video frames;

obtaining, by the augmented reality content item and from the object tracking service, fourth AR graphics display data indicating an angle at which the graphics input tool is being held based on first images of the graphics input tool and second images of the hand grasping the graphics input tool within the number of video frames; and

causing, by the augmented reality content item and based on AR graphics display data obtained from the plurality of AR graphics services, by at least one computing device of the one or more computing devices, display of a user interface overlaid on the AR graphics display surface, the user interface including an AR graphic having at least one of a shape or one or more contours that (i) follow the first path of motion of the graphics input tool and the second path of motion of the hand grasping the graphics input tool and (ii) have a width that is based on the angle at which the graphics input tool is being held.

9. The computer-implemented method of claim 8 , comprising:

determining, by at least one computing device of the one or more computing devices and at a first time, that the AR graphics display surface is not detected within the field of view of the one or more cameras;

responsive to determining that the AR graphics display surface is not detected within the field of view of the one or more cameras at the first time, removing, by at least one computing device of the one or more computing devices, the AR graphic from the user interface;

determining, by at least one computing device of the one or more computing devices and at a second time, that the AR graphics display surface is within the field of view of the one or more cameras; and

responsive to determining that the AR graphics display surface is within the field of view of the one or more cameras at the second time, causing, by at least one computing device of the one or more computing devices, the AR graphic to be re-displayed in the user interface.

10. The computer-implemented method of claim 8 , comprising:

in response to determining that the graphics input tool is within a threshold distance of the AR graphics display surface, activating, by at least one computing device of the one or more computing devices, one or more features of the augmented reality content item to at least one of create or modify the AR graphic;

determining, by at least one computing device of the one or more computing devices, that the graphics input tool is outside of the threshold distance of the AR graphics display surface; and

in response to determining that the graphics input tool is outside of the threshold distance of the AR graphics display surface, deactivating, by at least one computing device of the one or more computing devices, the one or more features of the augmented reality content item.

11. The computer-implemented method of claim 8 , comprising:

causing, by at least one computing device of the one or more computing devices, an AR graphics menu to be displayed within the user interface, the AR graphics menu including a number of user interface elements, individual user interface elements of the number of user interface elements being selectable to activate one or more features of an augmented reality content item to at least one of create or modify at least a portion of the AR graphic.

12. The computer-implemented method of claim 11 , comprising:

determining, by at least one computing device of the one or more computing devices, that the graphics input tool is within an additional threshold distance of a user interface element of the number of user interface elements of the AR graphics menu; and

causing, by at least one computing device of the one or more computing devices, a feature of the user interface element to be activated.

13. The computer-implemented method of claim 8 , comprising:

obtaining, by at least one computing device of the one or more computing devices and based on the camera data, coordinates in real world space for the AR graphics display surface; and

determining, by at least one computing device of the one or more computing devices and based on the camera data, a distance from the one or more cameras to the AR graphics display surface.

14. The computer-implemented method of claim 13 , comprising:

determining, by at least one computing device of the one or more computing devices, that the distance from the one or more cameras to the AR graphics display surface has changed from a first distance to a second distance; and

modifying, by at least one computing device of the one or more computing devices, a level of magnification of the AR graphic within the user interface based on the distance from the one or more cameras to the AR graphics display surface changing from the first distance to the second distance.

15. One or more computer-readable storage media storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

capturing, by one or more cameras of one or more computing devices, one or more images within a field of view of the one or more cameras;

activating an augmented reality (AR) content item that is executed within a client application;

in response to activating the augmented reality content item, providing a continuous stream of camera data that includes the one or more images to a camera service of a plurality of AR graphics services using one or more first application programming interface (API) calls;

generating, by the augmented reality content item, one or more second API calls for an object detection service to implement one or more object detection models to detect, in the camera data, one or more objects of interest;

obtaining, by the augmented reality content item and from the object detection service, first AR graphics display data indicating (i) an AR graphics display surface within the field of view, (ii) a graphics input tool labeled as a first object of interest, and (iii) a hand grasping the graphics input tool labeled as a second object of interest, wherein the AR graphics display surface includes a surface of one or more physical objects included in an environment in which at least one computing device of the one or more computing devices is located;

generating, by the augmented reality content item, one or more third API calls for an object tracking service to implement one or more object tracking models to track, in the camera data, first motion of the graphics input tool and second motion of the hand grasping the graphics input tool;

obtaining, by the augmented reality content item and from the object tracking service, second AR graphics display data indicating a first path of motion of the graphics input tool based on differences in locations of the graphics input tool across a number of video frames;

obtaining, by the augmented reality content item and from the object tracking service, third AR graphics display data indicating a second path of motion of the hand grasping the graphics input tool based on differences in locations of the hand across the number of video frames;

obtaining, by the augmented reality content item and from the object tracking service, fourth AR graphics display data indicating an angle at which the graphics input tool is being held based on first images of the graphics input tool and second images of the hand grasping the graphics input tool within the number of video frames; and

causing, by the augmented reality content item and based on AR graphics display data obtained from the plurality of AR graphics services, display of a user interface overlaid on the AR graphics display surface, the user interface including an AR graphic having at least one of a shape or one or more contours that follow (i) the first path of motion of the graphics input tool and the second path of motion of the hand grasping the graphics input tool and (ii) have a width that is based on the angle at which the graphics input tool is being held.

16. The one or more computer-readable storage media of claim 15 , storing additional computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform additional operations comprising:

determining, based on the one or more images, an amount of bending of a tip of the graphics input tool during at least a portion of the first path of motion of the graphics input tool; and

based on the amount of bending, determining a width of one or more markings that correspond to the at least one of the shape or one or more contours that follow the first path of motion.

17. The one or more computer-readable storage media of claim 15 , storing additional computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform additional operations comprising:

in response to determining that the graphics input tool is within a threshold distance of the AR graphics display surface, activating one or more features of the augmented reality content item to at least one of create or modify the AR graphic;

determining that the graphics input tool is outside of the threshold distance of the AR graphics display surface; and

in response to determining that the graphics input tool is outside of the threshold distance of the AR graphics display surface, deactivating the one or more features of the augmented reality content item.

18. The one or more computer-readable storage media of claim 15 , wherein the object detection service determines that an object of interest of a plurality of objects of interest corresponds to the AR graphics display surface based on an analysis of characteristics of the plurality of objects of interest in relation to criteria that are representative of AR graphics display surfaces, the characteristics including one or more minimum dimensions, one or more maximum dimensions, and a maximum amount of surface roughness.

19. The one or more computer-readable storage media of claim 18 , wherein the object detection service ranks the plurality of objects of interest in response to the analysis and determines that the AR graphics display surface corresponds to the object of interest of the plurality of objects of interest having a highest ranking.

20. The one or more computer-readable storage media of claim 15 , storing additional computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform additional operations comprising:

obtaining, based on the camera data, coordinates in real world space for the AR graphics display surface; and

determining, based on the camera data, a distance from the one or more cameras to the AR graphics display surface;

determining that the distance from the one or more cameras to the AR graphics display surface has changed from a first distance to a second distance; and

modifying a level of magnification of the AR graphic within the user interface based on the distance from the one or more cameras to the AR graphics display surface changing from the first distance to the second distance.

Assignments (1)

Continuity (1)

References Cited (102)