Granted Patent B2

US 12,284,324 · App. 17/888,948 · Granted Apr 22, 2025

Inventors: Tianying Chen (Pittsburgh, PA); Timothy Chong (Kirkland, WA); Sven Kratz (Mercer Island, WA); Fannie Liu (New York, NY); Andrés Monroy-Hernández (Seattle, WA); Olivia Seow (Cambridge, MA); Yu Jiang Tham (Seattle, WA); Rajan Vaish (Beverly Hills, CA); Lei Zhang (Ann Arbor, MI)

Assignee: Snap Inc.

H04N13/111G06T3/60G06V10/457G06V10/764G06V20/20G06T2200/24

Abstract

Claims (97)

1. An augmented reality (AR) device for generating an AR experience, the AR device comprising:

a camera system configured to capture images of a scene within a field of view;

a display system configured to present AR overlays on a display, the display having a viewing area corresponding to the camera system field of view;

a user input system configured to receive input from a user, wherein the input includes hand gestures captured by the camera system; and

a processor coupled to the camera system, the display system, and the user input system, the processor configured to:

capture, via the camera system, images of the scene;

identify an object receiving surface and corresponding surface coordinates within the scene;

identify, responsive to the input received from the user via the user input system, including the hand gestures captured by the camera system, an AR primary object and a prop object, wherein the AR primary object is associated with at least one set of primary object coordinates and one of the AR primary object or the prop object is identified by the user capturing an image of a physical object with the camera system and designating, according and in response to the input received from the user, the physical object captured in the image as the one of the AR primary object or the prop object;

establish a logical connection between the AR primary object and the prop object;

generate AR overlays including actions associated with the AR primary object responsive to commands received from the user via the user input system that position the AR primary object adjacent the object receiving surface responsive to the primary object coordinates and the surface coordinates within the scene and that position the AR primary object and the prop object with respect to one another in accordance with the logical connection; and

present, via the display system, the generated AR overlays;

wherein the prop object is a physical object within the scene and wherein to identify the physical object the processor is configured to receive a selection of the physical object via the user input system; and

wherein the processor is further configured to:

identify a category for the selected physical object; and

select logic for the physical object responsive to the identified category, the selected logic specifying predefined interactions between the AR primary object and the physical object;

wherein the established logical connection is based on the selected logic and the generated AR overlays are responsive to commands received via the user input system.

2. The AR device of claim 1 , wherein the processor is further configured to:

record the AR overlays;

generate an AR file including the recorded overlays; and

transmit the AR file.

3. The AR device of claim 1 , wherein the predefined interactions include a movement of the AR primary object to present an appearance of the AR primary object manipulating the prop object.

4. An augmented reality (AR) device for generating an AR experience, the AR device comprising:

a camera system configured to capture images of a scene within a field of view;

a display system configured to present AR overlays on a display, the display having a viewing area corresponding to the camera system field of view;

a user input system configured to receive input from a user, wherein the input includes hand gestures captured by the camera system; and

a processor coupled to the camera system, the display system, and the user input system, the processor configured to:

capture, via the camera system, images of the scene;

identify an object receiving surface and corresponding surface coordinates within the scene;

identify, responsive to the input received from the user via the user input system, including the hand gestures captured by the camera system, an AR primary object and a prop object, wherein the AR primary object is associated with at least one set of primary object coordinates and one of the AR primary object or the prop object is identified by the user capturing an image of a physical object with the camera system and designating, according and in response to the input received from the user, the physical object captured in the image as the one of the AR primary object or the prop object;

establish a logical connection between the AR primary object and the prop object;

generate AR overlays including actions associated with the AR primary object responsive to commands received from the user via the user input system that position the AR primary object adjacent the object receiving surface responsive to the primary object coordinates and the surface coordinates within the scene and that position the AR primary object and the prop object with respect to one another in accordance with the logical connection;

present, via the display system, the generated AR overlays,

wherein the prop object is an AR secondary object and wherein to identify the AR secondary object the processor is configured to receive input via the user input system and generate the AR secondary object responsive to the user input, and

wherein the processor is further configured to:

identify a category for the generated AR secondary object; and

select logic for the generated AR secondary object responsive to the identified category, the selected logic specifying predefined interactions between the AR primary object and the generated AR secondary object;

wherein the established logical connection is based on the selected logic and the generated AR overlays are responsive to commands received via the user input system.

5. The AR device of claim 4 , wherein the processor is further configured to:

record the AR overlays;

generate an AR file including the recorded overlays; and

transmit the AR file.

6. An augmented reality (AR) method for generating an AR experience, the method comprising:

capturing, via a camera system, images of a scene;

identifying an object receiving surface and corresponding surface coordinates within the scene;

identifying, responsive to input received from a user via a user input system including hand gestures captured by the camera system, an AR primary object and a prop object, wherein the AR primary object is associated with at least one set of primary object coordinates and one of the AR primary object or the prop object is identified by the user capturing an image of a physical object with the camera system and designating, according and in response to the input received from the user, the physical object captured in the image as the one of the AR primary object or the prop object;

establishing a logical connection between the AR primary object and the prop object;

generating AR overlays including actions associated with the AR primary object responsive to commands received from the user via the user input system that position the AR primary object adjacent the object receiving surface responsive to the primary object coordinates and the surface coordinates within the scene and that position the AR primary object and the prop object with respect to one another in accordance with the logical connection; and

presenting, via a display system, the generated AR overlays;

wherein the prop object is a physical object within the scene and wherein identifying the physical object comprises receiving a selection of the physical object via the user input system; and

the method further comprises:

identifying a category for the selected physical object; and

selecting logic for the physical object responsive to the identified category, the selected logic specifying predefined interactions between the AR primary object and the physical object;

wherein the established logical connection is based on the selected logic and the generated AR overlays are responsive to commands received via the user input system.

7. The method of claim 6 , further comprising:

recording the AR overlays;

generating an AR file including the recorded overlays; and

transmitting the AR file.

8. The method of claim 6 , wherein the predefined interactions include a movement of the AR primary object to present an appearance of the AR primary object manipulating the AR secondary object.

9. An augmented reality (AR) method for generating an AR experience, the method comprising:

capturing, via a camera system, images of a scene;

identifying an object receiving surface and corresponding surface coordinates within the scene;

identifying, responsive to input received from a user via a user input system including hand gestures captured by the camera system, an AR primary object and a prop object, wherein the AR primary object is associated with at least one set of primary object coordinates and one of the AR primary object or the prop object is identified by the user capturing an image of a physical object with the camera system and designating, according and in response to the input received from the user, the physical object captured in the image as the one of the AR primary object or the prop object;

establishing a logical connection between the AR primary object and the prop object;

generating AR overlays including actions associated with the AR primary object responsive to commands received from the user via the user input system that position the AR primary object adjacent the object receiving surface responsive to the primary object coordinates and the surface coordinates within the scene and that position the AR primary object and the prop object with respect to one another in accordance with the logical connection;

presenting, via a display system, the generated AR overlays;

wherein the prop object is an AR secondary object and wherein identifying the AR secondary object comprises receiving input via the user input system and generating the AR secondary object responsive to the user input; and

the method further comprises:

identifying a category for the generated AR secondary object; and

selecting logic for the generated AR secondary object responsive to the identified category, the selected logic specifying predefined interactions between the AR primary object and the generated AR secondary object;

wherein the established logical connection is based on the selected logic and the generated AR overlays are responsive to commands received via the user input system.

10. The method of claim 9 , further comprising:

recording the AR overlays;

generating an AR file including the recorded overlays; and

transmitting the AR file.

11. A non-transitory computer-readable medium storing program code including instructions that, when executed, are operative to cause an electronic processor to perform the steps of:

capturing, via a camera system, images of a scene;

identifying an object receiving surface and corresponding surface coordinates within the scene;

identifying, responsive to input received from a user via a user input system including hand gestures captured by the camera system, an AR primary object and a prop object, wherein the AR primary object is associated with at least one set of primary object coordinates and one of the AR primary object or the prop object is identified by the user capturing an image of a physical object with the camera system and designating, according and in response to the input received from the user, the physical object captured in the image as the one of the AR primary object or the prop object;

establishing a logical connection between the AR primary object and the prop object;

generating AR overlays including actions associated with the AR primary object responsive to commands received from the user via the user input system that position the AR primary object adjacent the object receiving surface responsive to the primary object coordinates and the surface coordinates within the scene and that position the AR primary object and the prop object with respect to one another in accordance with the logical connection; and

presenting, via a display system, the generated AR overlays;

wherein the prop object is a physical object within the scene and wherein identifying the physical object comprises receiving a selection of the physical object via the user input system; and

wherein the instructions, when executed, are operative to cause the electronic processor to perform the additional steps of:

identifying a category for the selected physical object; and

selecting logic for the physical object responsive to the identified category, the selected logic specifying predefined interactions between the AR primary object and the physical object;

wherein the established logical connection is based on the selected logic and the generated AR overlays are responsive to commands received via the user input system.

12. A non-transitory computer-readable medium storing program code including instructions that, when executed, are operative to cause an electronic processor to perform the steps of:

capturing, via a camera system, images of a scene;

identifying an object receiving surface and corresponding surface coordinates within the scene;

identifying, responsive to input received from a user via a user input system including hand gestures captured by the camera system, an AR primary object and a prop object, wherein the AR primary object is associated with at least one set of primary object coordinates and one of the AR primary object or the prop object is identified by the user capturing an image of a physical object with the camera system and designating, according and in response to the input received from the user, the physical object captured in the image as the one of the AR primary object or the prop object;

establishing a logical connection between the AR primary object and the prop object;

generating AR overlays including actions associated with the AR primary object responsive to commands received from the user via the user input system that position the AR primary object adjacent the object receiving surface responsive to the primary object coordinates and the surface coordinates within the scene and that position the AR primary object and the prop object with respect to one another in accordance with the logical connection; and

presenting, via a display system, the generated AR overlays;

wherein the prop object is an AR secondary object and wherein identifying the AR secondary object comprises receiving input via the user input system and generating the AR secondary object responsive to the user input;

wherein the instructions, when executed, are operative to cause the electronic processor to perform the additional steps of:

identifying a category for the generated AR secondary object; and

selecting logic for the generated AR secondary object responsive to the identified category, the selected logic specifying predefined interactions between the AR primary object and the generated AR secondary object, wherein the established logical connection is based on the selected logic and the generated AR overlays are responsive to commands received via the user input system.

Assignments (1)

Continuity (2)

References Cited (44)