Granted Patent B2

US 12,524,124 · App. 17/901,611 · Granted Jan 13, 2026

Inventors: Lei Zhang (Ann Arbor, MI); Ava Robinson (Solana Beach, CA); Daekun Kim (Santa Monica, CA); Youjean Cho (New York, NY); Yu Jiang Tham (Seattle, WA); Rajan Vaish (Beverly Hills, CA); Andrés Monroy-Hernández (Seattle, WA)

Assignee: Snap Inc.

G06F3/04815G06F3/011G06F3/0486G06F3/0488G06T7/70G06T19/006G06F3/04842

Abstract

Claims (50)

1 . A method of generating an interactive AR experience using a portable electronic device comprising a microphone, a camera, a localization system, a display, and a memory, the method comprising:

presenting on the display a virtual object, a virtual target, a question, and one or more answer buttons, wherein the question is associated with a scene in a story;

receiving a user input associated with an answer selected from among the one or more answer buttons, wherein the answer is associated with the virtual target and a connecting action;

detecting on the display a selecting action relative to the virtual object, wherein the selecting action comprises at least one of tapping a finger on the display, pressing the finger on the display, or placing on the display a cursor at a start location;

detecting on the display the connecting action in accordance with the answer, wherein the connecting action is operative to connect the virtual object to the virtual target, and wherein the connecting action comprises at least one of tapping the finger at a location relative to the virtual target, sliding the finger toward the virtual target, or sliding the cursor toward the virtual target;

estimating a virtual object location associated with virtual object relative to a physical environment using the localization system;

estimating a target location associated with the virtual target relative to the virtual object location using the localization system;

presenting on the display a virtual tether extending from the virtual object location to the virtual target location, in accordance with the connecting action;

presenting on the display the virtual object persistently together with the virtual target according to the virtual tether, such that the virtual object and the virtual target are persistently separated by the virtual tether; and

storing in the memory an instruction comprising the connecting action, the virtual object, the virtual target, and a scene identifier associated with the scene.

2 . The method of claim 1 , wherein the connecting action further comprises at least one of lifting the finger off the display or releasing the cursor.

3 . The method of claim 1 , further comprising:

retrieving the instruction from the memory; and

presenting the scene on the display based on the instruction.

4 . The method of claim 1 , further comprising:

in response to detecting the connecting action, executing a connect response comprising an action selected from a response group consisting of presenting on the display a transcription of a name associated with the virtual target, playing a sound through a speaker, and playing a message through the speaker.

5 . A portable electronic device comprising:

a microphone,

a camera,

a display,

a processor; and

a memory storing a localization system and storing programming, wherein execution of the programming by the processor configures the portable electronic device to perform functions, including functions to:

present on the display a virtual object, a virtual target, a question, and one or more answer buttons, wherein the question is associated with a scene in a story;

receive a user input associated with an answer selected from among the one or more answer buttons, wherein the answer is associated with the virtual target and a connecting action;

detect on the display a selecting action relative to the virtual object wherein the selecting action comprises at least one of tapping a finger on the display, pressing the finger on the display, or placing on the display a cursor at a start location;

detect on the display the connecting action in accordance with the answer, wherein the connecting action is operative to connect the virtual object to the virtual target, and wherein the connecting action comprises at least one of tapping the finger at a location relative to the virtual target, sliding the finger toward the virtual target, or sliding the cursor toward the virtual target;

estimate a virtual object location associated with virtual object relative to a physical environment using the localization system;

estimate a target location associated with the virtual target relative to the virtual object location using the localization system;

present on the display a virtual tether extending from the virtual object location to the virtual target location, in accordance with the connecting action;

present on the display the virtual object persistently together with the virtual target according to the virtual tether, such that the virtual object and the virtual target are persistently separated by the virtual tether; and

store in the memory an instruction comprising the connecting action, the virtual object, the virtual target, and a scene identifier associated with the scene.

6 . The portable electronic device of claim 5 , wherein the connecting action further comprises at least one of lifting the finger off the display or releasing the cursor.

7 . The portable electronic device of claim 5 , wherein the execution of the programming by the processor further configures the portable electronic device to:

retrieve the instruction from the memory; and

present the scene on the display based on the instruction.

8 . The portable electronic device of claim 5 , wherein the execution of the programming by the processor further configures the portable electronic device to:

in response to the function to detect the connecting action, execute a connect response function comprising an action selected from a response group consisting of presenting on the display a transcription of a name associated with the virtual target, playing a sound through a speaker, and playing a message through the speaker.

9 . A non-transitory computer readable medium storing program code for generating an interactive AR experience using a portable electronic device comprising a microphone, a camera, a localization system, a display, and a memory, wherein the program code, when executed by a processor, is operative to cause the portable electronic device to perform functions, the functions including:

presenting on the display a virtual object, a virtual target, a question and one or more answer buttons, wherein the question is associated with a scene in a story;

receiving a user input associated with an answer selected from among the one or more answer buttons, wherein the answer is associated with the virtual target and a connecting action;

detecting on the display a selecting action relative to the virtual object, wherein the selecting action comprises at least one of tapping a finger on the display, pressing the finger on the display, or placing on the display a cursor at a start location;

detecting on the display the connecting action in accordance with the answer, wherein the connecting action is operative to connect the virtual object to the virtual target, and wherein the connecting action comprises at least one of tapping the finger at a location relative to the virtual target, sliding the finger toward the virtual target, or sliding the cursor toward the virtual target;

estimating a virtual object location associated with virtual object relative to a physical environment using the localization system;

estimating a target location associated with the virtual target relative to the virtual object location using the localization system;

presenting on the display a virtual tether extending from the virtual object location to the virtual target location, in accordance with the connecting action;

presenting on the display the virtual object persistently together with the virtual target according to the virtual tether, such that the virtual object and the virtual target are persistently separated by the virtual tether; and

storing in the memory an instruction comprising the connecting action, the virtual object, the virtual target, and a scene identifier associated with the scene.

10 . The non-transitory computer readable medium of claim 9 , wherein the program code, when executed by the processor, is operative to cause the portable electronic device to perform the additional functions of:

retrieving the instruction from the memory; and

presenting the scene on the display based on the instruction.

Assignments (1)

Continuity (1)

References Cited (72)