Granted Patent B2

US 12,472,435 · App. 17/887,215 · Granted Nov 18, 2025

Inventors: Ilteris Kaan Canberk (Marina Del Rey, CA); Matthew Hallberg (Los Angeles, CA); William Miles Miller (San Francisco, CA); Lien Le Hong Tran (Santa Monica, CA); Michael Benson Tucker (Marina Del Rey, CA)

Assignee: SNAP INC.

A63F13/655A63F13/211A63F13/533A63F13/573G02B27/0172G06F3/0346G06F3/04815G06F3/0482G06F3/04845G06F3/0488G06T7/20G06V20/20A63F2300/105A63F2300/308A63F2300/646A63F2300/695G02B2027/014G02B2027/0178

Abstract

Claims (70)

1 . A method comprising:

establishing, by one or more processors of an augmented reality (AR) device, a communication with an external user device;

overlaying, by the AR device, a first AR object on a real-world environment being viewed using the AR device;

receiving interaction data from the external user device representing one or more inputs received by the external user device;

in response to receiving the interaction data from the external user device, modifying the first AR object by the AR device;

determining that the interaction data indicates that a swipe input has been received by the external user device, the swipe input causing a 3D carousel to be animated representing different visual properties of the first AR object; and

changing a visual property of the first AR object in response to determining that the interaction data indicates that the swipe input has been received by the external user device.

2 . The method of claim 1 , wherein the external user device comprises a smartphone, and wherein the AR device comprises an AR eyewear device.

3 . The method of claim 1 , further comprising:

detecting, by the AR device, a depiction of the external user device in one or more images of the real-world environment captured by the AR device;

selecting a game controller associated with an external gaming console; and

presenting the game controller as the first AR object on top of the depiction of the external user device in the one or more images.

4 . The method of claim 3 , further comprising:

detecting, by the AR device, a depiction of one or more fingers on a portion of the game controller;

identifying one or more buttons of the game controller at a position where the depiction of the one or more fingers is detected; and

performing a gaming function corresponding to the identified one or more buttons on the external gaming console.

5 . The method of claim 4 , further comprising:

transmitting, by the AR device, a communication to the external gaming console that identifies the gaming function corresponding to the identified one or more buttons over which the depiction of the one or more fingers is detected.

6 . The method of claim 1 , wherein the real-world environment being viewed using the AR device comprises the external user device, and wherein the first AR object comprises an AR beam that extends away from the external user device.

7 . The method of claim 6 , further comprising:

detecting three-dimensional (3D) movement of the external user device based on the received interaction data; and

updating a direction of the AR beam based on the 3D movement.

8 . The method of claim 7 , wherein the external user device receives sensor data comprising at least one of one or more accelerometer measurements or one or more gyroscopic measurements, and wherein the external user device generates the interaction data based on the sensor data.

9 . The method of claim 7 , wherein detecting the 3D movement comprises:

receiving one or more images depicting the external user device by a camera of the AR device; and

applying image processing comprising visual odometry to the one or more images to track a position of the external user device in 3D space.

10 . The method of claim 1 , wherein the real-world environment being viewed using the AR device comprises the external user device, further comprising:

detecting three-dimensional (3D) movement of the external user device based on the received interaction data, the interaction data being generated based on sensor data comprising at least one of one or more accelerometer measurements or one or more gyroscopic measurements; and

propelling the first AR object towards a direction of the 3D movement at a rate corresponding to a rate of the 3D movement.

11 . The method of claim 1 , further comprising:

causing the external user device to display a marker;

capturing an image by the AR device of the marker displayed by the external user device; and

establishing a common coordinate system between the external user device and the AR device in response to capturing the image of the marker.

12 . The method of claim 1 , further comprising:

overlaying a second AR object comprising a wand on the real-world environment being viewed using the AR device;

animating movement of the first AR object towards the AR device along a trajectory;

detecting, based on the interaction data, movement of the external user device;

using a common coordinate system to determine a current 3D position of the external user device based on the detected movement; and

updating an orientation of the second AR object based on the detected movement and the current 3D position.

13 . The method of claim 12 , further comprising:

determining that the current 3D position of the external user device causes the updated orientation of the second AR object to correspond to a current position of the first AR object; and

in response to determining that the updated orientation of the second AR object corresponds to the current position of the first AR object, modifying the trajectory of the first AR object.

14 . The method of claim 1 , further comprising:

causing the external user device to display an option on a screen of the external user device;

determining that the interaction data indicates that the option has been selected on the external user device; and

in response to determining that the interaction data indicates that the option has been selected on the external user device, causing the first AR object to be presented on the AR device.

15 . The method of claim 1 , further comprising:

causing the external user device to display an option on a screen of the external user device;

determining that the interaction data indicates that the option continues to be selected on the external user device by a press-and-hold operation; and

in response to determining that the interaction data indicates that the option continues to be selected on the external user device, presenting a trackpad on the external user device comprising one or more options to change a 3D position of the first AR object.

16 . The method of claim 15 , further comprising:

rotating the first AR object about an axis of the first AR object in response to a first swipe motion along a first direction of the trackpad; and

modifying a perceived distance between the first AR object and the AR device in response to a second swipe motion along a second direction of the trackpad.

17 . A system comprising:

at least one processor of an augmented reality (AR) device configured to perform operations comprising:

establishing a communication with an external user device;

overlaying, by the AR device, a first AR object on a real-world environment being viewed using the AR device;

receiving interaction data from the external user device representing one or more inputs received by the external user device;

in response to receiving the interaction data from the external user device, modifying the first AR object by the AR device;

determining that the interaction data indicates that a swipe input has been received by the external user device, the swipe input causing a 3D carousel to be animated representing different visual properties of the first AR object; and

changing a visual property of the first AR object in response to determining that the interaction data indicates that the swipe input has been received by the external user device.

18 . A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

establishing, by the one or more processors of an augmented reality (AR) device, a communication with an external user device;

overlaying, by the AR device, a first AR object on a real-world environment being viewed using the AR device;

receiving interaction data from the external user device representing one or more inputs received by the external user device;

in response to receiving the interaction data from the external user device, modifying the first AR object by the AR device;

determining that the interaction data indicates that a swipe input has been received by the external user device, the swipe input causing a 3D carousel to be animated representing different visual properties of the first AR object; and

changing a visual property of the first AR object in response to determining that the interaction data indicates that the swipe input has been received by the external user device.

19 . The non-transitory machine-readable storage medium of claim 18 , wherein the external user device comprises a smartphone, and wherein the AR device comprises an AR eyewear device.

20 . The non-transitory machine-readable storage medium of claim 18 , wherein the external user device determines a first origin three-dimensional (3D) position of the external user device relative to a common coordinate system based on a current measurement obtained from one or more sensors of the external user device, and the AR device determining a second origin 3D position of the AR device relative to the common coordinate system based on a position of a marker displayed by the external user device.

Assignments (6)

Continuity (1)

References Cited (400)