Granted Patent B2

US 12,374,059 · App. 18/216,784 · Granted Jul 29, 2025

Inventors: Ilteris Canberk (Marina Del Rey, CA); Sumant Milind Hanumante (Aliso Viejo, CA); Stanislav Minakov (Playa Vista, CA); Dhritiman Sagar (Marina Del Rey, CA)

Assignee: Snap Inc.

G06T19/006G06F3/167G06T7/73

Abstract

Claims (83)

1. An augmented reality environment enhancement method for guiding a user through an environment using an eyewear device having an image capture system, a position detection system, and a display system, the method comprising:

capturing, with the image capture system, frames of image data in the environment;

monitoring, with the position detection system, a current location and orientation of the eyewear device within the environment with respect to objects in the environment as the eyewear device moves within the environment containing the objects;

identifying feature points within the frames of image data;

registering, with the position detection system, environment enhancement locations at the identified feature points;

generating, with the display system responsive to the current location and orientation of the eyewear device relative to at least one registered environment enhancement location, an overlay image responsive to the position of the eyewear device relative to registered environment enhancement locations, the overlay image including environment enhancement graphics that are adjusted for display at the environment enhancement locations as the eyewear is moved relative to the registered environment enhancement locations, wherein generating the overlay image comprises analyzing positions of the registered environment enhancement locations, identifying objects located at the registered environment enhancement locations, comparing an identified object to environment enhancement objects, each environment enhancement object associated with an environment enhancement graphic, selecting the environment enhancement graphic associated with the environment enhancement object matching the identified object, and generating the overlay image including the selected environment enhancement graphic positioned at an environment enhancement location of the environment enhancement object matching the identified object; and

presenting, by the display system, the overlay image.

2. The method of claim 1 , wherein at least one of the environment enhancement locations is a dynamic image including a plurality of images, each image having a different orientation, and wherein generating the overlay image includes:

monitoring a counter; and

generating, with the display system responsive to the counter and the current location and orientation of the eyewear device, successive overlay images from the plurality of images, the at least one registered environment enhancement location having a different orientation in adjacent images of the successive overlay images.

3. The method of claim 1 , wherein generating the overlay image comprises analyzing positions of the environment enhancement locations and respective shapes of the environment enhancement graphics, identifying environment enhancement graphics that would overlap if displayed at positions of the registered environment enhancement locations, replacing the environment enhancement graphics that would overlap with a larger enhancement graphic for use in generating the overlay image, and generating the overlay image including the larger enhancement graphic positioned adjacent positions of the environment enhancement locations.

4. The method of claim 1 , wherein identifying feature points comprises:

detecting potential feature points in the frames of image data;

identifying common potential feature points in the frames of image data that are in a plurality of adjacent frames; and

designating the common potential feature points as feature points.

5. The method of claim 1 , wherein generating the overlay image comprises:

analyzing positions of the environment enhancement locations;

identifying objects located at the environment enhancement locations;

comparing an identified object to environment enhancement objects, each environment enhancement object associated with an environment enhancement graphic;

selecting the environment enhancement graphic associated with the environment enhancement object matching the identified object; and

generating the overlay image including selected environment enhancement graphics positioned at environment enhancement locations.

6. The method of claim 1 , wherein the eyewear device further comprises an audio system and wherein the method further comprises:

presenting an audio track by the audio system;

monitoring a number of feature points; and

adjusting a volume of the audio track responsive to the number of feature points.

7. The method of claim 1 , wherein the environment enhancement graphics have a plurality of visual attributes and wherein generating the overlay image comprises:

determining a distance between the environment enhancement locations and a current position of the eyewear device;

adjusting at least one of the plurality of visual attributes responsive to the determined distance; and

generating, with the display system responsive to the current position of the eyewear device, the overlay image including the environment enhancement graphics as adjusted responsive to the determined distance.

8. The method of claim 7 , wherein the at least one attribute includes size of the environment enhancement graphic.

9. An electronic eyewear device that provides augmented reality environment enhancement, comprising:

an image capture system;

a position detection system;

a display system;

a processor;

a memory; and

programming in the memory, wherein execution of the programming by the processor configures the eyewear device to perform functions, including functions to:

capture, with the image capture system, frames of image data in an environment;

monitor, with the position detection system, a current location and orientation of the eyewear device within the environment with respect to objects in the environment as the eyewear device moves within the environment containing the objects;

identify feature points within the frames of image data;

register, with the position detection system, environment enhancement locations at the identified feature points;

generate, with the display system responsive to the current location and orientation of the eyewear device relative to at least one registered environment enhancement location, an overlay image responsive to the position of the eyewear device relative to registered environment enhancement locations, the overlay image including environment enhancement graphics that are adjusted for display at the environment enhancement locations as the eyewear is moved relative to the registered environment enhancement locations, wherein generating the overlay image comprises analyzing positions of the registered environment enhancement locations, identifying objects located at the registered environment enhancement locations, comparing an identified object to environment enhancement objects, each environment enhancement object associated with an environment enhancement graphic, selecting the environment enhancement graphic associated with the environment enhancement object matching the identified object, and generating the overlay image including the selected environment enhancement graphic positioned at an environment enhancement location of the environment enhancement object matching the identified object; and

present, by the display system, the overlay image.

10. The device of claim 9 , wherein at least one of the environment enhancement locations is a dynamic image including a plurality of images, each image having a different orientation, and wherein the function to generate the overlay image includes functions to:

monitor a counter; and

generate, with the display system responsive to the counter and the current location and orientation of the eyewear device, successive overlay images from the plurality of images, the at least one environment enhancement location having a different orientation in adjacent images of the successive overlay images.

11. The device of claim 9 , wherein the function to generate the overlay image includes functions to analyze positions of the environment enhancement locations and respective shapes of the environment enhancement graphics, identify environment enhancement graphics that would overlap if displayed at positions of the registered environment enhancement locations, replace the environment enhancement graphics that would overlap with a larger enhancement graphic for use in generating the overlay image, and generate the overlay image including the larger enhancement graphic positioned adjacent positions of the environment enhancement locations.

12. The device of claim 9 , wherein the function to identify feature points includes functions to:

detect potential feature points in the frames of image data;

identify common potential feature points in the frames of image data that are in a plurality of adjacent frames; and

designate the common potential feature points as feature points.

13. The device of claim 9 , wherein the function to generate the overlay image includes functions to:

analyze positions of the environment enhancement locations;

identify objects located at the environment enhancement locations;

compare an identified object to environment enhancement objects, each environment enhancement object associated with an environment enhancement graphic;

select the environment enhancement graphic associated with the environment enhancement object matching the identified object; and

generate the overlay image including selected environment enhancement graphics positioned at environment enhancement locations.

14. The device of claim 9 , wherein the eyewear device further comprises an audio system and wherein execution of the programming by the processor further configures the eyewear device to perform additional functions, including functions to:

present an audio track by the audio system;

monitor a number of feature points; and

adjust a volume of the audio track responsive to the number of feature points.

15. The device of claim 9 , wherein the environment enhancement graphics have a plurality of visual attributes and wherein the function to generate the overlay image includes functions to:

determine a distance between the environment enhancement locations and the current position of the eyewear device;

adjust at least one of the plurality of visual attributes responsive to the determined distance; and

generate, with the display system responsive to the current position of the eyewear device, the overlay image including the environment enhancement graphics as adjusted responsive to the determined distance.

16. The device of claim 15 , wherein the at least one attribute includes size of the environment enhancement graphic.

17. The device of claim 9 , wherein the display system comprises:

a see-through display supported by the eyewear device configured to present the environment enhancement locations.

18. A non-transitory computer-readable medium storing program code for guiding a user through an environment when executed by an eyewear device having an imaging system, a position detection system, and a display system, the program code, when executed, is operative to cause an electronic processor to perform the steps of:

capturing, with the image capture system, frames of image data in the environment;

monitoring, with the position detection system, a current location and orientation of the eyewear device within the environment with respect to objects in the environment as the eyewear device moves within the environment containing the objects;

identifying feature points within the frames of image data;

registering, with the position detection system, environment enhancement locations at the identified feature points;

generating, with the display system responsive to the current location and orientation of the eyewear device relative to at least one registered environment enhancement location, an overlay image responsive to the position of the eyewear device relative to registered environment enhancement locations, the overlay image including environment enhancement graphics that are adjusted for display at the environment enhancement locations as the eyewear is moved relative to the registered environment enhancement locations, wherein generating the overlay image comprises analyzing positions of the registered environment enhancement locations, identifying objects located at the registered environment enhancement locations, comparing an identified object to environment enhancement objects, each environment enhancement object associated with an environment enhancement graphic, selecting the environment enhancement graphic associated with the environment enhancement object matching the identified object, and generating the overlay image including the selected environment enhancement graphic positioned at an environment enhancement location of the environment enhancement object matching the identified object; and

presenting, by the display system, the overlay image.

19. The non-transitory computer-readable medium storing the program code of claim 18 , wherein at least one of the environment enhancement locations is a dynamic image including a plurality of images, each image having a different orientation, and wherein the program code to generate the overlay image, when executed, is operative to cause the electronic processor to perform the steps of:

monitoring a counter; and

generating, with the display system responsive to the counter and the current location and orientation of the eyewear device, successive overlay images from the plurality of images, the at least one registered environment enhancement location having a different orientation in adjacent images of the successive overlay images.

20. The non-transitory computer-readable medium storing the program code of claim 18 , wherein the program code for generating the overlay image, when executed, is operative to cause the electronic processor to perform the steps of:

analyzing positions of the environment enhancement locations and respective shapes of the environment enhancement graphics;

identifying environment enhancement graphics that would overlap if displayed at positions of the registered environment enhancement locations;

replacing the environment enhancement graphics that would overlap with a larger enhancement graphic for use in generating the overlay image, and

generating the overlay image including the larger enhancement graphic positioned adjacent positions of the environment enhancement locations.

Assignments (1)

Continuity (3)

References Cited (13)