IP Library › Granted Patent US 11,508,130
Granted Patent B2
US 11,508,130 · App. 16/900,897 · Granted Nov 22, 2022

Augmented reality environment enhancement

Inventors: Ilteris Canberk (Marina Del Rey, CA); Sumant Hanumante (Marina Del Rey, CA); Dhritiman Sagar (Marina Del Rey, CA); Stanislav Minakov (Playa Vista, CA)
Assignee: Snap Inc.
G06T19/006G06F3/167G06T7/73
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,508,130
App. No.
16/900,897
Granted
Nov 22, 2022
Kind
B2
Abstract

Augmented reality (AR) and virtual reality (VR) environment enhancement using an eyewear device. The eyewear device includes an image capture system, a display system, and a position detection system. The image capture system and position detection system identify feature points within a point cloud that represents captured images of an environment. The display system presents image overlays to a user including enhancement graphics positioned at the feature points within the environment.

Claims (83)

1. An electronic eyewear device that provides augmented reality environment enhancement, comprising:

an image capture system;

a position detection system;

a display system;

a processor;

a memory; and

programming in the memory, wherein execution of the programming by the processor configures the eyewear device to perform functions, including functions to:

capture, with the image capture system, frames of image data in an environment;

monitor, with the position detection system, a current location and orientation of the eyewear device within the environment with respect to objects in the environment as the eyewear device moves within the environment containing the objects;

identify feature points within the frames of image data;

register, with the position detection system, environment enhancement locations at the identified feature points;

generate, with the display system responsive to the current location and orientation of the eyewear device relative to at least one registered environment enhancement location, an overlay image responsive to the position of the eyewear device relative to registered environment enhancement locations, the overlay image including environment enhancement graphics that are adjusted for display at the environment enhancement locations as the eyewear is moved relative to the registered environment enhancement locations, wherein generating the overlay image comprises analyzing positions of the environment enhancement locations and respective shapes of the environment enhancement graphics, identifying environment enhancement graphics that would overlap if displayed at positions of the registered environment enhancement locations, replacing the environment enhancement graphics that would overlap with a larger enhancement graphic for use in generating the overlay image, and generating the overlay image including the larger enhancement graphic positioned adjacent positions of the environment enhancement locations; and

present, by the display system, the overlay image.

2. The device of claim 1 , wherein the function to identify feature points includes functions to:

detect potential feature points in the frames of image data;

identify common potential feature points in the frames of image data that are in a plurality of adjacent frames; and

designate the common potential feature points as feature points.

3. The device of claim 1 , wherein the function to generate an overlay image includes functions to:

analyze positions of the environment enhancement locations;

identify objects located at the environment enhancement locations;

compare an identified object to environment enhancement objects, each environment enhancement object associated with an environment enhancement graphic;

select the environment enhancement graphic associated with the environment enhancement object matching the identified object; and

generate the overlay image including selected environment enhancement graphics positioned at environment enhancement locations.

4. The device of claim 1 , wherein the eyewear device further comprises an audio system and wherein execution of the programming by the processor further configures the eyewear device to perform additional functions, including functions to:

present an audio track by the audio system;

monitor a number of feature points; and

adjust a volume of the audio track responsive to the number of feature points.

5. The device of claim 1 , wherein at least one of the environment enhancement locations is a dynamic image including a plurality of images, each image having a different orientation, and wherein the function to generate the overlay image includes functions to:

monitor a counter; and

generate, with the display system responsive to the counter and the current location and orientation of the eyewear device, successive overlay images from the plurality of images, the at least one environment enhancement location having a different orientation in adjacent images of the successive overlay images.

6. The device of claim 1 , wherein the environment enhancement graphics have a plurality of visual attributes and wherein the function to generate the overlay image includes functions to:

determine a distance between the environment enhancement locations and the current position of the eyewear device;

adjust at least one of the plurality of visual attributes responsive to the determined distance; and

generate, with the display system responsive to the current position of the eyewear device, the overlay image including the environment enhancement graphics as adjusted responsive to the determined distance.

7. The device of claim 6 , wherein the at least one attribute includes size of the environment enhancement graphic.

8. The device of claim 1 , wherein the display system comprises:

a see-through display supported by the eyewear device configured to present the environment enhancement locations.

9. An augmented reality environment enhancement method for guiding a user through an environment using an eyewear device having an image capture system, a position detection system, and a display system, the method comprising:

capturing, with the image capture system, frames of image data in the environment;

monitoring, with the position detection system, a current location and orientation of the eyewear device within the environment with respect to objects in the environment as the eyewear device moves within the environment containing the objects;

identifying feature points within the frames of image data;

registering, with the position detection system, environment enhancement locations at the identified feature points;

generating, with the display system responsive to the current location and orientation of the eyewear device relative to at least one registered environment enhancement location, an overlay image responsive to the position of the eyewear device relative to registered environment enhancement locations, the overlay image including environment enhancement graphics that are adjusted for display at the environment enhancement locations as the eyewear is moved relative to the registered environment enhancement locations, wherein generating the overlay image comprises analyzing positions of the environment enhancement locations and respective shapes of the environment enhancement graphics, identifying environment enhancement graphics that would overlap if displayed at positions of the registered environment enhancement locations, replacing the environment enhancement graphics that would overlap with a larger enhancement graphic for use in generating the overlay image, and generating the overlay image including the larger enhancement graphic positioned adjacent positions of the environment enhancement locations; and

presenting, by the display system, the overlay image.

10. The method of claim 9 , wherein the identifying comprises:

detecting potential feature points in the frames of image data;

identifying common potential feature points in the frames of image data that are in a plurality of adjacent frames; and

designating the common potential feature points as feature points.

11. The method of claim 9 , wherein the generating comprises:

analyzing positions of the environment enhancement locations;

identifying objects located at the environment enhancement locations;

comparing an identified object to environment enhancement objects, each environment enhancement object associated with an environment enhancement graphic;

selecting the environment enhancement graphic associated with the environment enhancement object matching the identified object; and

generating the overlay image including selected environment enhancement graphics positioned at environment enhancement locations.

12. The method of claim 9 , wherein the eyewear device further comprises an audio system and wherein the method further comprising:

presenting an audio track by the audio system;

monitoring a number of feature points; and

adjusting a volume of the audio track responsive to the number of feature points.

13. The method of claim 9 , wherein at least one of the environment enhancement locations is a dynamic image including a plurality of images, each image having a different orientation, and wherein generating the overlay image includes:

monitoring a counter; and

generating, with the display system responsive to the counter and the current location and orientation of the eyewear device, successive overlay images from the plurality of images, the at least one environment enhancement location having a different orientation in adjacent images of the successive overlay images.

14. The method of claim 9 , wherein the environment enhancement graphics have a plurality of visual attributes and wherein the generating the overlay image includes:

determining a distance between the environment enhancement locations and the current position of the eyewear device;

adjusting at least one of the plurality of visual attributes responsive to the determined distance; and

generating, with the display system responsive to the current position of the eyewear device, the overlay image including the environment enhancement graphics as adjusted responsive to the determined distance.

15. The method of claim 14 , wherein the at least one attribute includes size of the environment enhancement graphic.

16. A non-transitory computer-readable medium storing program code for guiding a user through an environment when executed by an eyewear device having an imaging system, a position detection system, and a display system, the program code, when executed, is operative to cause an electronic processor to perform the steps of:

capturing, with the image capture system, frames of image data in the environment;

monitoring, with the position detection system, a current location and orientation of the eyewear device within the environment with respect to objects in the environment as the eyewear device moves within the environment containing the objects;

identifying feature points within the frames of image data;

registering, with the position detection system, environment enhancement locations at the identified feature points;

generating, with the display system responsive to the current location and orientation of the eyewear device relative to at least one registered environment enhancement location, an overlay image responsive to the position of the eyewear device relative to registered environment enhancement locations, the overlay image including environment enhancement graphics that are adjusted for display at the environment enhancement locations as the eyewear is moved relative to the registered environment enhancement locations, wherein generating the overlay image comprises analyzing positions of the environment enhancement locations and respective shapes of the environment enhancement graphics, identifying environment enhancement graphics that would overlap if displayed at positions of the registered environment enhancement locations, replacing the environment enhancement graphics that would overlap with a larger enhancement graphic for use in generating the overlay image, and generating the overlay image including the larger enhancement graphic positioned adjacent positions of the environment enhancement locations; and

presenting, by the display system, the overlay image.

17. The non-transitory computer-readable medium storing the program code of claim 16 , wherein the program code to identity feature points, when executed, is operative to cause the electronic processor to perform the steps of:

detecting potential feature points in the frames of image data;

identifying common potential feature points in the frames of image data that are in a plurality of adjacent frames; and

designating the common potential feature points as feature points.

18. The non-transitory computer-readable medium storing the program code of claim 16 , wherein the program code for generating the overlay image, when executed, is operative to cause the electronic processor to perform the steps of:

analyzing positions of the environment enhancement locations;

identifying objects located at the environment enhancement locations;

comparing an identified object to environment enhancement objects, each environment enhancement object associated with an environment enhancement graphic;

selecting the environment enhancement graphic associated with the environment enhancement object matching the identified object; and

generating the overlay image including selected environment enhancement graphics positioned at environment enhancement locations.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 19, 2022
From: CANBERK, ILTERIS; HANUMANTE, SUMANT; MINAKOV, STANISLAV; SAGAR, DHRITIMAN
To: SNAP INC.
Reel/Frame 061464/0235 →
Continuity (1)
Related Publication 20210390780A1 · Dec 16, 2021
Cited By (1)
US 12,374,059