IP Library › Granted Patent US 12,100,066

Granted Patent B2

US 12,100,066 · App. 17/736,142 · Granted Sep 24, 2024

Curated contextual overlays for augmented reality experiences

Inventor: Tejas Bahulkar (Sunnyvale, CA)

Assignee: Snap Inc.

G06T11/00 G06V20/20 G06V20/68 G06F3/04842 G06F3/0488 G06F3/167

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12,100,066
App. No.: 17/736,142
Granted: Sep 24, 2024
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Example systems, devices, media, and methods are described for curating and presenting a contextual overlay that includes graphical elements and virtual elements in an augmented reality experience. A contextual overlay application implements and controls the capturing of frames of video data within a field of view of the camera. The image processing system detects, in the captured frames of video data, one or more food items in the physical environment. Detecting food items may involve computer vision and machine-trained classification models. The method includes retrieving data associated with the detected food item, curating a contextual overlay based on the retrieved data and a configurable profile, and presenting the contextual overlay on the display.

Claims (94)

1. A method of presenting a contextual overlay in response to items detected with an eyewear device in a physical environment, the eyewear device comprising a memory, a camera, a microphone, a loudspeaker, a contextual overlay application, an image processing system, a localization system, and a display, the method comprising:

maintaining in the memory a profile comprising an activity and a locale;

perceiving a start command with the microphone;

playing through the loudspeaker a confirming message in response to the perceived start command;

capturing frames of video data at a frame rate and within a field of view of the camera;

with the image processing system, detecting in the captured frames of video data a food item at a current item position relative to the display;

determining with the localization system a current eyewear location relative to the physical environment based exclusively on the captured frames of video data;

updating the current eyewear location in accordance with the frame rate;

retrieving data associated with the detected food item with the contextual overlay application;

curating a contextual overlay based on the retrieved data and the profile, populating the contextual overlay in accordance with the current eyewear location relative to the locale; and

presenting on the display the contextual overlay at an overlay position relative to the display.

2. The method of claim 1 , further comprising:

detecting a subsequent food item;

retrieving subsequent data associated with the detected subsequent food item; and

curating further the contextual overlay based on the retrieved subsequent data.

3. The method of claim 1 , wherein the process of retrieving data further comprises:

gathering food information from a source selected from the group consisting of a food data library, a recipe library, and one or more websites on the Internet.

4. The method of claim 1 , wherein the process of curating a contextual overlay further comprises:

storing in the memory a content setting associated with the profile; and

populating the contextual overlay in accordance with the maintained profile and the retrieved data.

5. The method of claim 1 , wherein the process of presenting a contextual overlay further comprises:

presenting on the display one or more graphical elements, each associated with the retrieved data;

presenting a label associated with the detected food item on the display adjacent the current item position; and

playing, through the loudspeaker, a presentation message in association with the presented contextual overlay.

6. The method of claim 1 , wherein the process of detecting a food item further comprises:

accessing a classification model to identify the food item among a plurality of other items in the captured frames of video data.

7. The method of claim 1 , wherein the method further comprises:

determining, with the localization system, the current eyewear location relative to the current item position;

calculating a correlation between the current item position and the display, in accordance with the current eyewear location; and

presenting the contextual overlay in accordance with the calculated correlation, such that the contextual overlay is persistently presented adjacent the current item position as the eyewear moves through the physical environment.

8. The method of claim 1 , wherein the eyewear device further comprises a touchpad and wherein the process of presenting the contextual overlay further comprises:

detecting a current fingertip location relative to the touchpad;

presenting a movable element at a current element position on the display in accordance with the detected current fingertip location;

presenting a highlight adjacent a first graphical element positioned on the display nearest the current element position;

detecting a tapping gesture relative to the touchpad, based on the detected current fingertip location; and

executing a selecting action in accordance with the detected tapping gesture and the current element position.

9. The method of claim 8 , wherein the process of executing a selecting action further comprises:

identifying a selected graphical element positioned on the display nearest the current element position, wherein the selecting action relative to the first graphical element is a process selected from the group consisting of opening, saving, sharing, and removing.

10. A contextual overlay system, comprising:

an eyewear device comprising a camera, a microphone, a loudspeaker, a contextual overlay application, an image processing system, a localization system, a memory, a processor, and a display; and

programming in the memory, wherein execution of the programming by the processor configures the eyewear device to perform functions, including functions to:

maintain in the memory a profile comprising an activity and a locale;

perceive a start command with the microphone;

play through the loudspeaker a confirming message in response to the perceived start command;

capture frames of video data at a frame rate and within a field of view of the camera;

with the image processing system, detect in the captured frames of video data a food item at a current item position relative to the display;

determine with the localization system a current eyewear location relative to the physical environment based exclusively on the captured frames of video data;

updating the current eyewear location in accordance with the frame rate;

retrieve data associated with the detected food item with the contextual overlay application;

curate a contextual overlay based on the retrieved data and the profile, such that the contextual overlay is populated in accordance with current eyewear location relative to the locale; and

present on the display the contextual overlay at an overlay position relative to the display.

11. The contextual overlay system of claim 10 , wherein the function to retrieve data further comprises functions to:

gather food information from a source selected from the group consisting of a food data library, a recipe library, and one or more websites on the Internet.

12. The contextual overlay system of claim 10 , wherein the function to curate a contextual overlay further comprises functions to:

store in the memory a content setting associated with the profile; and

populate the contextual overlay in accordance with the maintained profile and the retrieved data.

13. The contextual overlay system of claim 10 , wherein the function to present a contextual overlay on the display further comprises functions to:

present on the display one or more graphical elements, each associated with the retrieved data;

present a label associated with the detected food item on the display adjacent the current item position; and

play, through the loudspeaker, a presentation message in association with the presented contextual overlay.

14. The contextual overlay system of claim 10 , wherein the function to detect a food item further comprises functions to:

access a classification model to identify the food item among a plurality of other items in the captured frames of video data.

15. The contextual overlay system of claim 10 , wherein the eyewear device further comprises a touchpad and the function to present the contextual overlay on the display further comprises functions to:

detect a current fingertip location relative to the touchpad;

present a movable element at a current element position on the display in accordance with the detected current fingertip location;

present a highlight adjacent a first graphical element positioned on the display nearest the current element position;

detect a tapping gesture relative to the touchpad, based on the detected current fingertip location; and

execute a selecting action in accordance with the detected tapping gesture and the current element position, wherein the selecting action relative to the first graphical element is a process selected from the group consisting of opening, saving, sharing, and removing.

16. A non-transitory computer-readable medium storing program code which, when executed, is operative to cause an electronic processor to perform the steps of:

perceiving a start command with a microphone of an eyewear device, the eyewear device further comprising a memory, a camera, a loudspeaker, a contextual overlay application, an image processing system, a localization system, and a display;