Granted Patent B2

US 12,380,713 · App. 17/932,152 · Granted Aug 5, 2025

Inventor: Omar Estrada Diaz (Sacramento, CA)

Assignee: Google LLC

G06V20/68G06V10/82G06V20/50G06V20/70G06V40/20G06F3/011G06F3/14

Abstract

Claims (58)

1. A method, comprising:

capturing, by an image sensor of a wearable device, one or more images of a physical environment around the wearable device;

identifying, based on at least one recognition engine of the wearable device, ingredients detected in the one or more images;

determining an amount of each of the ingredients detected in the one or more images;

identifying, based on the at least one recognition engine, utensils detected in the one or more images;

identifying, based on the at least one recognition engine, actions of a user based on the one or more images;

determining a time that each action of the actions is performed based on the one or more images;

building a recipe based on the ingredients, the amount of each of the ingredients, the utensils, the actions of the user, and the time that each action is performed as captured in the one or more images and processed by the at least one recognition engine;

determining a name for the recipe by the at least one recognition engine;

annotating the recipe with captions on at least a portion of the one or more images including the name of the recipe, the ingredients, the amount of each of the ingredients, the utensils, the actions of the user and the time that each action is performed; and

storing the recipe in a memory.

2. The method of claim 1 , further comprising outputting the recipe on a display of the wearable device.

3. The method of claim 1 , further comprising detecting a time spent on each action and a time between each action.

4. The method of claim 1 , further comprising a smart phone in communication with the wearable device and the at least one recognition engine being disposed on the smart phone.

5. The method of claim 1 , wherein the wearable device comprises smart glasses and identifying the actions comprises identifying the actions based on the one or more images and data output from one or more inertial sensors installed in another wearable device.

6. The method of claim 5 , wherein the one or more inertial sensors comprises any one or any combination of an accelerometer, a gyroscope, and a magnetometer.

7. The method of claim 5 , further comprising projecting, on a display of the smart glasses, the recipe onto a field of view of the user.

8. The method of claim 1 , wherein the at least one recognition engine comprises a neural network trained to recognize any one or any combination of the ingredients, the utensils, and the actions of the user.

9. The method of claim 1 , further comprising:

receiving sensor data; and

activating a recipe building mode of the wearable device, in response to the sensor data satisfying a commencement condition,

wherein the commencement condition comprises at least one of a location indicating the user is proximate to a kitchen or a location indicating the user is proximate to a place of creation of a previous recipe, and the at least one recognition engine identifying one or more objects associated with the kitchen.

10. The method of claim 9 , wherein the commencement condition comprises data from a hunger sensor indicating the user being hungry.

11. The method of claim 9 , wherein activating of the recipe building mode comprises activating the recipe building mode, in response to the at least one recognition engine recognizing an input from the user.

12. The method of claim 1 , further comprising generating a textual recipe as an output based on the recipe with the captions.

13. A wearable device, comprising:

at least one processor; and

a memory storing instructions that, when executed by the at least one processor, configures the at least one processor to:

capture, by an image sensor of the wearable device, one or more images of a physical environment around the wearable device;

identify, based on at least one recognition engine of the wearable device, ingredients detected in the one or more images;

determine an amount of each of the ingredients detected in the one or more images;

identify, based on the at least one recognition engine, utensils detected in the one or more images;

identify, based on the at least one recognition engine, actions of a user based on the one or more images;

determine a time that each action of the actions is performed based on the one or more images;

build a recipe based on the ingredients, the amount of each of the ingredients, the utensils, the actions of the user, and the time that each action is performed as captured in the one or more images and processed by the at least one recognition engine;

determine a name for the recipe;

annotate the recipe with captions on at least a portion of the one or more images including the name of the recipe, the ingredients, the amount of each of the ingredients, the utensils, the actions of the user and the time that each action is performed; and

store the recipe in the memory.

14. The wearable device of claim 13 , wherein the at least one processor is further configured to output the recipe on a display of the wearable device.

15. The wearable device of claim 13 , further comprising a smart phone in communication with the wearable device and the at least one recognition engine being installed on the smart phone.

16. The wearable device of claim 13 , wherein the at least one recognition engine comprises a neural network trained to recognize at least one of the ingredients, the utensils, or the actions of the user.

17. The wearable device of claim 13 , wherein the at least one processor is further configured to:

receive sensor data; and

activate a recipe building mode of the wearable device, in response to the sensor data satisfying a commencement condition,

wherein the commencement condition comprises at least one of a location indicating the user is proximate to a kitchen or a location indicating the user is proximate to a place of creation of a previous recipe, and the at least one recognition engine identifying one or more objects associated with the kitchen.

18. The wearable device of claim 13 , wherein the wearable device includes smart glasses.

19. A computer program product tangibly embodied on a non-transitory computer-readable medium and including executable code that, when executed, causes a wearable device to:

capture, by an image sensor of the wearable device, one or more images of a physical environment around the wearable device;

identify, based on at least one recognition engine of the wearable device, ingredients detected in the one or more images;

determine an amount of each of the ingredients detected in the one or more images;

identify, based on the at least one recognition engine, utensils detected in the one or more images;

identify, based on the at least one recognition engine, actions of a user based on the one or more images;

determine a time that each action of the actions is performed based on the one or more images;

build a recipe based on the ingredients, the amount of each of the ingredients, the utensils, the actions of the user, and the time that each action is performed as captured in the one or more images and processed by the at least one recognition engine;

determine a name for the recipe;

annotate the recipe with captions on at least a portion of the one or more images including the name of the recipe, the ingredients, the amount of each of the ingredients, the utensils, the actions of the user and the time that each action is performed; and

store the recipe in the memory.

20. The computer program product of claim 19 , wherein the wearable device includes smart glasses.

Assignments (1)

Continuity (1)

References Cited (35)