Method and system for automatic cook program determination
In variants, the method can include: sampling cavity measurements of the cook cavity; determining an image representation using the cavity measurements; determining a food class based on the image representation; optionally comparing the image representation to prior image representations; optionally determining a new food class based on the image representation; optionally updating a food identification module with the new food class; and optionally determining a cook program associated with the new food class.
1. A method comprising:
sampling an image of an unknown food within a cooking cavity of a cooking system;
comparing the image to a set of prior images of other food;
determining a cluster of prior images for the image based on the comparison;
training a custom classification module to identify the unknown food, wherein the custom classification module is trained based on the prior images within the cluster; and
associating a custom cook program with an output of the custom classification module.
2. The method of claim 1 , wherein the prior images within the cluster comprise images of other food sharing a similar appearance with the food.
3. The method of claim 1 , wherein the custom classification module is trained based on features extracted from the prior images within the cluster.
4. The method of claim 1 , wherein the cooking system comprises at least one of a grill or an oven.
5. The method of claim 1 , further comprising determining a food class for the unknown food based on the image using a pretrained machine learning model, wherein the custom classification module is trained when the pretrained machine learning model outputs an unknown food class based on the image.
6. The method of claim 5 , wherein the custom classification module is concurrently executed with the pretrained machine learning model on a subsequent image of a subsequent unknown food.
7. The method of claim 6 , wherein the food is cooked according to the custom cook program when a food class output by the custom classification module has a higher confidence score than a food class output by the pretrained machine learning model.
8. The method of claim 1 , wherein comparing the image to a set of prior images comprises:
extracting features from the image;
extracting features from each of the set of prior images; and
comparing the features from the image with the features from each of the set of prior images.
9. The method of claim 8 , wherein the cluster of prior images is determined by clustering the prior images based on the respective features.
10. The method of claim 8 , wherein the comparison comprises clustering the features from the image with features from the set of prior images.
11. The method of claim 10 , wherein the features are extracted using a set of encoding layers of a model pretrained to identify a plurality of known foods.
12. The method of claim 10 , wherein the features are extracted by a set of encoding layers trained end to end to identify a plurality of known foods.
13. A cooking system, comprising:
a set of cooking elements thermally connected to a cook cavity; and
a processing system configured to:
receive an image of food within the cooking cavity;
train a custom classification module to identify the food, comprising:
determining embeddings for the image and each of a set of prior images;
determining a cluster of prior images based on a comparison between the embedding for the image and the embeddings for each of the set of prior images; and
training the custom classification module using the cluster of prior images; and
associate a custom cook program with an output of the custom classification module.
14. The cooking system of claim 13 , wherein the comparison comprises clustering the embedding for the image with the embeddings for each of the set of prior images.
15. The cooking system of claim 14 , wherein the embeddings are clustered using at least one of locality-sensitive hashing or vector distances.
16. The cooking system of claim 13 , wherein the output is assigned a user-defined label.
17. The cooking system of claim 13 , wherein the cooking system comprises a grill.
18. The cooking system of claim 13 , wherein the image is sampled by an image sensor of a user device.
19. The cooking system of claim 13 , wherein the embeddings for the image and each of the set of prior images are determined by a set of encoding layers of a model pretrained to identify a plurality of known foods.
20. The cooking system of claim 19 , wherein the model is concurrently executed with the custom classification module on a subsequent image of a subsequent food, wherein the set of cooking elements are controlled according to the custom cook program when an output of the custom classification module has a higher confidence score than an output of the model.