Image-based kitchen tracking system with dynamic labeling management

1. A method, comprising:

obtaining image data comprising one or more image frames indicative of a state of an area in a restaurant, wherein the one or more image frames have been captured by a camera, wherein the camera has a field of view that includes the area in the restaurant;

providing the image data as input into a machine learning model that has been trained to detect objects in the one or more image frames, wherein the machine learning model includes an artificial neural network;

obtaining one or more outputs from the machine learning model, wherein the one or more outputs are indicative of a first feature characterization;

obtaining, by a processing device, the first feature characterization that is representative of a first object or action, wherein the first feature characterization is based on the one or more outputs of the machine learning model and includes a first feature vector of values;

determining, based on output by the machine learning model, that the first feature characterization does not meet one or more classification criteria for a set of object or action classifications;

determining, by the processing device, a proximity between the first feature characterization and one of a set of previous feature characterizations corresponding to unclassified objects or actions, wherein the set of previous feature characterizations includes one or more feature vectors, and wherein the proximity is determined by using a distancing algorithm between the first feature vector and the one or more feature vectors;

generating, by the processing device, a new object classification based on the first feature characterization and the one of the set of previous feature characterizations, wherein the new object classification is added to the one or more classification criteria for the set of object or action classifications; and

providing, by the processing device, a notification related to the new object classification.

2. The method of claim 1 , further comprising:

determining, based on the new object classification and the first feature characterization, that an error has occurred, wherein the notification is indicative of the error.

3. The method of claim 2 , wherein the error comprises at least one of:

providing an incorrect ingredient to a food item associated with the first feature characterization;

omitting an intended ingredient for the food item associated with the first feature characterization; or

including the food item associated with the first feature characterization in an incorrect order.

4. The method of claim 1 , further comprising:

responsive to providing the notification, receiving a user indication that the new object classification is proper and not indicative of an error.

5. The method of claim 1 , wherein the area in the restaurant is one of an order placement area, an order payment area, and an order delivery area.

6. The method of claim 1 , wherein the distancing algorithm calculates a Euclidean distance or a cosine similarity, and wherein determining the proximity includes using a clustering algorithm to determine a cluster metric of the one or more feature vectors of the set of previous feature characterizations.

7. The method of claim 1 , further comprising:

obtaining, by the processing device, menu configuration data indicating a first menu item and a set of ingredients corresponding to the first menu item; and

determining, by the processing device, an association between the first object or action and the first menu item based on a comparison between the first feature characterization and the set of ingredients, wherein the new object classification corresponds to the first menu item.

8. The method of claim 1 , wherein the area in the restaurant is an order preparation area.

9. The method of claim 1 , further comprising:

obtaining, by the processing device, from a point of sale system, order data indicative of one or more pending meal orders; and

associating the first object or action with a first pending meal order of the one or more pending meal orders.

10. The method of claim 9 , wherein the first pending meal order comprises a first menu item and a first customization to the first menu item, wherein the new object classification is further based on the first customization to the first menu item.

11. A system, comprising memory and a processing device coupled to the memory, wherein the processing device is configured to:

obtain image data comprising one or more image frames indicative of a state of an area in a restaurant, wherein the one or more image frames have been captured by a camera, wherein the camera has a field of view that includes the area in the restaurant;

provide the image data as input into a machine learning model that has been trained to detect objects in the one or more image frames, wherein the machine learning model includes an artificial neural network;

obtain one or more outputs from the machine learning model, wherein the one or more outputs are indicative of a first feature characterization;

obtain the first feature characterization that is representative of a first object or action, wherein the first feature characterization is based on the one or more outputs of the machine learning model and includes a first feature vector of values;

determine, based on output by the machine learning model, that the first feature characterization does not meet one or more classification criteria for a set of object or action classifications;

determine a proximity between the first feature characterization and one of a set of previous feature characterizations corresponding to unclassified objects or actions, wherein the set of previous feature characterizations includes one or more feature vectors, and wherein the proximity is determined by using a distancing algorithm between the first feature vector and the one or more feature vectors;

generate a new object classification based on the first feature characterization and the one of the set of previous feature characterizations, wherein the new object classification is added to the one or more classification criteria for the set of object or action classifications; and

provide a notification related to the new object classification.

12. The system of claim 11 , wherein the processing device is further configured to determine, based on the new object classification and the first feature characterization, that an error has occurred, wherein the notification is indicative of the error.

13. The system of claim 11 , wherein the processing device is further configured to:

responsive to providing the notification, receive a user indication that the new object classification is proper and not indicative of an error.

14. The system of claim 11 , wherein the processing device is further configured to:

obtain menu configuration data indicating a first menu item and a set of ingredients corresponding to the first menu item; and

determine an association between the first object or action and the first menu item based on a comparison between the first feature characterization and the set of ingredients, wherein the new object classification corresponds to the first menu item.

15. The system of claim 11 , wherein the processing device is further configured to:

obtain, from a point of sale system, order data indicative of one or more pending meal orders; and

associate the first object or action with a first pending meal order of the one or more pending meal orders.