Machine learning-based assessment of food item quality
Described herein are systems and methods for determining quality levels for food items using image data, such as time lapse RGB, hyperspectral, thermal, and/or multispectral images. The method can include receiving, from imaging devices, image data of food items, performing object detection on the image data to identify a bounding box around each food item, and identifying a quality level of each food item by applying trained models to the bounding boxes. The models were trained using image training data of other food items that was annotated based on previous identifications of a first portion of the other food items as having poor quality features and a second portion as having good quality features. The other food items and the food items are a same type. The method also includes determining, for each food item, a quality level score based on the identified quality level of the food item.
1 . A method for determining quality levels for food items using image data, the method comprising:
receiving, by a computing system and from an imaging device, image data of food items;
performing, by the computing system, object detection on the image data to identify a bounding box around each of the food items in the image data;
determining, by the computing system and based on the bounding box around each of the food items in the image data, a grid structure for the image data, wherein each bounding box is assigned a grid index in the grid structure, wherein the grid index is used to identify the food item in a data store;
identifying, by the computing system and for each of the food items, a quality level of the food item by applying a plurality of trained models to bounding box portions of the image data that contains each of the food items, wherein each of the trained models were trained using image training data of other food items, the image training data being annotated based on previous identifications of a first portion of the other food items as having poor quality features and a second portion of the other food items as having good quality features, wherein the other food items are of a same food type as the food items, wherein the plurality of trained models include a first trained model trained to identify a first quality feature and a second trained model trained to identify a second quality feature that is different than the first quality feature, wherein identifying, by the computing system and for each of the food items, the quality level of the food item comprises:
calibrating colors of the image data to maintain color consistency in the image data;
determining, based on applying a color analyzer trained model to the calibrated bounding box portions of the image data, a median Euclidean distance between a color of the food item in the bounding box portions of the image data and a reference color for the food item, wherein the reference color is associated with a preferred quality level of the food item;
determining whether the median Euclidean distance exceeds a threshold value;
in response to determining that the median Euclidean distance exceeds the threshold value, identifying the quality level of the food item as poor quality; and
in response to determining that the median Euclidean distance is less than the threshold value, identifying the quality level of the food item as good quality;
determining, by the computing system and for each of the food items, a quality level score for the food item based on the identified quality level of the food item; and
storing, by the computing system and for each of the food items, (i) the bounding box portions of the image data that contains the food items, (ii) the grid index, (iii) the identified quality level of the food item, and (iv) the determined quality level score for the food item in the data store.
2 . The method of claim 1 , further comprising:
retrieving, by the computing system and for each of the food items, the quality level score for the food item from the data store;
identifying, by the computing system, supply chain information for the food item that includes a preexisting supply chain schedule and destination for the food item;
determining, by the computing system, whether to modify the supply chain information for the food item based on the retrieved quality level score;
in response to a determination to modify the supply chain information, generating, by the computing system, modified supply chain information based on the received quality level score, wherein the modified supply chain information includes one or more of a modified supply chain schedule and modified destination for the food item; and
transmitting, by the computing system, the modified supply chain information to one or more supply chain actors to implement the modified supply chain information.
3 . The method of claim 2 , wherein the modified supply chain information includes instructions that, when executed by the one or more supply chain actors, cause at least one of (i) the food item to be moved for outbound shipment to a food processing plant, (ii) a change in controlled atmospheric conditions surrounding the food item, (iii) a change in ripening conditions of the food item, (iv) a change to cold storage conditions for the food item, or (v) the food item to be moved for outbound shipment to end-consumers that are geographically closest to a location of the food item.
4 . The method of claim 1 , further comprising:
identifying, by the computing system and using object recognition, a type of the food items depicted in the image data; and
selecting, by the computing system and based on the identified type of the food items, one or more of the plurality of trained models to apply to the bounding box portions of the image data, wherein the selected trained models were trained to identify quality features of food items of the same type.
5 . The method of claim 1 , wherein identifying, by the computing system and for each of the food items, a quality level of the food item comprises:
determining, based on applying an apple yellowing trained model to the bounding box portions of the image data that contain each of the food items, a degree to which the food item has yellowed; and
assigning, based on the degree to which the food item has yellowed, the quality level score of the food item, wherein a score greater than or equal to 0 but less than a threshold level indicates (i) good quality and (ii) no yellowing and a score equal to or less than 100 but greater than the threshold level indicates (i) poor quality and (ii) maximum yellowing of the food item.
6 . The method of claim 1 , wherein identifying, by the computing system and for each of the food items, a quality level of the food item comprises:
determining, based on applying a lime yellowing trained model to the bounding box portions of the image data that contain each of the food items, a degree to which the food item has yellowed; and
assigning, based on the degree to which the food item has yellowed, the quality level score of the food item, wherein a score greater than or equal to a first threshold level but less than a second threshold level indicates (i) good quality and (ii) no yellowing and a score equal to or less than a third threshold level but greater than the second threshold level indicates (i) poor quality and (ii) maximum yellowing.
7 . A method for determining quality levels for food items using image data, the method comprising, receiving, by a computing system and from an imaging device, image data of food items;
performing, by the computing system, object detection on the image data to identify a bounding box around each of the food items in the image data;
determining, by the computing system and based on the bounding box around each of the food items in the image data, a grid structure for the image data, wherein each bounding box is assigned a grid index in the grid structure, wherein the grid index is used to identify the food item in a data store;
identifying, by the computing system and for each of the food items, a quality level of the food item by applying a plurality of trained models to bounding box portions of the image data that contains each of the food items, wherein each of the trained models were trained using image training data of other food items, the image training data being annotated based on previous identifications of a first portion of the other food items as having poor quality features and a second portion of the other food items as having good quality features, wherein the other food items are of a same food type as the food items, wherein the plurality of trained models include a first trained model trained to identify a first quality feature and a second trained model trained to identify a second quality feature that is different than the first quality feature, wherein the quality level score for the food item is an overall quality metric for the food item and wherein the overall quality metric is determined by:
accessing, from a data store, rule-based mappings of (i) different ranges of values for scores corresponding to identification of the first and second quality features to (ii) enumerated categories of food item quality, wherein the enumerated categories of food item quality include at least one of salable, unsalable, edible, nonedible, good quality, bad quality, and okay quality;
iteratively determining, for each of the first and second quality features, whether respective rule-based mappings are met;
identifying, based on the respective rule-based mappings being met, an enumerated category of food item quality for the food item; and
assigning the identified category to the food item as the overall quality metric for the food item;
determining, by the computing system and for each of the food items, a quality level score for the food item based on the identified quality level of the food item; and
storing, by the computing system and for each of the food items, (i) the bounding box portions of the image data that contains the food items, (ii) the grid index, (iii) the identified quality level of the food item, and (iv) the determined quality level score for the food item in the data store.
8 . The method of claim 1 , wherein identifying, by the computing system and for each of the food items, a quality level of the food item comprises:
calibrating colors of the image data to maintain color consistency in the image data;
obtaining color samples for each food item in the calibrated bounding box portions of the image data;
mapping the color samples into a multi-dimensional color space;
identifying a direction of maximal color change across all of the color samples;
predicting a location of the maximal color change along the identified direction for the food item, wherein the location of the maximal color change is a one-dimensional quality metric for the food item; and
identifying the quality level of the food item as good quality based on determining that the one-dimensional quality metric for the food item is within a threshold quality range.
9 . The method of claim 7 , wherein identifying, by the computing system and for each of the food items, a quality level of the food item comprises:
assigning, based on applying a wrinkle analyzer model to each of the bounding box portions of the image data that contain each of the food items, a binary value indicating whether wrinkles are present or absent on a surface of the food item represented in the bounding box portion;
determining, a percentage of wrinkle coverage on the surface of the food item represented by the bounding box portions of the image data based on (i) summating a quantity of the bounding box portions that are assigned a binary value indicating that wrinkles are present on the surface of the food item represented in the corresponding bounding box portions and (ii) dividing the summation by a total quantity of the bounding box portions of the image data; and
assigning, based on the determined percentage of wrinkle coverage satisfying threshold wrinkle criteria, the quality level score of the food item, wherein the score is a binary value indicating whether wrinkles are present on or absent from the surface of the food item.
10 . The method of claim 1 , wherein identifying, by the computing system and for each of the food items, the quality level of the food item comprises:
determining, based on applying a calyx browning-trained model to the bounding box portions of the image data that contain each of the food items, a median color value of the food item; and
assigning, based on the median color value of the food item, the quality level score of the food item, wherein the quality level score is assigned (i) a string value of “good” based on the median color value being less than a first threshold color range, (ii) a string value of “ok” based on the median color value being greater than the first threshold color range and less than a second threshold color range, and (iii) a string value of “bad” based on the median color value being greater than the second threshold color range.
11 . The method of claim 1 , wherein identifying, by the computing system and for each of the food items, the quality level of the food item comprises:
determining, based on applying a banana stage analyzer-trained model to the bounding box portions of the image data that contain each of the food items, a color value of the food item; and
assigning, based on mapping the color value of the food item to expected threshold color values for a plurality of ripeness stages, the quality level score of the food item, wherein the quality level score is a string value indicating a current ripeness stage of the food item.
12 . The method of claim 1 , wherein identifying, by the computing system and for each of the food items, the quality level of the food item comprises:
extracting, based on applying an object detection model to the bounding box portions of the image data that contain each of the food items, a stem of the food item;
determining, based on applying a cherry stem color-trained model to the extracted stem of the food item, a median color value of the stem of the food item; and
assigning, based on the median color value of the stem of the food item, the quality level score of the food item, wherein the quality level score is assigned (i) a string value of “good” based on the median color value being within a first threshold color range, (ii) a string value of “ok” based on the median color value being within a second threshold color range that does not include the first threshold color range, and (iii) a string value of “bad” based on the median color value being within a third threshold color range that does not include at least the first threshold color range.
13 . The method of claim 1 , wherein the plurality of trained models were trained using at least one of convolutional neural networks (CNNs) and Partial Least Squares (PLSs).
14 . The method of claim 1 , wherein the image data includes at least one of RGB images, hyperspectral images, multispectral images, thermal images, nuclear magnetic resonance (NMR) images, and magnetic resonance images (MRIs).
15 . A method for determining quality levels for food items using image data, the method comprising:
receiving, by a computing system and from an imaging device, image data of food items;
performing, by the computing system, object detection on the image data to identify a bounding box around each of the food items in the image data;
determining, by the computing system and based on the bounding box around each of the food items in the image data, a grid structure for the image data, wherein each bounding box is assigned a grid index in the grid structure, wherein the grid index is used to identify the food item in a data store;
identifying, by the computing system and for each of the food items, a quality level of the food item by applying a plurality of trained models to bounding box portions of the image data that contains each of the food items, wherein each of the trained models were trained using image training data of other food items, the image training data being annotated based on previous identifications of a first portion of the other food items as having poor quality features and a second portion of the other food items as having good quality features, wherein the other food items are of a same food type as the food items, wherein the plurality of trained models include a first trained model trained to identify a first quality feature and a second trained model trained to identify a second quality feature that is different than the first quality feature, wherein identifying, by the computing system and for each of the food items, the quality level of the food item comprises:
identifying, based on applying an internal quality trained model to the bounding box portions of the image data that contain each of the food items, internal quality defects in the food item, wherein the internal quality trained model was trained using image training data of other food items being previously annotated as having no internal defects, diffuse browning, vascular browning, flesh adhesion, internal rot, stem rot, and tissue breakdown; and
determining, based on the internal quality defects, an overall quality level score of the food item, wherein a score of “good” indicates no internal quality defects in the food item, a score of “okay” indicates some internal quality defects in the food item, and a score of “bad” indicates most internal quality defects in the food item;
determining, by the computing system and for each of the food items, a quality level score for the food item based on the identified quality level of the food item; and
storing, by the computing system and for each of the food items, (i) the bounding box portions of the image data that contains the food items, (ii) the grid index, (iii) the identified quality level of the food item, and (iv) the determined quality level score for the food item in the data store.
16 . The method of claim 15 , wherein identifying, by the computing system and for each of the food items, the quality level of the food item comprises:
determining, based on applying dry matter trained model to the bounding box portions of the image data that contain each of the food items, dry matter content of the food item; and
wherein the quality level score of the food item is identified as a numeric value that indicates a quantity of the dry matter content of the food item.
17 . A method for determining quality levels for food items using image data, the method comprising
receiving, by a computing system and from an imaging device, image data of food items;
performing, by the computing system, object detection on the image data to identify a bounding box around each of the food items in the image data;
determining, by the computing system and based on the bounding box around each of the food items in the image data, a grid structure for the image data, wherein each bounding box is assigned a grid index in the grid structure, wherein the grid index is used to identify the food item in a data store;
identifying, by the computing system and for each of the food items, a quality level of the food item by applying a plurality of trained models to bounding box portions of the image data that contains each of the food items, wherein each of the trained models were trained using image training data of other food items, the image training data being annotated based on previous identifications of a first portion of the other food items as having poor quality features and a second portion of the other food items as having good quality features, wherein the other food items are of a same food type as the food items, wherein the plurality of trained models include a first trained model trained to identify a first quality feature and a second trained model trained to identify a second quality feature that is different than the first quality feature, wherein identifying, by the computing system and for each of the food items, the quality level of the food item comprises:
identifying, based on applying a product identifier trained model to the bounding box portions of the image data that contain each of the food items, a product label on the food item; and
decoding, using optical character recognition (OCR), at least one of text and a product identifier on the product label,
wherein identifying, based on the decoded at least one of text and a product identifier on product label, the quality level score of the food item as a string value that includes at least one of the decoded text and the product identifier;
determining, by the computing system and for each of the food items, a quality level score for the food item based on the identified quality level of the food item; and
storing, by the computing system and for each of the food items, (i) the bounding box portions of the image data that contains the food items, (ii) the grid index, (iii) the identified quality level of the food item, and (iv) the determined quality level score for the food item in the data store.
18 . The method of claim 7 , wherein identifying, by the computing system and for each of the food items, the quality level of the food item comprises:
determining, based on applying a shape trained model to the bounding box portions of the image data that contain each of the food items, a shape of the food item; and
wherein the quality level score of the food item is determined as a numeric value that indicates a bend radius of the food item.
19 . A system for determining quality levels for food items using image data, the system comprising:
one or more imaging devices configured to measure image data for food items of a same food type; and
at least one computing system configured to:
receive, from the one or more imaging devices, image data of the food items;
perform object detection on the image data to identify a bounding box around each of the food items in the image data;
determine, based on the bounding box around each of the food items in the image data, a grid structure for the image data, wherein each bounding box is assigned a grid index in the grid structure, wherein the grid index is used to identify the food item in a data store;
identify, for each of the food items, a quality level of the food item by applying a plurality of trained models to bounding box portions of the image data that contains each of the food items, wherein each of the trained models were trained using image training data of other food items, the image training data being annotated based on previous identifications of a first portion of the other food items as having poor quality features and a second portion of the other food items as having good quality features, wherein the other food items are of a same food type as the food items, wherein the plurality of trained models include a first trained model trained to identify a first quality feature and a second trained model trained to identify a second quality feature that is different than the first quality feature, wherein identifying, by the computing system and for each of the food items, the quality level of the food item comprises:
calibrating colors of the image data to maintain color consistency in the image data;
determining, based on applying a color analyzer trained model to the calibrated bounding box portions of the image data, a median Euclidean distance between a color of the food item in the bounding box portions of the image data and a reference color for the food item, wherein the reference color is associated with a preferred quality level of the food item;
determining whether the median Euclidean distance exceeds a threshold value;
in response to determining that the median Euclidean distance exceeds the threshold value, identifying the quality level of the food item as poor quality; and
in response to determining that the median Euclidean distance is less than the threshold value, identifying the quality level of the food item as good quality;
determine, for each of the food items, a quality level score for the food item based on the identified quality level of the food item; and
store, for each of the food items, (i) the bounding box portions of the image data that contains the food items, (ii) the grid index, (iii) the identified quality level of the food item, and (iv) the determined quality level score for the food item in the data store.
20 . A system for determining quality levels for food items using image data, the system comprising:
a photobox having first, second, third, and fourth walls and a ceiling, wherein the photobox comprises:
an opening on the first wall that is configured to receive a flat containing food items of a same food type;
a flap configured to cover the opening to prevent ambient light from entering the photobox;
at least one light configured inside the photobox that provides consistent lighting to illuminate the flat containing the food items; and
one or more imaging devices attached to the ceiling of the photobox and configured to capture image data of the food items on the flat; and
at least one computing system configured to:
receive, from the one or more imaging devices, image data of the food items;
perform object detection on the image data to identify a bounding box around each of the food items in the image data;
determine, based on the bounding box around each of the food items in the image data, a grid structure for the image data, wherein each bounding box is assigned a grid index in the grid structure, wherein the grid index is used to identify the food item in a data store;
identify, for each of the food items, a quality level of the food item by applying a plurality of trained models to bounding box portions of the image data that contains each of the food items, wherein each of the trained models were trained using image training data of other food items, the image training data being annotated based on previous identifications of a first portion of the other food items as having poor quality features and a second portion of the other food items as having good quality features, wherein the other food items are of a same food type as the food items, wherein the plurality of trained models include a first trained model trained to identify a first quality feature and a second trained model trained to identify a second quality feature that is different than the first quality feature;
determine, for each of the food items, a quality level score for the food item based on the identified quality level of the food item; and
store, for each of the food items, (i) the bounding box portions of the image data that contains the food items, (ii) the grid index, (iii) the identified quality level of the food item, and (iv) the determined quality level score for the food item in the data store.
21 . The system of claim 20 , wherein the photobox further comprises a camera rig configured to extend across the ceiling of the photobox, wherein:
the camera rig includes three tubes connected at a tee connector,
a first tube extends from the tee connector to the first wall, a second tube extends from the tee connector to the second wall,
the second wall is opposite the first wall, and a third tube extends from the tee connector to the third wall,
the third tube is perpendicular to the first and second tubes, and
the one or more imaging devices are mounted to the camera rig proximate the tee connector and along at least one of the three tubes.
22 . A system for determining quality levels for food items using image data, the system comprising:
one or more imaging devices configured to measure image data for food items of a same food type; and
at least one computing system configured to perform operations comprising:
receiving, by a computing system and from an imaging device, image data of food items;
performing, by the computing system, object detection on the image data to identify a bounding box around each of the food items in the image data;
determining, by the computing system and based on the bounding box around each of the food items in the image data, a grid structure for the image data, wherein each bounding box is assigned a grid index in the grid structure, wherein the grid index is used to identify the food item in a data store;
identifying, by the computing system and for each of the food items, a quality level of the food item by applying a plurality of trained models to bounding box portions of the image data that contains each of the food items, wherein each of the trained models were trained using image training data of other food items, the image training data being annotated based on previous identifications of a first portion of the other food items as having poor quality features and a second portion of the other food items as having good quality features, wherein the other food items are of a same food type as the food items, wherein the plurality of trained models include a first trained model trained to identify a first quality feature and a second trained model trained to identify a second quality feature that is different than the first quality feature, wherein identifying, by the computing system and for each of the food items, the quality level of the food item comprises:
identifying, based on applying an internal quality trained model to the bounding box portions of the image data that contain each of the food items, internal quality defects in the food item, wherein the internal quality trained model was trained using image training data of other food items being previously annotated as having no internal defects, diffuse browning, vascular browning, flesh adhesion, internal rot, stem rot, and tissue breakdown; and
determining, based on the internal quality defects, an overall quality level score of the food item, wherein a score of “good” indicates no internal quality defects in the food item, a score of “okay” indicates some internal quality defects in the food item, and a score of “bad” indicates most internal quality defects in the food item;
determining, by the computing system and for each of the food items, a quality level score for the food item based on the identified quality level of the food item; and
storing, by the computing system and for each of the food items, (i) the bounding box portions of the image data that contains the food items, (ii) the grid index, (iii) the identified quality level of the food item, and (iv) the determined quality level score for the food item in the data store.
23 . A system for determining quality levels for food items using image data, the system comprising:
one or more imaging devices configured to measure image data for food items of a same food type; and
at least one computing system configured to perform operations comprising:
receiving, by a computing system and from an imaging device, image data of food items;
performing, by the computing system, object detection on the image data to identify a bounding box around each of the food items in the image data;
determining, by the computing system and based on the bounding box around each of the food items in the image data, a grid structure for the image data, wherein each bounding box is assigned a grid index in the grid structure, wherein the grid index is used to identify the food item in a data store;
identifying, by the computing system and for each of the food items, a quality level of the food item by applying a plurality of trained models to bounding box portions of the image data that contains each of the food items, wherein each of the trained models were trained using image training data of other food items, the image training data being annotated based on previous identifications of a first portion of the other food items as having poor quality features and a second portion of the other food items as having good quality features, wherein the other food items are of a same food type as the food items, wherein the plurality of trained models include a first trained model trained to identify a first quality feature and a second trained model trained to identify a second quality feature that is different than the first quality feature, wherein identifying, by the computing system and for each of the food items, the quality level of the food item comprises:
identifying, based on applying a product identifier trained model to the bounding box portions of the image data that contain each of the food items, a product label on the food item; and
decoding, using optical character recognition (OCR), at least one of text and a product identifier on the product label,
wherein identifying, based on the decoded at least one of text and a product identifier on product label, the quality level score of the food item as a string value that includes at least one of the decoded text and the product identifier;
determining, by the computing system and for each of the food items, a quality level score for the food item based on the identified quality level of the food item; and
storing, by the computing system and for each of the food items, (i) the bounding box portions of the image data that contains the food items, (ii) the grid index, (iii) the identified quality level of the food item, and (iv) the determined quality level score for the food item in the data store.
24 . A system for determining quality levels for food items using image data, the system comprising:
one or more imaging devices configured to measure image data for food items of a same food type; and
one or more imaging devices configured to measure image data for food items of a same food type; and
at least one computing system configured to perform operations comprising:
receiving, by a computing system and from an imaging device, image data of food items;
performing, by the computing system, object detection on the image data to identify a bounding box around each of the food items in the image data;
determining, by the computing system and based on the bounding box around each of the food items in the image data, a grid structure for the image data, wherein each bounding box is assigned a grid index in the grid structure, wherein the grid index is used to identify the food item in a data store;
identifying, by the computing system and for each of the food items, a quality level of the food item by applying a plurality of trained models to bounding box portions of the image data that contains each of the food items, wherein each of the trained models were trained using image training data of other food items, the image training data being annotated based on previous identifications of a first portion of the other food items as having poor quality features and a second portion of the other food items as having good quality features, wherein the other food items are of a same food type as the food items, wherein the plurality of trained models include a first trained model trained to identify a first quality feature and a second trained model trained to identify a second quality feature that is different than the first quality feature, wherein the quality level score for the food item is an overall quality metric for the food item and wherein the overall quality metric is determined by:
accessing, from a data store, rule-based mappings of (i) different ranges of values for scores corresponding to identification of the first and second quality features to (ii) enumerated categories of food item quality, wherein the enumerated categories of food item quality include at least one of salable, unsalable, edible, nonedible, good quality, bad quality, and okay quality;
iteratively determining, for each of the first and second quality features, whether respective rule-based mappings are met;
identifying, based on the respective rule-based mappings being met, an enumerated category of food item quality for the food item; and
assigning the identified category to the food item as the overall quality metric for the food item;
determining, by the computing system and for each of the food items, a quality level score for the food item based on the identified quality level of the food item; and
storing, by the computing system and for each of the food items, (i) the bounding box portions of the image data that contains the food items, (ii) the grid index, (iii) the identified quality level of the food item, and (iv) the determined quality level score for the food item in the data store.