System and methods for performing order cart audits
Techniques are described for automating and computerizing order cart audits to reduce the overall costs associated with shipping incorrect items, missing items, and/or additional items. In some cases, the system may be configured to perform an audit of a completed or filled order cart. The facility operator may pass the order cart through an audit area prior to loading the items on a vehicle for transport. During the audit, the system may capture sensor data associated with the order cart and identify each item present. The system may determine whether or not each item is part of the order and notify an operator accordingly.
1. A method comprising:
receiving first sensor data associated with an item;
generating first machine readable content from the first sensor data based at least in part on a first optical character recognition system, the first machine readable content representing a label associated with the item;
generating second machine readable content from the first sensor data based at least in part on a second optical character recognition system, the second machine readable content representing the label associated with the item;
generating third machine readable content from the first sensor data based at least in part on the first machine readable content and the second machine readable content;
generating a first classification for the item based at least in part on the third machine readable content and a first classification system;
generating a second classification for the item based at least in part on the third machine readable content and a second classification system;
generating an assigned classification for the item based at least in part on the first classification and the second classification;
generating extracted data from the third machine readable content, the extracted data associated with one or more key value descriptors assigned based at least in part on the assigned classification, the extracted data includes a first code, a second code, and a third code associated with the item, the first code different from the second code and the third code, the second code different than the third code, and each of the first code, the second code, and the third code representing an identity of the item;
determining, based at least in part on the assigned classification for the item and the extracted data, an identifier for the item; and
verifying the identifier of the item based at least in part on the first code, the second code, and the third code, an identity of the item.
2. The method of claim 1 , further comprising segmenting the first sensor data of the item into multiple images, individual images associate with a portion of the item; and
wherein the first classification system differs from the second classification system and the first classification is based at least in part on at least one of:
a machine learned model;
a dictionary; or
a heuristics model.
3. The method of claim 1 , wherein verifying the identifier of the item is based at least in part on a stored list.
4. The method of claim 1 , further comprising:
identifying an imperfection within the first sensor data;
determining a first bounding box associated with the imperfection;
determining a second bounding box associated with content of the label;
preforming at least one first operation on the first bounding box to reduce a visibility of the imperfection; and
preforming at least one second operation on the second bounding box to increase a visibility of the label.
5. The method of claim 1 , wherein:
the first classification system generates a first confidence value associated with the first classification; and
the second classification system generates a second confidence value associated with the second classification.
6. The method of claim 5 , wherein:
the first classification differs from the second classification;
a difference in the first confidence value and the second confidence value is less than or equal to a threshold;
the method further comprises:
generating, responsive to the difference being less than or equal to the threshold, a third classification and third confidence value for the label based at least in part on the third machine readable content and a third classification system; and
generating the assigned classification is based at least in part on the third classification and the third confidence value.
7. The method of claim 1 , wherein the extracted data includes an optical mark and the generating the extracted data comprises:
detecting the optical mark within the machine readable content;
determining the optical mark is selected;
detecting content within the third machine readable content that is associated with the optical mark based at least in part on one or more of a textual analysis, a geometric pattern analysis, or a measurement thresholds; and
extracting the content as the extracted data.
8. The method of claim 1 , wherein
generating fourth machine readable content by removing content associated with the extracted data from the third machine readable content; and
generating additional extracted data from the fourth machine readable content, the additional extracted data associated with the one or more of the key value descriptors assigned.
9. A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving second sensor data associated with the item;
determining a region associated with the item within the second sensor data; and
determining first sensor data representing the region at a desired zoom;
detecting, based at least in part on the first sensor data, a foreground label associated with the item;
determining, based at least on the first sensor data representing the foreground label, a first region associate with the foreground label;
determine, based at least in part on the first region associated with the foreground label, a first code, a second code, and a third code associated with the item, the first code different from the second code and the third code, the second code different than the third code, and each of the first code, the second code, and the third code representing an identity of the item;
generating first machine readable content based at least in part on a first optical character recognition applied to the first sensor data, the first machine readable content representing the foreground label associated with the item;
generating second machine readable content based at least in part on a second optical character recognition applied to the first sensor data, the second machine readable content representing the foreground label associated with the item;
generating third machine readable content based at least in part on the first machine readable content and the second machine readable content;
generating a first classification for the item based at least in part on the third machine readable content and a first classification system;
generating a second classification for the item based at least in part on the third machine readable content and a second classification system;
generating an assigned classification for the item based at least in part on the first classification and the second classification; and
determining, based at least in part on the first code, the second code, and the third code, and the assigned classification, an identity of the item; and
outputting the identity of the item to a remote system.
10. The system as recited in claim 9 , wherein the operations further comprise:
verifying the identity of the item based at least in part on a stored list.
11. The system as recited in claim 9 , wherein the operations further comprise:
verifying the identity of the item based at least in part on a classification of the item determined based at least in part on the first sensor data.
12. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving image data of a label associated with an item;
generating first machine readable content based at least in part on a first optical character recognition technique and the image data, the first machine readable content representing the label;
generating second machine readable content based at least in part on a second optical character recognition technique and the image data, the second machine readable content representing the label;
generating third machine readable content based at least in part on the first machine readable content and the second machine readable content;
generating a first classification for the label based at least in part on the third machine readable content and a first classification system;
generating a second classification for the label based at least in part on the third machine readable content and a second classification system;
generating an assigned classification for the label based at least in part on the first classification and the second classification;
generating extracted data from the third machine readable content, the extracted data associated with one or more key value descriptors assigned based at least in part on the assigned classification, the extracted data includes a first code, a second code, and a third code associated with the item, the first code different from the second code and the third code, the second code different than the third code, and each of the first code, the second code, and the third code representing an identity of the item;
determining, based at least in part on the assigned classification for the item and the extracted data, an identifier for the item; and
verifying the identifier of the item based at least in part on the first code, the second code, and the third code, an identity of the item.
13. The one or more non-transitory computer-readable media of claim 12 , further comprising segmenting the image data into multiple images, individual images associate with a portion of the label; and
wherein the first classification system differs from the second classification system and the first classification is based at least in part on at least one of:
a machine learned model;
a dictionary; or
a heuristics model.
14. The one or more non-transitory computer-readable media of claim 12 , wherein the operations further comprises:
identifying an imperfection within the image data;
determining a first bounding box associated with the imperfection;
determining a second bounding box associated with content of the label;
preforming at least one first operation on the first bounding box to reduce a visibility of the imperfection; and
preforming at least one second operation on the second bounding box to increase a visibility of the content of the label.
15. The one or more non-transitory computer-readable media of claim 12 , wherein the one or more key value descriptors includes an identity of the item.
16. The one or more non-transitory computer-readable media of claim 12 , wherein the extracted data includes an optical mark and the generating the extracted data comprises:
detecting the optical mark within the machine readable content;
determining the optical mark is selected;
detecting content within the third machine readable content that is associated with the optical mark based at least in part on one or more of a textual analysis, a geometric pattern analysis, or a measurement thresholds; and
extracting the content as the extracted data.
17. The method of claim 1 , wherein the one or more key value descriptors includes an identity of the item.
18. The system as recited in claim 9 , wherein:
the first classification system generates a first confidence value associated with the first classification; and
the second classification system generates a second confidence value associated with the second classification.
19. The system as recited in claim 9 , wherein the extracted data includes an optical mark and the generating the extracted data comprises:
detecting the optical mark within the machine readable content;
determining the optical mark is selected;
detecting content within the third machine readable content that is associated with the optical mark based at least in part on one or more of a textual analysis, a geometric pattern analysis, or a measurement thresholds; and
extracting the content as the extracted data.
20. The one or more non-transitory computer-readable media of claim 12 , wherein verifying the identifier of the item is based at least in part on a stored list.