Identifying barcode-to-product mismatches using point of sale devices
Disclosed herein are systems and methods for determining whether an unknown product matches a scanned barcode during a checkout process. An edge computing device or other computer system can receive, from an overhead camera at a checkout lane, image data of an unknown product that is placed on a flatbed scanning area, identify candidate product identifications for the unknown product based on applying a classification model and/or product identification models to the image data, and determine based on the candidate product identifications, whether the unknown product matches a product associated with a barcode that is scanned at a POS terminal in the checkout lane. The classification model can be used to determine n-dimensional space feature values for the unknown product and determine which product the unknown product likely matches. The product identification models can be used to determine whether the unknown product is one of the products that are modeled.
1 . A point of sale (POS) terminal for responding in real-time to a ticket swapping event occurring during a checkout process in a retail environment, the POS terminal comprising:
a barcode scanning device positioned in a flatbed scanning area of the POS terminal, the barcode scanning device configured to automatically scan a product identifier affixed to a product during a checkout process and transmit, to an edge computing device, information about the product associated with the scanned product identifier;
a plurality of imaging sensors oriented toward the scanning area, wherein
a first imaging sensor of the plurality of imaging sensors is mounted to a vertical post positioned proximate the scanning area and having a top down field of view (FOV) of the scanning area, the first imaging sensor configured to generate first image data associated with the product as the product is moved over the scanning area, and
a second imaging sensor of the plurality of imaging sensors is integrated within the flatbed scanning area and having a FOV in a horizontal plane parallel to a top surface of the scanning area, the second imaging sensor configured to generate second image data associated with the product as the product is moved over the scanning area,
wherein the FOV of the first imaging sensor and the FOV of the second imaging sensor are configured to at least partially overlap, causing the first image data and the second image data to depict the product as it moves over the scanning area at the same time and from different angles;
a data store configured to store n-dimensional bounded regions associated with a plurality of known product identifiers, wherein each of the n-dimensional bounded regions associated with each known product identifier corresponds to a feature identified for an associated known product; and
an edge computing device in network communication with a user device of a store employee, the edge computing device comprising a processor and memory storing (i) a machine learning model trained to identify product attributes for products during checkout processes, wherein the machine learning model is trained using image data captured from one or more imaging sensors positioned with a FOV matching the FOV of at least one of the first imaging sensor or the second imaging sensor and (ii) instructions that, when executed, cause the processor to perform operations comprising:
receiving, from the barcode scanning device, the information about the product associated with the scanned product identifier;
retrieving, from the data store and based on the received information, a subset of the n-dimensional bounded regions that is associated with the scanned product identifier;
receiving, from the plurality of imaging sensors, the first image data and the second image data associated with the product;
determining, based on processing the first image data and the second image data, an n-dimensional space vector associated with the product, wherein
the processing is performed based on applying the machine learning model to the first image data and the second image data;
determining a likelihood of a ticket swapping event based on
a distance in n-dimensional space between the n-dimensional space vector and the subset of the n-dimensional bounded regions for the product, wherein
a greater distance in the n-dimensional space between the n-dimensional space vector and the subset of the n-dimensional bounded regions for the product indicates a higher likelihood of the ticket swapping event than a smaller distance in the n-dimensional space between the n-dimensional space vector and the subset of the n-dimensional bounded regions;
determining whether the likelihood of the ticket swapping event satisfies one or more threshold ticket swapping criteria; and
based on determining that the likelihood of the ticket swapping event satisfies the one or more threshold ticket swapping criteria, transmitting an indication of the ticket swapping event to the user device of the store employee.
2 . The POS terminal of claim 1 , wherein determining, based on processing the first image data and the second image data, an n-dimensional space vector associated with the product comprises:
identifying features of the product in the first image data and the second image data;
mapping the identified features into the n-dimensional space;
identifying clusters in the n-dimensional space; and
determining the n-dimensional space vector for the product based on the clusters.
3 . The POS terminal of claim 2 , wherein determining, based on processing the first image data and the second image data, an n-dimensional space vector associated with the product comprises:
identifying, for each cluster, a centroid;
determining, for each cluster, distances between each nearest neighbor and the centroid; and
determining the n-dimensional space vector based on averaging the distances between each nearest neighbor and the centroid for each of the clusters.
4 . The POS terminal of claim 1 , wherein determining whether the likelihood of the ticket swapping event satisfies one or more threshold ticket swapping criteria comprises determining whether the distance in the n-dimensional space between the n-dimensional space vector and the subset of the n-dimensional bounded regions associated with the scanned product identifier is greater than a threshold distance value.
5 . The POS terminal of claim 1 , wherein transmitting an indication of the ticket swapping event comprises presenting information about the ticket swapping event in a GUI display of the checkout screen.
6 . The POS terminal of claim 1 , wherein the scanning device comprises a barcode scanner.
7 . The POS terminal of claim 1 , wherein the POS terminal further comprises a flatbed scanning area that receives the products to be purchased by a user during the checkout process, wherein the flatbed scanning area includes the scanning device.
8 . The POS terminal of claim 7 , wherein the POS terminal further comprises a vertical post positioned proximate the flatbed scanning area, the vertical post including at least one of the plurality of imaging sensors positioned above the flatbed scanning area and having a field of view that includes the flatbed scanning area, wherein the at least one imaging sensor is configured to generate third image data associated with the product as it is moved over the flatbed scanning area.
9 . The POS terminal of claim 1 , wherein the POS terminal is further configured to identify, based on the scanned product identifier, a product associated with the scanned product identifier.
10 . The POS terminal of claim 1 , wherein the n-dimensional space vector comprises physical feature values for the product, the physical feature values indicating a plurality of confidence values indicating likelihoods that the product matches the scanned product identifier.
11 . The POS terminal of claim 1 , wherein transmitting an indication of the ticket swapping event comprises:
selecting a response amongst a plurality of responses to the ticket swapping event; and
returning instructions for executing the selected response.
12 . The POS terminal of claim 11 , wherein the processor is further configured to execute the selected response to cause the checkout screen to present updated transaction information, wherein the updated transaction information indicates an updated price of the product.
13 . The POS terminal of claim 1 , wherein transmitting an indication of the ticket swapping event comprises transmitting information associated with the ticket swapping event to a computing device of security personnel.
14 . The POS terminal of claim 1 , wherein the processor comprises an edge computing device.
15 . The POS terminal of claim 1 , wherein each dimension in the n-dimensional space represents a known feature of a product associated with the scanned product identifier.
16 . The POS terminal of claim 1 wherein determining, based on processing the first image data and the second image data, an n-dimensional space vector associated with the product comprises:
retrieving, from the data store, a model that was trained to translate features in the first image data and the second image data associated with the product to an n-dimensional space vector in the n-dimensional space;
applying the model to the first image data and the second image data associated with the product; and
receiving, as output from the model, the n-dimensional space vector associated with the product in the n-dimensional space.
17 . A method for responding in real-time to a ticket swapping event occurring during a checkout process in a retail environment, the method comprising:
receiving, by an edge computing device at a POS terminal from a barcode scanning device located in a flatbed scanning area, information about a product associated with a scanned product identifier;
retrieving, by the edge computing device from a data store, a subset of n-dimensional bounded regions associated with the scanned product identifier, wherein the data store is configured to store n-dimensional bounded regions associated with a plurality of known product identifiers, wherein each of the n-dimensional bounded regions associated with each known product identifier corresponds to a feature identified for an associated known product;
receiving, by the edge computing device from a plurality of imaging sensors, first image data and second image data associated with the product, wherein:
a first imaging sensor of the plurality of imaging sensors is mounted to a vertical post positioned proximate the scanning area and having a top down field of view (FOV) of the scanning area, the first imaging sensor configured to generate first image data associated with the product as the product is moved over the scanning area, and
a second imaging sensor of the plurality of imaging sensors is integrated within the flatbed scanning area and having a FOV in a horizontal plane parallel to a top surface of the scanning area, the second imaging sensor configured to generate second image data associated with the product as the product is moved over the scanning area,
wherein the FOV of the first imaging sensor and the FOV of the second imaging sensor are configured to at least partially overlap causing the first image data and the second image data to depict the product as it moves over the scanning area at the same time and from different angles;
determining, by the edge computing device and based on processing the first image data and the second image data, an n-dimensional space vector associated with the product, wherein the processing is performed based on applying, at the edge computing device, a machine learning model to the first image data and the second image data, wherein the machine learning model was trained to identify one or more product attributes for products during checkout processes, wherein the machine learning model is trained using image data captured from one or more imaging sensors positioned with a FOV matching the FOV of at least one of the first imaging sensor or the second imaging sensor;
determining, by the edge computing device, a likelihood of a ticket swapping event based on
a distance in n-dimensional space between the n-dimensional space vector and the subset of the n-dimensional bounded regions for the product associated with the scanned product identifier, wherein:
a greater distance in the n-dimensional space between the n-dimensional space vector and the subset of the n-dimensional bounded regions for the product indicates a higher likelihood of the ticket swapping event than a smaller distance in the n-dimensional space between the n-dimensional space vector and the subset of the n-dimensional bounded regions;
determining, by the edge computing device, whether the likelihood of the ticket swapping event satisfies one or more threshold ticket swapping criteria; and
based on determining that the likelihood of the ticket swapping event satisfies the one or more threshold ticket swapping criteria, transmitting an indication of the ticket swapping event to the user device of the store employee.
18 . The method of claim 17 , wherein the scanning device is positioned at a scanning area of the POS terminal, the scanning device configured to automatically scan the product identifiers affixed to products during the checkout process.
19 . The method of claim 18 , wherein the plurality of imaging sensors are oriented toward the scanning area, the plurality of imaging sensors configured to generate the first image data and the second image data associated with the products at or around a time of scanning by the scanning device during the checkout process.
20 . The method of claim 17 , wherein determining, by the edge computing device and based on processing the first image data and the second image data, an n-dimensional space vector associated with the product comprises:
identifying features of the product in the first image data and the second image data;
mapping the identified features into the n-dimensional space;
identifying clusters in the n-dimensional space; and
determining the n-dimensional space vector for the product based on the clusters.