Multiview association of multiple items for item recognition
Images of a transaction area comprising items are received during a checkout. Red-Green-Blue (RGB) data and Depth data are received with each image. Pixels captured in any given image by a given camera are pre-aligned with physical locations of a transaction area. Depth data provided by each camera and that camera's pre-alignment to the area are processed to map pixels in each image taken to X-Y coordinates within the area. X-Y coordinates for each item and for each image are grouped together as a single item within the area. RGB data for each image and item is used as a set of image patches per item. For each item, the corresponding patches are passed to a classification Machine-Learning Model (MLM) that returns an item code for each patch. A particular item code is selected for each set and the item codes are used to process the checkout.
1 . A method, comprising:
obtaining a bounding box for each item depicted within each of a plurality of images that are captured of a transaction area, each image depicting multiple items located within the transaction area;
obtaining depth information and Red, Green, Blue (RGB) data for each bounding box of each image;
mapping each item represented within each bounding box to physical locations within the transaction area using the depth information;
identifying an RGB image patch in each bounding box using the mapping;
associating a set of RGB image patches with a unique item present in the transaction area; and
providing item codes for the multiple items using sets of RGB image patches for a transaction associated with a customer by processing preconfigured rules to select a particular item code for a corresponding item when a given set of patches for the corresponding item results in different item codes being supplied by a machine-learning model (MLM), wherein an item code is selected when a majority of outputs from the MLM is associated with that item code;
wherein the transaction is processed during a frictionless checkout that does not require item code scanning by an attendant or the customer.
2 . The method of claim 1 , wherein obtaining the bounding box for each item further includes providing the plurality of images to a bounding box MLM and receiving particular bounding boxes for each image as output from the bounding box MLM.
3 . The method of claim 2 , wherein obtaining the depth information further comprises mapping the depth information of each bounding box for each image to corresponding RGB data.
4 . The method of claim 3 , wherein mapping further includes averaging the depth information associated with corresponding RGB data identified within each bounding box to determine a distance a given item is away from a camera that supplied a corresponding image.
5 . The method of claim 4 , wherein averaging further includes identifying a center of each bounding box within a corresponding image.
6 . The method of claim 5 , wherein identifying the center further includes calculating X-Y coordinates of the transaction area for each item within each bounding box using a corresponding distance, a corresponding center, and a known pixel map for each camera that maps corresponding pixels taken by a corresponding camera to corresponding physical locations.
7 . The method of claim 6 , wherein associating further includes grouping RGB image patches together in corresponding sets based on corresponding X-Y coordinates associated with a corresponding item across corresponding bounding boxes.
8 . The method of claim 7 , wherein providing further includes passing each set of RGB image patches to an item classification MLM and receiving a candidate item code as output from the item classification MLM for each RGB patch in a corresponding set.
9 . The method of claim 8 , wherein passing further includes selecting a particular item code for a particular set of RGB image patches to assign to a particular item based on the particular item code being provided as output from the item classification MLM more times than any other item codes that were provided as output from the item classification MLM for the particular set of RGB image patches.
10 . The method of claim 9 , wherein selecting further includes determining a total item count for the multiple items present within the transaction area for the transaction by counting a total number of each set of the RGB image patches.
11 . The method of claim 10 , wherein providing further includes providing the total item count with the item codes associated with the multiple items for the transaction.
12 . A method, comprising:
obtaining multiple images from multiple cameras taken of a transaction area comprising multiple items;
generating a three-dimensional (3D) point cloud rendering of the transaction area with the multiple items using Red-Green-Blue (RGB) data and depth information associated with the multiple images by aligning each camera's 3D point cloud such that each camera can be used in unison to build on top of one another, similar to image stitching but occurring in 3D because of the depth information;
removing background information from the 3D point cloud rendering creating a modified 3D point cloud rendering;
clustering a corresponding RGB data and corresponding depth information into clusters;
reprojecting the corresponding RGB data within each cluster back to corresponding images creating two-dimensional (2D) bounding boxes around each portion of each image associated with a corresponding cluster;
using particular RGB data corresponding to each 2D bounding box across the multiple images to generate sets of image patches; and
processing each set of image patches to identify an item code for each item present within the transaction area;
wherein corresponding item codes are provided to a transaction manager that processes a transaction during a frictionless checkout without requiring any item code scanning.
13 . The method of claim 12 further comprising, identifying a total item count from a total number of the clusters.
14 . The method of claim 13 further comprising, providing the total item count and each of the corresponding item codes to the transaction manager that is processing a transaction of a customer for the multiple items present within the transaction area.
15 . The method of claim 12 , wherein obtaining further includes obtaining each image from a separate one of the multiple cameras oriented and positioned in a different angle and view of the transaction area from remaining ones of the multiple cameras.
16 . The method of claim 12 , wherein generating further includes aligning and unifying camera specific 3D point cloud renderings of each camera into the 3D point cloud rendering.
17 . The method of claim 16 , wherein removing further includes processing 3D cropping or background subtraction to remove the background information from the 3D point cloud rendering.
18 . The method of claim 16 , wherein processing further includes passing each image patch for each set to an item classification machine-learning model (MLM) and receiving a candidate item code for each image patch as output, selecting a particular item code for a corresponding set based on rules, and associating the particular item code with a particular item present within the transaction area.
19 . A system, comprising:
a plurality of depth cameras;
a server comprising at least one processor and a non-transitory computer-readable storage medium;
the non-transitory computer-readable storage medium comprises executable instructions; and
the executable instructions when executed by the at least one processor from the non-transitory computer-readable storage medium cause the at least one processor to perform operations comprising:
obtaining images captured by the plurality of depth cameras of a transaction area, wherein the transaction area comprises items in a transaction of a customer;
processing depth information and Red-Green-Blue (RGB) data associated with each image and pixel alignment maps associated with each depth camera to determine physical locations of each of the items within the transaction area by averaging depth values to estimate a distance each item, in a bounding box, is away from a corresponding camera and calculating X-Y coordinates of each item within a corresponding bounding box into a physical location of the transaction area using preconfigured pixel coordinates mapping to real-world coordinates of the transaction area;
grouping corresponding RGB data associated with corresponding physical locations into groupings;
identifying a RGB image patch for each item within each image using the groupings;
assembling a set of RGB image patches per grouping;
passing each RGB image patch of each set to an item classification Machine-Learning Model (MLM) and receiving candidate item codes for each set of RGB image patches;
processing rules to select a particular item code for each set from corresponding candidate item codes and associating the particular item code with a particular one of the items represented in the images by a particular one of the groupings;
identifying a total item count for the items as a total number of the groupings; and
providing the total item count and corresponding item codes for the items to a transaction manager that processes the transaction during a frictionless checkout for the customer without requiring any manual scanning of item codes by either the customer or an attendant.
20 . The system of claim 19 , wherein the plurality of depth cameras are affixed to a basket, or a cart carried by the customer or wherein the plurality of depth cameras are affixed to or surround the transaction area that is associated with a transaction terminal where the customer is performing the transaction.