SYSTEM FOR ITEM RECOGNITION USING COMPUTER VISION
An item recognition system uses a top camera and one or more peripheral cameras to identify items. The item recognition system may use image embeddings generated based on images captured by the cameras to generate a concatenated embedding that describes an item depicted in the image. The item recognition system may compare the concatenated embedding to reference embeddings to identify the item. Furthermore, the item recognition system may detect when items are overlapping in an image. For example, the item recognition system may apply an overlap detection model to a top image and a pixel-wise mask for the top image to detect whether an item is overlapping with another in the top image. The item recognition system notifies a user of the overlap if detected.
1 . An item recognition system comprising:
a receiving surface;
a top camera coupled to a top portion of the automated checkout system, wherein the top camera is configured to capture images of the receiving surface from a top-down view;
one or more peripheral cameras coupled to one or more side portions of the automated checkout system, wherein the one or more peripheral cameras are configured to capture images of the receiving surface from different peripheral views;
a processor; and
a non-transitory, computer-readable medium storing instructions that, when executed by the processor, cause the processor to:
access a top image comprising an image captured by the top camera;
access one or more peripheral images, each comprising an image captured by a peripheral camera of the one or more peripheral cameras;
identify a region of the top image and a region of each of the one or more peripheral images that depicts an item on the receiving surface;
generate an image embedding for each of the identified regions of the top image and the one or more peripheral images;
concatenate the image embeddings based on a pre-determined ordering of the top camera and the one or more peripheral cameras to form a concatenated embedding; and
identify the item by comparing the concatenated embedding to one or more reference item embeddings, wherein each reference item embedding is associated with an item identifier.
2 . The item recognition system of claim 1 , wherein the top camera and the one or more peripheral cameras are configured to capture 2D images of the receiving surface.
3 . The item recognition system of claim 1 , wherein the instructions for identifying a region of the top image and a region of the one or more peripheral images comprise instructions that cause the processor to:
generate a pixel-wise mask for the top image and a pixel-wise mask for each of the one or more peripheral images, wherein the pixel-wise masks identify pixels of the top image and the one or more peripheral images that include an item.
4 . The item recognition system of claim 3 , wherein the instructions for identifying a region of the top image and a region of the one or more peripheral images comprise instructions that cause the processor to:
generate a bounding box for the item for the top image and a bounding box for the item for each of the one or more peripheral images based on the pixel-wise mask of the top image and the one or more peripheral images.
5 . The item recognition system of claim 4 , wherein the identified regions of the top image and the one or more peripheral images comprise a cropped image based on the bounding boxes of the top image and the one or more peripheral images.
6 . The item recognition system of claim 1 , wherein the instructions for generating the image embedding for each of the identified regions is comprise instructions that cause the processor to:
apply an image embedding model to each of the identified regions, wherein the image embedding model is a machine-learning model trained to generate image embeddings for identified regions of images.
7 . The item recognition system of claim 1 , wherein the instructions for identifying the item comprise instructions that cause the processor to:
receive a set of candidate reference embeddings from a remote server.
8 . The item recognition system of claim 1 , wherein the computer-readable medium further stores instructions that cause the processor to generate an image embedding for each of the identified regions of the top image and the one or more peripheral images responsive to determining that the item does not overlap with another item on the receiving surface.
9 . The item recognition system of claim 1 , wherein the computer-readable medium further stores instructions that cause the processor to:
detect that an item was placed on the receiving surface; and
access the top image and the one or more peripheral images responsive to detecting an item was placed on the receiving surface.
10 . The item recognition system of claim 9 , wherein the instructions for detecting that an item was placed on the receiving surface comprise instructions that cause the processor to:
detect that an item was placed on the receiving surface based on sensor data from one or more weight sensors coupled to the receiving surface.
11 . A non-transitory, computer-readable medium storing instructions that, when executed by a processor, cause the processor to:
access a top image comprising an image captured by a top camera of an item recognition system, wherein the top camera is configured to capture images of a receiving surface of the item recognition system from a top-down view;
access one or more peripheral images, each comprising an image captured by a peripheral camera of one or more peripheral cameras of the item recognition system, wherein the one or more peripheral cameras are configured to capture images of the receiving surface from different peripheral views;
identify a region of the top image and a region of each of the one or more peripheral images that depicts an item on the receiving surface;
generate an image embedding for each of the identified regions of the top image and the one or more peripheral images;
concatenate the image embeddings based on a pre-determined ordering of the top camera and the one or more peripheral cameras to form a concatenated embedding; and
identify the item by comparing the concatenated embedding to one or more reference item embeddings, wherein each reference item embedding is associated with an item identifier.
12 . The computer-readable medium of claim 11 , wherein the top camera and the one or more peripheral cameras are configured to capture 2D images of the receiving surface.
13 . The computer-readable medium of claim 11 , wherein the instructions for identifying a region of the top image and a region of the one or more peripheral images comprise instructions that cause the processor to:
generate a pixel-wise mask for the top image and a pixel-wise mask for each of the one or more peripheral images, wherein the pixel-wise masks identify pixels of the top image and the one or more peripheral images that include an item.
14 . The computer-readable medium of claim 13 , wherein the instructions for identifying a region of the top image and a region of the one or more peripheral images comprise instructions that cause the processor to:
generate a bounding box for the item for the top image and a bounding box for the item for each of the one or more peripheral images based on the pixel-wise mask of the top image and the one or more peripheral images.
15 . The computer-readable medium of claim 14 , wherein the identified regions of the top image and the one or more peripheral images comprise a cropped image based on the bounding boxes of the top image and the one or more peripheral images.
16 . The computer-readable medium of claim 11 , wherein the instructions for generating the image embedding for each of the identified regions is comprise instructions that cause the processor to:
apply an image embedding model to each of the identified regions, wherein the image embedding model is a machine-learning model trained to generate image embeddings for identified regions of images.
17 . The computer-readable medium of claim 11 , wherein the instructions for identifying the item comprise instructions that cause the processor to:
receive a set of candidate reference embeddings from a remote server.
18 . The computer-readable medium of claim 11 , further storing instructions that cause the processor to generate an image embedding for each of the identified regions of the top image and the one or more peripheral images responsive to determining that the item does not overlap with another item on the receiving surface.
19 . The computer-readable medium of claim 11 , wherein further storing instructions that cause the processor to:
detect that an item was placed on the receiving surface; and
access the top image and the one or more peripheral images responsive to detecting an item was placed on the receiving surface.
20 . A method comprising:
accessing a top image comprising an image captured by a top camera of an item recognition system, wherein the top camera is configured to capture images of a receiving surface of the item recognition system from a top-down view;
accessing one or more peripheral images, each comprising an image captured by a peripheral camera of one or more peripheral cameras of the item recognition system, wherein the one or more peripheral cameras are configured to capture images of the receiving surface from different peripheral views;
identifying a region of the top image and a region of each of the one or more peripheral images that depicts an item on the receiving surface;
generating an image embedding for each of the identified regions of the top image and the one or more peripheral images;
concatenating the image embeddings based on a pre-determined ordering of the top camera and the one or more peripheral cameras to form a concatenated embedding; and
identifying the item by comparing the concatenated embedding to one or more reference item embeddings, wherein each reference item embedding is associated with an item identifier.