Overlap detection for an item recognition system
An item recognition system uses a top camera and one or more peripheral cameras to identify items. The item recognition system may use image embeddings generated based on images captured by the cameras to generate a concatenated embedding that describes an item depicted in the image. The item recognition system may compare the concatenated embedding to reference embeddings to identify the item. Furthermore, the item recognition system may detect when items are overlapping in an image. For example, the item recognition system may apply an overlap detection model to a top image and a pixel-wise mask for the top image to detect whether an item is overlapping with another in the top image. The item recognition system notifies a user of the overlap if detected.
1. An item recognition system comprising:
a receiving surface;
a top camera coupled to a top portion of the item recognition system, wherein the top camera is configured to capture images of the receiving surface from a top-down view;
one or more peripheral cameras coupled to one or more side portions of the item recognition system, wherein the one or more peripheral cameras are configured to capture images of the receiving surface from different peripheral views;
a user interface;
a processor; and
a non-transitory, computer-readable medium storing instructions that, when executed by the processor, cause the processor to:
access a top image comprising an image captured by the top camera, wherein the top image depicts a first item and a second item on the receiving surface;
access one or more peripheral images, each comprising an image captured by a peripheral camera of the one or more peripheral cameras;
generate a pixel-wise mask for the top image based on the top image, wherein pixel-wise mask indicates one or more portions of the top image where an item is depicted;
apply an overlap detection model to the top image, the one or more peripheral images, and the pixel-wise mask to detect whether the first item overlaps with the second item, wherein the overlap detection model is a machine-learning model trained to detect overlapping items in top images based on the top images, peripheral images, and pixel-wise masks of the top images; and
responsive to detecting that the first item overlaps with the second item, present a notification of the overlap through the user interface.
2. The item recognition system of claim 1 , wherein the top camera and the one or more peripheral cameras are configured to capture 2D images of the receiving surface.
3. The item recognition system of claim 1 , wherein the computer-readable medium further stores instructions that cause the processor to:
generate a masked image for the top image based on the top image and the pixel-wise mask for the top image; and
apply the overlap detection model to the masked image.
4. The item recognition system of claim 1 , wherein the computer-readable medium further stores instructions that cause the processor to:
generate a depth value for each pixel of the top image based on the top image and the one or more peripheral images; and
apply the overlap detection model to the generated depth values of the top image.
5. The item recognition system of claim 1 , wherein the computer-readable medium further stores instructions that cause the processor to:
responsive to not detecting that the first item overlaps with the second item, identifying the first item and the second item based on the top image, the one or more peripheral images, and the pixel-wise mask.
6. The item recognition system of claim 1 , wherein the computer-readable medium further stores instructions that cause the processor to:
detect that the first item no longer overlaps with the second item based on a subsequent top image, a subsequent one or more peripheral images, and a subsequent pixel-wise mask.
7. The item recognition system of claim 1 , wherein the instructions for notifying the user of the overlap further comprise instructions that cause the processor to:
display a message on a display of the user interface notifying the user of the overlap.
8. The item recognition system of claim 1 , wherein the pixel-wise mask comprises a set of binary values that indicate which pixels of the top image include the first item or the second item.
9. The item recognition system of claim 1 , wherein the receiving surface comprises a weight sensor configured to capture weight sensor data describing a weight of items placed on the receiving surface, and wherein the computer-readable medium further stores instructions that, when executed by the processor, cause the processor to:
access weight sensor data captured by the weight sensor, wherein the weight sensor data describes a weight of the first item and the second item; and
apply the overlap detection model to the weight sensor data to detect whether the first item overlaps with the second item.
10. The item recognition system of claim 1 , wherein the computer-readable medium further stores instructions that cause the processor to:
apply the overlap detection model to the one or more peripheral images to detect whether the first item overlaps with the second item in the one or more peripheral images.
11. A non-transitory, computer-readable medium storing instructions that, when executed by a processor, cause the processor to:
access a top image comprising an image captured by a top camera of an item recognition system, wherein the top image depicts a first item and a second item on a receiving surface of the item recognition system;
access one or more peripheral images, each comprising an image captured by a peripheral camera of one or more peripheral cameras of the item recognition system;
generate a pixel-wise mask for the top image based on the top image, wherein pixel-wise mask indicates one or more portions of the top image where an item is depicted;
apply an overlap detection model to the top image, the one or more peripheral images, and the pixel-wise mask to detect whether the first item overlaps with the second item, wherein the overlap detection model is a machine-learning model trained to detect overlapping items in top images based on the top images, peripheral images, and pixel-wise masks of the top images; and
responsive to detecting that the first item overlaps with the second item, present a notification of the overlap through a user interface of the item recognition system.
12. The computer-readable medium of claim 11 , wherein the top camera and the one or more peripheral cameras are configured to capture 2D images of the receiving surface.
13. The computer-readable medium of claim 11 , further storing instructions that cause the processor to:
generate a masked image for the top image based on the top image and the pixel-wise mask for the top image; and
apply the overlap detection model to the masked image.
14. The computer-readable medium of claim 11 , further storing instructions that cause the processor to:
generate a depth value for each pixel of the top image based on the top image and the one or more peripheral images; and
apply the overlap detection model to the generated depth values of the top image.
15. The computer-readable medium of claim 11 , further storing instructions that cause the processor to:
responsive to not detecting that the first item overlaps with the second item, identifying the first item and the second item based on the top image, the one or more peripheral images, and the pixel-wise mask.
16. The computer-readable medium of claim 11 , further storing instructions that cause the processor to:
detect that the first item no longer overlaps with the second item based on a subsequent top image, a subsequent one or more peripheral images, and a subsequent pixel-wise mask.
17. The computer-readable medium of claim 11 , wherein the instructions for notifying the user of the overlap further comprise instructions that cause the processor to:
display a message on a display of the user interface notifying the user of the overlap.
18. The computer-readable medium of claim 11 , wherein the pixel-wise mask comprises a set of binary values that indicate which pixels of the top image include the first item or the second item.
19. The computer-readable medium of claim 11 , further storing instructions that cause the processor to:
access weight sensor data captured by a weight sensor coupled to the receiving surface, wherein the weight sensor is configured to capture weight sensor data describing a weight of items placed on the receiving surface; and
apply the overlap detection model to the weight sensor data to detect whether the first item overlaps with the second item.
20. A method comprising:
accessing a top image comprising an image captured by a top camera of an item recognition system, wherein the top image depicts a first item and a second item on a receiving surface of the item recognition system;
accessing one or more peripheral images, each comprising an image captured by a peripheral camera of one or more peripheral cameras of the item recognition system;
generating a pixel-wise mask for the top image based on the top image, wherein pixel-wise mask indicates one or more portions of the top image where an item is depicted;
applying an overlap detection model to the top image, the one or more peripheral images, and the pixel-wise mask to detect whether the first item overlaps with the second item, wherein the overlap detection model is a machine-learning model trained to detect overlapping items in top images based on the top images, peripheral images, and pixel-wise masks of the top images; and
responsive to detecting that the first item overlaps with the second item, presenting a notification of the overlap through a user interface of the item recognition system.
21. The system of claim 5 , wherein the computer-readable medium further stores instructions that cause the processor to:
responsive to identifying the first item and the second item, completing a checkout process based on a first identifier associated with the first item and a second identifier associated with the second item.