Automatic labeling of products via expedited checkout system
A portable checkout unit automatically generates training data for an automatic checkout system as a customer collects items in a store. A customer uses an item scanner of portable checkout unit to generate a virtual shopping list of items collected in the shopping cart. When the customer adds a new item to the shopping cart or on some regular interval, the portable checkout unit captures images of the items contained by the shopping cart and can generate bounding boxes for each product in each image. The bounding boxes can be associated with item identifiers from previously-generated bounding boxes to identify the items captured by the bounding boxes. Each bounding box paired with an item identifier can then be used as training data for an automated checkout system.
1. A method comprising:
receiving, at a portable checkout unit attached to a shopping cart, an item identifier identifying an item sold by a store, the shopping cart containing a plurality of items prior to receiving the item identifier;
updating a virtual shopping list based on the received item identifier, the virtual shopping list comprising item identifiers for the plurality of items contained in the shopping cart, the item identifiers comprising the received item identifier based on the updating;
detecting that the item was added to the shopping cart;
responsive to detecting that the item was added to the shopping cart:
capturing one or more images using one or more cameras attached to the shopping cart, the one or more cameras being directed towards the plurality of items contained by the shopping cart;
identifying, for each item of the plurality of items, one or more portions of the one or more images that contain the item; and
generating training data that comprises, for each item of the plurality of items, the one or more images that contain the item, the identified portions of the one or more images, and a label comprising information about the item contained in the one or more images.
2. The method of claim 1 , further comprising: training a machine-learned model for an automated checkout system using the generated training data.
3. The method of claim 1 , further comprising: receiving sensor data from one or more sensors attached to the shopping cart.
4. The method of claim 3 , wherein the one or more sensors comprise at least one of a pressure sensor, a proximity sensor, a motion sensor, a depth sensor, or an RFID sensor.
5. The method of claim 3 , wherein the generated training data further comprises the received sensor data.
6. The method of claim 3 , further comprising: detecting that the item associated with the item identifier was added to the shopping cart based on the received sensor data.
7. The method of claim 6 , wherein the one or more images are captured in response to detecting that the item was added to the shopping cart.
8. The method of claim 1 , wherein the received item identifier is received from an item scanner of the portable checkout unit.
9. The method of claim 1 , wherein each of the one or more portions comprises a bounding box.
10. The method of claim 9 , wherein each bounding box for the one or more portions is associated with an item identifier of the one or more item identifiers.
11. The method of claim 9 , wherein each bounding box for the one or more portions is generated based on previously-generated bounding boxes.
12. The method of claim 11 , wherein each bounding box for the one or more portions is paired with a previously-generated bounding box, and wherein each bounding box for the one or more portions is associated with an item identifier associated with the previously-generated bounding box associated with the bounding box.
13. A portable checkout unit comprising:
a processor; and
a computer-readable medium comprising instructions that, when executed by the processor, cause the processor to:
receive, at a portable checkout unit attached to a shopping cart, an item identifier identifying an item sold by a store, the shopping cart containing a plurality of items prior to receiving the item identifier;
update a virtual shopping list based on the received item identifier, the virtual shopping list comprising item identifiers for the plurality of items contained in the shopping cart, the item identifiers comprising the received item identifier based on the updating;
capture one or more images using one or more cameras attached to the shopping cart, the one or more cameras being directed towards the one or more items contained by the shopping cart;
detect that the item was added to the shopping cart;
responsive to detecting that the item was added to the shopping cart:
capture one or more images using one or more cameras attached to the shopping cart, the one or more cameras being directed towards the plurality of items contained by the shopping cart;
identify, for each item of the plurality of items, one or more portions of the one or more images that contain the item; and
generate training data that comprises, for each item of the plurality of items, the one or more images that contain the item, the identified portions of the one or more images, and a label comprising information about the item contained in the one or more images.
14. The portable checkout unit of claim 13 , wherein the computer-readable medium comprises instructions that cause the processor to: receive sensor data from one or more sensors attached to the shopping cart.
15. The portable checkout unit of claim 14 , wherein the one or more sensors comprise at least one of a pressure sensor, a proximity sensor, a motion sensor, a depth sensor, or an RFID sensor.
16. The portable checkout unit of claim 14 , wherein the computer-readable medium further comprises instructions that cause the processor to: detect that the item associated with the item identifier was added to the shopping cart based on the received sensor data.
17. The portable checkout unit of claim 13 , wherein the received item identifier is received from an item scanner of the portable checkout unit.
18. The portable checkout unit of claim 13 , wherein each of the one or more portions comprises a bounding box.
19. The portable checkout unit of claim 18 , wherein each bounding box for the one or more portions is associated with an item identifier of the one or more item identifiers.