Systems and methods for building image databases to facilitate computer vision-based consumer product identification
Systems and methods for facilitating computer-vision-based detection and identification of consumer products comprise a support surface for supporting thereon one or more consumer products to be detected and a plurality of digital cameras each arranged at a different height relative to the support surface and positioned such that a field of view of each of the digital cameras includes the consumer product(s) located on the support surface. Movement of the support surface and/or the digital cameras permits the digital cameras to capture images of the consumer product from different angles of view and up to a full 360 degree of rotation around the at least one consumer product. A computing device uses at least the image data from such images to train a computer vision consumer product identification model to generate a reference image data model to be used for recognition of the at least one consumer product.
1. A system for facilitating computer-vision-based identification of consumer products, the system comprising:
a support surface for supporting thereon at least one consumer product to be identified;
at least one digital camera positioned such that:
a field of view of the at least one digital camera includes the at least one consumer product located on the support surface; and
the at least one camera is permitted to capture images of the at least one consumer product;
an electronic database configured to store electronic data including:
image data associated with the images of the at least one consumer product captured the at least one digital camera; and
identifier data associated with the at least one consumer product; and
a computing device in communication with the at least one digital camera and the electronic database, the computing device including a control circuit configured to:
obtain the image data and the identifier data from the electronic database; and
using the image data and the identifier data obtained by the computing device from the electronic database, train a computer vision consumer product identification model to generate a reference image data model to be used for recognition of the at least one consumer product when the at least one consumer product is located in a product storage unit.
2. The system of claim 1 , wherein the at least one digital camera is configured to move relative to the at least one consumer product located on the support surface to capture the images of the at least one consumer product from at least one angle.
3. The system of claim 2 , wherein the at least one digital camera is configured to capture from 1 to 10 frames per second.
4. The system of claim 1 , wherein the support surface is monochrome, and further comprising a monochrome background positioned around at least a portion of the support surface in the field of view of the at least one digital camera, wherein the monochrome support surface and the monochrome background increase capacity of the images captured by the at least one digital camera for background subtraction.
5. The system of claim 1 , further comprising at least one scanner including at least a first sensor configured to:
scan an identifier of the at least one consumer product; and
generate the identifier data for storage in the electronic database based on a scan of the identifier of the at least one consumer product by the scanner.
6. The system of claim 5 , wherein:
the at least one scanner further includes at least a second sensor configured to detect a fill level in an interior of the at least one consumer product and to generate fill level data for storage in the electronic database in association with the at least one consumer product;
the electronic database further stores reference fill level data indicating a fill level in the interior of the at least one consumer product when in an unconsumed state and when in a at least partially consumed state; and
the control circuit is programmed to correlate the fill level in the interior of the at least one consumer product detected by the at least one scanner to the reference fill level data to determine a level of consumption of the at least one consumer product relative to the fill level in the interior of the at least one consumer product when in the unconsumed state.
7. The system of claim 5 , wherein:
the at least one scanner includes at least a third sensor configured to detect a number of individual items in a pack of the at least one consumer product;
the electronic database further stores reference pack number data indicating a number of the individual items in the pack when the pack is in a fully stocked state; and
the control circuit is programmed to correlate the number of individual items in the pack of the at least one consumer product detected by the third sensor to the reference pack number data to determine how many of the individual items in the pack have been consumed relative to the number of the individual items in the pack when the pack is in a fully stocked state.
8. The system of claim 1 , wherein the control circuit of the computing device is further configured to:
obtain a first image of a plurality of consumer products located in the product storage unit captured by at least one digital camera located proximate the product storage unit;
detect each one of the plurality of individual consumer products in the first image; and
train a computer vision consumer product localization model to generate a reference bounding box model for drawing virtual boundary lines around each one of the individual consumer products located in the product storage unit.
9. The system of claim 8 , wherein the electronic database further stores the reference image data model and the reference bounding box model for the at least one consumer product in association with the identifier data associated with the at least one consumer product.
10. The system of claim 9 , wherein the computing device is further configured to:
obtain a second image of a consumer product located with other consumer products in the consumer product storage unit, the second image being captured by the at least one digital camera located proximate the product storage unit;
detect the individual consumer products located in the obtained image; and
correlate image data from the obtained second image to the reference image data model and the reference bounding box model to recognize an identity of each one of the individual consumer products detected in the obtained second image.
11. A method of facilitating computer-vision-based identification of consumer products, the method comprising:
providing a support surface for supporting thereon at least one consumer product to be identified;
providing at least one digital camera having a field of view that includes the at least one consumer product located;
capturing images of the at least one consumer product via the at least one digital camera;
providing an electronic database configured to store electronic data including:
image data associated with the images of the at least one consumer product captured by the at least one digital camera; and
identifier data associated with the at least one consumer product; and
providing a computing device in communication with the at least one digital camera and the electronic database, the computing device including a control circuit; and
by the control circuit:
obtaining the image data and the identifier data from the electronic database; and
using the image data and the consumer product identifier data obtained by the computing device from the electronic database, training a computer vision consumer product identification model to generate a reference image data model for recognition of the at least one consumer product when the at least one consumer product is located in a product storage unit.
12. The method of claim 11 , further comprising moving the at least one digital camera relative to the at least one consumer product to capture the images of the at least one consumer product from at least one angle.
13. The method of claim 12 , further comprising capturing from 1 to 10 frames per second by the at least one digital camera the at least one consumer product located in the product storage unit.
14. The method of claim 11 , wherein the support surface is monochrome, and further comprising providing a monochrome background positioned around at least a portion of the support surface in the field of view of the at least one digital camera, wherein the monochrome support surface and the monochrome background increase capacity of the images captured by the at least one digital camera for background subtraction.
15. The method of claim 11 , further comprising:
providing at least one scanner including at least a first sensor;
scanning an identifier of the at least one consumer product via the at least one scanner; and
generating the identifier data for storage in the electronic database based on the scanning of the identifier of the at least one consumer product by the scanner.
16. The method of claim 15 , further comprising:
providing the at least one scanner with at least a second sensor;
via the at least one scanner, detecting a fill level in an interior of the at least one consumer product, and generating fill level data for storage in the electronic database in association with the at least one consumer product;
storing in the electronic database further reference fill level data indicating a fill level in the interior of the at least one consumer product when in an unconsumed state and when in a at least partially consumed state; and
by the control circuit, correlating the fill level in the interior of the at least one consumer product detected by the at least one scanner to the reference fill level data to determine a level of consumption of the at least one consumer product relative to the fill level in the interior of the at least one consumer product when in the unconsumed state.
17. The method of claim 15 , further comprising:
providing the at least one scanner with at least a third sensor:
detecting, via the at least one scanner, a number of individual items in a pack of the at least one consumer product;
storing in the electronic database reference pack number data indicating a number of the individual items in the pack when the pack is in a fully stocked state; and
by the control circuit, correlating the number of individual items in the pack of the at least one consumer product detected by the third sensor to the reference pack number data to determine how many of the individual items in the pack have been consumed relative to the number of the individual items in the pack when the pack is in a fully stocked state.
18. The method of claim 11 , further comprising, by the computing device:
obtaining a first image of a plurality of consumer products located in the product storage unit captured by at least one digital camera located proximate the product storage unit;
detecting each one of the plurality of individual consumer products in the first image; and
training a computer vision consumer product localization model to generate a reference bounding box model for drawing virtual boundary lines around each one of the individual consumer products located in the product storage unit.
19. The method of claim 18 , further comprising storing in the electronic database the reference image data model and the reference bounding box model for the at least one consumer product in association with the identifier data associated with the at least one consumer product.
20. The method of claim 19 , further comprising, by the computing device:
obtaining a second image of a consumer product located with other consumer products in the consumer product storage unit, the second image being captured by the at least one digital camera located proximate the product storage unit;
detecting the individual consumer products located in the obtained image; and
correlating image data from the obtained second image to the reference image data model and the reference bounding box model to recognize an identity of each one of the individual consumer products detected in the obtained second image.