System and method for locating, identifying and counting items
A system for building a product library without requiring a planogram includes an image capture unit operated to provide images of items. Taking data from a shelf label detector and depth map creation unit, a processing module can be used to compare detected shelf labels to a depth map, define a product bounding box, and associate the bounding box with an image provided by the image capture unit to build image descriptors. The system can include one or more autonomous robots for supporting and moving the image capture unit and other components.
1. A method comprising:
capturing one or more images of a portion of a shelf with a camera;
detecting and reading content of a shelf label on the portion of the shelf within the one or more images;
defining one or more product facing bounding boxes surrounding one or more corresponding facings of a product within the one or more images;
associating the shelf label content with each of the one or more product facing bounding boxes that includes and defines each of the one or more corresponding product facings;
cross-correlating the one or more product facing bounding boxes and one or more other product facing bounding boxes across one or more stores by matching the shelf label content with other shelf label content associated with the one or more other product facing bounding boxes;
building training data based on the association of the shelf label content with the cross-correlated product facing bounding boxes;
using the training data to build a product classifier; and
using the product classifier to classify an item as the product.
2. The method of claim 1 , wherein capturing one or more images comprises capturing the one or more images with a camera attached to a moveable robot.
3. The method of claim 1 , further comprising using manual input to identify potential inventory designated in the one or more product facing bounding boxes.
4. The method of claim 1 , wherein using the training data to build a product classifier comprises integrating training to: (1) learn product image descriptors and (2) classify items into a neural network.
5. The method of claim 1 , wherein associating the shelf label content with each of the one or more product facing bounding boxes comprises manually associating the shelf label content with each of the one or more product facing bounding boxes.
6. The method of claim 1 , wherein associating the shelf label content with each of the one or more product facing bounding boxes comprises automatically associating the shelf label content with each of the one or more product facing bounding boxes.
7. The method of claim 1 , wherein capturing one or more images of a portion of a shelf comprises capturing one or more images of a portion of a shelf in a first store; and
further comprising:
capturing one or more further images of portion of another shelf in a second store;
reading content of another shelf label on the portion of the other shelf within the one or more further images;
defining one or more other product facing bounding boxes surrounding one or more other corresponding facings of another product within the one or more further images; and
associating the other shelf label content with each of the one or more other product facing bounding boxes that includes and defines each of the one or more other corresponding product facings;
wherein cross-correlating comprises cross-correlating the one or more product facing bounding boxes and the one or more other product facing bounding boxes across the first store and the second store by matching the shelf label content with the other shelf label content; and
wherein building training data comprises building the training data based on the association of the shelf label content and the other shelf label content with the cross-correlated product facing bounding boxes correlated across the first store and the second store.
8. The method of claim 7 , wherein capturing one or more further images comprises capturing the one or more further images subsequent to capturing the one or more images.
9. The method of claim 7 , wherein capturing one or more further images comprises capturing the one or more further images prior to capturing the one or more images.
10. A method comprising:
capturing one or more images of a portion of a shelf with a camera;
detecting and reading content of a shelf label on the portion of the shelf within the one or more images;
defining one or more product facing bounding boxes surrounding one or more facings of a product within the one or more images;
associating the shelf label content with each of the one or more product facing bounding boxes that includes and defines each of the one or more corresponding product facings;
cross-correlating the one or more product facing bounding boxes and one or more other product facing bounding boxes across one or more stores by matching the shelf label content with other shelf label content associated with the one or more other product facing bounding boxes;
building training data based on the association of the shelf label content with the cross-correlated product facing bounding boxes;
using the training data to build a product classifier; and
using the product classifier to perform an inventory management activity related to the product.
11. The method of claim 10 , wherein capturing one or more images of a portion of a shelf comprises capturing one or more images of a portion of a shelf in a first store; and
further comprising:
capturing one or more further images of a portion of another shelf in a second store;
reading content of another shelf label on the portion of the other shelf within the one or more further images;
defining one or more other product facing bounding boxes surrounding one or more other corresponding facings of another product within the one or more further images; and
associating the other shelf label content with each of the one or more other product facing bounding boxes that includes and defines each of the one or more other corresponding product facings;
wherein cross-correlating comprises cross-correlating the one or more product facing bounding boxes and the one or more other product facing bounding boxes across the first store and the second store by matching the shelf label content with the other shelf label content; and
wherein building a training data comprises building the training data set based on the association of the shelf label content and the other shelf label content with the cross-correlated product facing bounding boxes correlated across the first store and the second store.
12. The method of claim 11 , wherein capturing one or more further images comprises capturing the one or more further subsequent to capturing the one or more images.
13. The method of claim 11 , wherein capturing one or more further images comprises capturing the one or more further images prior to capturing the one or more images.
14. The method of claim 10 , wherein capturing one or more images of a portion of a shelf with a camera comprises capturing the one or more images with a camera attached to a moveable robot.
15. The method of claim 10 , further comprising using manual input to identify potential inventory designated in the one or more bounding boxes.
16. The method of claim 10 , wherein the one or more bounding boxes surround gaps between potential inventory.
17. The method of claim 10 , wherein associating the shelf label content with each of the one or more product facing bounding boxes comprises automatically associating the shelf label content with each of the one or more product facing bounding boxes.
18. A system comprising:
a processor;
system memory coupled to the processor and storing instructions configured to cause the processor to:
capture one or more images of a portion of a shelf with a camera;
detect and reading content of a shelf label on the portion of the shelf within the one or more images;
define one or more product facing bounding boxes that surround an surrounding one or more corresponding facings of a product within the one or more images;
associate the shelf label content with each of the one or more product facing bounding boxes that includes and defines each of the one or more corresponding product facings;
cross-correlate the one or more product facing bounding boxes and one or more other product facing bounding boxes across one or more stores by matching the shelf label content with other shelf label content associated with the one or more other product facing bounding boxes;
build training data based on the association of the shelf label content with the cross-correlated product facing bounding boxes;
use the training data to build a product classifier; and
use the product classifier to perform one or more of: (1) an inventory management activity related to the product or (2) classifying an item as the product.
19. The system of claim 18 , wherein instructions configured to capture one or more images of a portion of a shelf comprise instructions configured to capture one or more images of a portion of a shelf in a first store; and
further comprising instructions configured to:
capture one or more further images of a portion of another shelf in a second store;
read content of another shelf label on a portion of the other shelf within the one or more further images;
define one or more other product facing bounding boxes surrounding one or more other corresponding facings of another product within the one or more further images; and
associate the other shelf label content with each of the one or more other product facing bounding boxes that includes and defines each of the one or more other corresponding product facings;
wherein instructions configured to cross-correlate the one or more product facing bounding boxes and the one or more other product facing bounding boxes across the first store and the second store by matching the shelf label content with the other shelf label content; and
wherein instructions configured to build a training data comprising building the training data sent based on the association of the shelf label content and the other shelf label content with the cross-correlated product facing bounding boxes correlated across the first store and the second store.
20. The system of claim 18 , wherein instructions configured to capture one or more further images comprise instructions configured to capture the one or more further images subsequent to capturing the one or more images.
21. The system of claim 18 , wherein instructions configured to capture one or more further images of another portion of a shelf comprise instructions configured to capture the one or more further images.