System for detecting and classifying consumer packaged goods
A system is provided for identifying consumer packaged goods (CPGs). The system comprises an imaging device mounted on a mobile platform; a CPG detector which is equipped with a neural network and which (a) utilizes the imaging device to obtain an image containing a plurality of CPGs, (b) detects a set of CPG superclass features in the image, and (c) generates a CPG feature map which maps the location of CPG features in the image, wherein said neural network has been trained on a set of images of CPGs to recognize which of a set of superclasses a detected CPG belongs to; a region proposal network which accepts the CPG feature map as input and which returns a first set of regions in the image where a CPG could be located; a superclassifier which examines each region in the first set of regions to determine if the region contains an instance of a superclass of a CPG, and which outputs a second set of regions containing a CPG; and a superclass proposal layer which operates on each member of the second set of regions and returns a set of superclass proposals for each region.
1. A system for identifying consumer packaged goods (CPGs), comprising: an imaging device mounted on a mobile platform:
a CPG detector which is equipped with a neural network and which
(a) utilizes the imaging device to obtain an image containing a plurality of CPGs,
(b) detects a set of CPG superclass features in the image, and
(c) generates a CPG feature map which maps the location of CPG features in the image,
wherein said neural network has been trained on a set of images of CPGs to recognize which of a set of superclasses a detected CPG belongs to;
a region proposal network which accepts the CPG feature map as input and which returns a first set of regions in the image where a CPG could be located;
a superclassifier which examines each region in the first set of regions to determine if the region contains an instance of a superclass of a CPG, and which outputs a second set of regions containing a CPG; and
a superclass proposal layer which operates on each member of the second set of regions and returns a set of superclass proposals for each region;
wherein said imaging device is mounted on a drone;
wherein said region proposal network, said superclassifier, and said superclass proposal layer are defined in an operating system running in a tangible, non-transient memory medium installed on said drone.
2. The system of claim 1 , wherein said neural network is a deep convolutional neural network (CNN).
3. The system of claim 2 , wherein said CNN is equipped with a convolutional layer having features recognized in the image.
4. The system of claim 1 , wherein said proposal network finds all possible locations in the image where a CPG could be located.
5. The system of claim 1 , wherein the set of regions is combined using a region of interest pooling (RoI) technique.
6. The system of claim 5 , wherein said RoI technique uses a neural-net layer of the neural network for object detection.
7. The system of claim 1 , wherein the superclassifier implements a Faster R-CNN technique.
8. The system of claim 1 , wherein said superclass proposal layer operates on each member of the second set of regions and returns a plurality of superclass proposals for each region.
9. The system of claim 8 , further comprising:
a plurality of subclassifiers which input the superclass proposals and output CPGs which fall within a superclass category.
10. The system of claim 9 , wherein each superclass has a sub-classifier associated with it which identifies CPGs that fall within that superclass.
11. The system of claim 10 , wherein the set of superclass proposals contains subsets B=[B 1 , . . . , B k ], wherein k≥1, wherein each subset B n is a set of superclass proposals which identifies superclass S n as the corresponding superclass, wherein subclassifier C n identifies CPG classes that fall within superclass S n , and wherein subclassifier C n operates on subset B n to return a CPG class which corresponds to each proposal in subset B n .
12. The system of claim 10 , wherein the set of superclass proposals contains subsets B=[B 1 , . . . , B k ], wherein k≥1, wherein each subset B n is a set of superclass proposals which identifies superclass S n as the corresponding superclass, wherein subclassifier C n identifies CPG classes that fall within superclass S n , wherein subclassifier C n operates on subset B n to return a CPG class which corresponds to each proposal in subset B n and a confidence score, and wherein the confidence score reflects the certainty with which the CPG was identified.
13. The system of claim 12 , wherein the CPG class returned by subclassifier C n maps to a single SKU or UPC.
14. The system of claim 1 , wherein said superclass proposal layer operates on each member of the second set of regions with a normalized exponential function to generate the set of superclass proposals for each region.
15. The system of claim 1 , wherein each superclass proposal in the set of superclass proposals is a trinomial variable whose values are CPG superclass, confidence score and bounding box.
16. The system of claim 1 , wherein said region proposal network, said superclassifier, and said superclass proposal layer are defined in an operating system running in a tangible, non-transient memory medium installed in a computer system that is in communication with said drone.