IP Library › Granted Patent US 12,646,291
Granted Patent B2
US 12,646,291 · App. 17/849,341 · Granted Jun 2, 2026

Method for scaling fine-grained object recognition of consumer packaged goods

Inventors: Joel Iventosch (Austin, TX); James E. Dutton (Clinton, AR)
Assignee: Pensa Systems, Inc.
G06V10/7715G06V10/26G06V10/764G06V10/774G06V10/82
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,646,291
App. No.
17/849,341
Granted
Jun 2, 2026
Kind
B2
Abstract

A method is provided for assigning a classification to consumer packaged goods (CPGs). The method includes capturing an image of a plurality of CPGs arranged on a shelf; providing the captured image to a CPG detector; identifying all of the CPGs in the image; producing a set of cropped images, wherein each cropped image shows a single CPG as it appears in the image; and for each member of the set of cropped images, assigning a classification to the CPG in the member of the set of cropped images and establishing a confidence for the assigned classification through a process that includes the steps of (a) identifying a first set of reference images of CPGs whose classification is known, wherein each member of the first set of reference images is semantically similar to the member of the set of cropped images, and (b) identifying details in the member of the set of cropped images that differentiates it from a second set of reference images of CPGs whose classification is known.

Claims (47)

1 . A method for assigning a classification to consumer packaged goods (CPGs), comprising:

capturing an image of a plurality of CPGs arranged on a shelf;

providing the captured image to a CPG detector;

identifying all of the CPGs in the image;

producing a set of cropped images, wherein each cropped image shows a single CPG as it appears in the image; and

for each member of the set of cropped images, assigning a classification to the CPG in the member of the set of cropped images and establishing a confidence for the assigned classification through a process that includes the steps of

(a) identifying a first set of reference images of CPGs whose classification is known, wherein each member of the first set of reference images is semantically similar to the member of the set of cropped images, and

(b) identifying details in the member of the set of cropped images that differentiates it from a second set of reference images of CPGs whose classification is known;

wherein each of the CPGs in the first set of reference images is assigned to a superclass, and wherein assigning a classification to the CPG in the member of the set of cropped images includes (a) training a subclassifier to recognize differences between CPGs in the same superclass, thereby obtaining a trained subclassifier, and (b) using the trained subclassifier to recognize differences between CPGs in the same superclass.

2 . The method of claim 1 , wherein training a subclassifier to recognize differences between CPGs in the same superclass includes training the subclassifier on a first set of product images, wherein each product image in the first set of product images has a specified class identifier.

3 . The method of claim 2 , further comprising:

using an artificial neural network to extract semantic features from the first set of product images.

4 . The method of claim 3 , further comprising:

using the artificial neural network to compute a deep hash for each image in the first set of product images, thereby producing a hash set.

5 . The method of claim 4 , wherein the deep hash is a vector.

6 . The method of claim 5 , further comprising:

using the vector to compare CPGs in the first set of product images and to determine which CPGs in the first set of product images are semantically similar to each other.

7 . The method of claim 4 , wherein the vector is a binary vector.

8 . The method of claim 4 , further comprising:

using the hash set to train the subclassifier.

9 . The method of claim 4 , further comprising:

using the hash set and the first set of product images to train the subclassifier.

10 . The method of claim 4 , further comprising:

using a second set of product images which are distinct from the first set of product images to train the subclassifier.

11 . The method of claim 10 , further comprising:

using the trained classifier to contrast each cropped image with at least one image from the first set of product images which has the same deep hash as the cropped image, thereby identifying the class of the product in the cropped image.

12 . The method of claim 11 , wherein the class of the product is a UPC class.

13 . A system for assigning a classification to consumer packaged goods (CPGs), comprising:

an image capture device mounted on a mobile platform, said image capture device being adapted to capture images of CPGs arranged on a shelf;

a CPG detector which accepts images captured by the image capture device and which identifies CPGs in the captured images;

an image cropper which produces cropped images from the captured images such that each cropped image shows a single CPG as it appears in the image; and

a classifier which operates on each cropped image produced by the image cropper to assign a classification to the CPG in the cropped image and to establish a confidence for the assigned classification, wherein the classifier

(a) identifies a first set of reference images of CPGs whose classification is known, wherein each member of the first set of reference images is semantically similar to the member of the set of cropped images, and

(b) identifies details in the member of the set of cropped images that differentiates it from a second set of reference images of CPGs whose classification is known;

wherein each of the CPGs in the first set of reference images is assigned to a superclass, and further comprising a subclassifier which recognizes differences between CPGs in the same superclass.

14 . The system of claim 4 , further comprising: a trainer which trains the subclassifier on a first set of product images to recognize differences between CPGs in the same superclass, wherein each product image in the first set of product images has a specified class identifier.

15 . The system of claim 14 , further comprising:

an artificial neural network which extracts semantic features from the first set of product images.

16 . The system of claim 15 , wherein the artificial neural network computes a deep hash for each image in the first set of product images, thereby producing a hash set.

17 . The system of claim 16 , wherein the deep hash is a vector.

18 . The system of claim 17 , wherein the artificial neural network uses the vector to compare CPGs in the first set of product images and to determine which CPGs in the first set of product images are semantically similar to each other.

19 . The system of claim 16 , wherein the vector is a binary vector.

20 . The system of claim 16 , wherein the trainer uses the hash set to train the subclassifier.

21 . The system of claim 16 , wherein the trainer uses the hash set and the first set of product images to train the subclassifier.

22 . The system of claim 16 , wherein the trainer uses a second set of product images which are distinct from the first set of product images to train the subclassifier.

23 . The system of claim 22 , wherein the trained classifier identifies the class of the product in the cropped image by contrasts each cropped image with at least one image from the first set of product images which has the same deep hash as the cropped image.

24 . The system of claim 23 , wherein the class of the product is a UPC class.

Assignments (1)
SECURITY INTEREST Recorded Dec 11, 2025
From: PENSA SYSTEMS, INC.
To: LAGO EVERGREEN CREDIT
Reel/Frame 073192/0296 →
Continuity (2)
Provisional Application 63214414 · Jun 24, 2021
Related Publication 20220415029A1 · Dec 29, 2022
References Cited (54)
US 9569692B2 · Xie · 2017 [cited by examiner]
US 9848150B2 · Uchida · 2017 [cited by examiner]
US 10242036B2 · Kwon · 2019 [cited by examiner]
US 10467501B2 · N · 2019 [cited by examiner]
US 10664722B1 · Sharma · 2020 [cited by examiner]
US 10885395B2 · Iventosch et al. · 2021 [cited by applicant]
US 11106901B2 · He · 2021 [cited by examiner]
US 11314992B2 · Iventosch et al. · 2022 [cited by applicant]
US 11531838B2 · Siskind · 2022 [cited by examiner]
US 11544501B2 · Dong · 2023 [cited by examiner]
US 20190385106A1 · Iventosch et al. · 2019 [cited by applicant]
US 20220044298A1 · Oshinaike et al. · 2022 [cited by applicant]
Chen, Wei-Yu, et al. “A Closer Look At Few-Shot Classification.” ICLR, 2019, pp. 1-16. [cited by applicant]
Xu, Jiaming, et al. “Convolutional Neural Networks for Text Hashing.” Proceedings of the 24th International Joint Conference on Artificial Intelligence, 2015, pp. 1369-1375. [cited by applicant]
Lai, Hanjiang, et al. “Simultaneous Feature Learning and Hash Coding with Deep Neural Networks.” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Apr. 14, 2015, pp. 3270-3278., doi:10.1109/cvpr.20… [cited by applicant]
Cao, Yue, et al. “Correlation Hashing Network for Efficient Cross-Modal Retrieval.” Procedings of the British Machine Vision Conference 2017, Feb. 20, 2017, doi:10.5244/c.31.128. [cited by applicant]
Xian, Yongqin, et al. “Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly.” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, No. 9, Jan. 2019, pp. 2251-2265., doi:10.… [cited by applicant]
Wang, Xiaolong, et al. “Zero-Shot Recognition via Semantic Embeddings and Knowledge Graphs.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Apr. 8, 2018, doi:10.1109/cvpr.2018.00717. [cited by applicant]
Zhu, Pengkai, et al. “Generalized Zero-Shot Recognition Based on Visually Semantic Embedding.” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Apr. 9, 2019, doi:10.1109/cvpr.2019.00311. [cited by applicant]
Schonfeld, Edgar, et al. “Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders.” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Apr. 5, 2019, doi:10.1109/cvpr.2019.00… [cited by applicant]
Atzmon, Yuval, and Gal Chechik. “Adaptive Confidence Smoothing for Generalized Zero-Shot Learning.” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), May 13, 2019, doi:10.1109/cvpr.2019.01194. [cited by applicant]
Zhao, An, et al. “Domain-Invariant Projection Learning for Zero-Shot Recognition.” Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018. [cited by applicant]
Liu, Shichen, et al. “Generalized Zero-Shot Learning with Deep Calibration Network.” Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018. [cited by applicant]
Yu, Yunlong, et al. “Stacked Semantics-Guided Attention Model for Fine-Grained Zero-Shot Learning.” Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018. [cited by applicant]
Guo, Yuchen, et al. “Zero-Shot Recognition via Direct Classifier Learning with Transferred Samples and Pseudo Labels.” Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17), 2017. [cited by applicant]
Wang, Wenlin, et al. “Zero-Shot Learning via Class-Conditioned Deep Generative Models.” Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), 2018. [cited by applicant]
Cao, Yue et al. “Collective Deep Quantization for Efficient Cross-Modal Retrieval.” Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17), 2017. [cited by applicant]
Jiang, Qing-Yuan, and Wu-Jun Li. “Deep Cross-Modal Hashing.” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, doi:10.1109/cvpr.2017.348. [cited by applicant]
Chen, Binghui, and Weihong Deng. “Hybrid-Attention Based Decoupled Metric Learning for Zero-Shot Image Retrieval.” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, doi:10.1109/cvpr.2019.… [cited by applicant]
Lin, Kevin, et al. “Deep Learning of Binary Hash Codes for Fast Image Retrieval.” 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2015, doi:10.1109/cvprw.2015.7301269. [cited by applicant]
Cao, Yue, et al. “Deep Cauchy Hashing for Hamming Space Retrieval.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, doi:10.1109/cvpr.2018.00134. [cited by applicant]
Zhu, Han et al. “Deep Hashing Network for Efficient Similarity Retrieval.” Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16), 2016. [cited by applicant]
Cao, Yue et al. “Deep Quantization Network for Efficient Image Retrieval.” Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16), 2016. [cited by applicant]
Cao, Yue, et al. “Deep Visual-Semantic Quantization for Efficient Image Retrieval.” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, doi:10.1109/cvpr.2017.104. [cited by applicant]
Cao, Zhangjie, et al. “HashNet: Deep Learning to Hash by Continuation.” 2017 IEEE International Conference on Computer Vision (ICCV), 2017, doi:10.1109/iccv.2017.598. [cited by applicant]
Jiang, Huajie, et al. “Learning Class Prototypes via Structure Alignment for Zero-Shot Recognition.” Computer Vision—ECCV 2018 Lecture Notes in Computer Science, 2018, pp. 121-138., doi:10.1007/978-3-030-01249-6_8. [cited by applicant]
Li, Wu-Jun, Sheng Wang, and Wang-Cheng Kang. “Feature learning based deep supervised hashing with pairwise abels.” 2015, doi:arXiv:1511.03855. [cited by applicant]
Song, Jie, et al. “Selective Zero-Shot Classification with Augmented Attributes.” Computer Vision—ECCV 2018 Lecture Notes in Computer Science, 2018, pp. 474-490., doi:10.1007/978-3-030-01240-3_29. [cited by applicant]
Lee, Chung-Wei, et al. “Multi-Label Zero-Shot Learning with Structured Knowledge Graphs.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, doi:10.1109/cvpr.2018.00170. [cited by applicant]
Niu, Li, et al. “Webly Supervised Learning Meets Zero-Shot Learning: A Hybrid Approach for Fine-Grained Classification.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, doi:10.1109/cvpr.2018.0… [cited by applicant]
“One-Shot Learning.” Wikipedia, Wikimedia Foundation, Mar. 27, 2020, en.wikipedia.org/wiki/One-shot_learning. [cited by applicant]
Felix, Rafael, et al. “Multi-Modal Cycle-Consistent Generalized Zero-Shot Learning.” Computer Vision—ECCV 2018 Lecture Notes in Computer Science, 2018, pp. 21-37., doi:10.1007/978-3-030-01231-1_2. [cited by applicant]
Cao, Yue, et al. “Deep Visual-Semantic Hashing for Cross-Modal Retrieval.” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining—KDD 16, 2016, doi:10.1145/2939672.2939812. [cited by applicant]
Verma, Vinay Kumar, et al. “Generalized Zero-Shot Learning via Synthesized Examples.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, doi:10.1109/cvpr.2018.00450. [cited by applicant]
Sinha, Smita. “What Is Zero-Shot Learning?” Analytics India Magazine, Jun. 18, 2018, analyticsindiamag.com/what-is-zero-shot-learning/. [cited by applicant]
Xian, Yongqin, et al. “Feature Generating Networks for Zero-Shot Learning.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, doi:10.1109/cvpr.2018.00581. [cited by applicant]
Zhao, Fang, et al. “Deep Semantic Ranking Based Hashing for Multi-Label Image Retrieval.” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, doi:10.1109/cvpr.2015.7298763. [cited by applicant]
Grover D, Bauhoff S, Friedman J (2019) Using supervised learning to select audit targets in performance-based financing in health: An example from Zambia. PLoS One 14(1): e0211262. https://doi.org/10.1371/journal.pone.0… [cited by applicant]
Rich, Michael D.; Mills, Robert F.; Dube, Thomas E.; and Rogers, Steven K. (2016) “Evaluating Machine Learning Classifiers for Defensive Cyber Operations,” Military Cyber Affairs: vol. 2 : Iss. 1, Article 6. https://www… [cited by applicant]
Nishtha Hooda, Seema Bawa & Prashant Singh Rana (2020) Optimizing Fraudulent Firm Prediction Using Ensemble Machine Learning: A Case Study of an External Audit, Applied Artificial Intelligence, 34:1, 20-30, DOI: 10.1080… [cited by applicant]
Eid, FE., Elmarakeby, H.A., Chan, Y.A. et al. Systematic auditing is essential to debiasing machine learning in biology. Commun Biol 4, 183 (2021). https://doi.org/10.1038/s42003-021-01674-5. [cited by applicant]
Hooda, N., Bawa, S., & Rana, P.S. (2018). Fraudulent Firm Classification: A Case Study of an External Audit. Applied Artificial Intelligence, 32, 48-64. [cited by applicant]
Barredo-Arrieta, A., & Del Ser, J. (2020). Plausible Counterfactuals: Auditing Deep Learning Classifiers with Realistic Adversarial Examples. 2020 International Joint Conference on Neural Networks (IJCNN), 1-7. [cited by applicant]
San Diego Now Intelligence, published by ServiceNow (Jun. 2, 2022). [cited by applicant]