IP Library › Granted Patent US 12,657,876
Granted Patent B2
US 12,657,876 · App. 17/840,586 · Granted Jun 16, 2026

AI-assisted human data augmentation and continuous training for machine learning models

Inventors: Joel Iventosch (Austin, TX); Michael Pav (St. Petersburg, FL); Bora Yavuz (Istanbul, TR); Pinar Kaprali (Istanbul, TR); James E. Dutton (Spicewood, TX)
Assignee: Pensa Systems, Inc.
G06V10/765G06V10/776G06V10/7788G06V10/82
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,657,876
App. No.
17/840,586
Granted
Jun 16, 2026
Kind
B2
Abstract

A method is provided for training at least one classifier model used by an artificial intelligence (AI) system to recognize each of a set of objects and to assign each of the set of objects to a class. The method includes training the at least one classifier model on a training dataset, thereby producing at least one trained classifier model; using the at least one trained classifier model to detect and classify each member of a set of objects, thereby generating a set of inferences, wherein each inference includes (a) a cropped image of a classified object, (b) the classified object's inferred class, and (c) a confidence score associated with the inferred classification; examining the set of inferences with a machine implemented audit trigger, wherein the audit trigger identifies a subset of the set of inferences whose members have (i) a confidence score that falls below a predetermined threshold value, or (ii) a missing classification; and if the identified subset has at least one member, subjecting the identified subset to a human audit, thereby yielding a corrected set of observations, wherein, for each member of the corrected set of observations, the inferred class of the corresponding member of the set of inferences is replaced with a corrected class. The corrected set of observations is then added to a training dataset and used to improve the future accuracy of the classifier model.

Claims (52)

1 . A method for training at least one classifier model used by an artificial intelligence (AI) system to recognize each of a set of objects and to assign each of the set of objects to a class, the method comprising:

training the at least one classifier model on a training dataset, thereby producing at least one trained classifier model;

using the at least one trained classifier model to detect and classify each member of a set of objects, thereby generating a set of inferences, wherein each inference includes

(a) a cropped image of a classified object,

(b) the classified object's inferred classification, and

(c) a confidence score associated with the inferred classification;

examining the set of inferences with a machine implemented audit trigger, wherein the audit trigger identifies a subset of the set of inferences whose members have (i) a confidence score that falls below a predetermined threshold value, or (ii) a missing inferred classification;

if the identified subset has at least one member, subjecting the identified subset to a human audit, thereby yielding a corrected set of observations, wherein, for each member of the corrected set of observations, the inferred classification of the corresponding member of the set of inferences is replaced with a corrected classification;

examining the corrected set of observations with a benefit scoring process, thereby identifying at least one proposed modification to the training dataset; and

incorporating the at least one proposed modification into the training dataset in a subsequent iteration of the method.

2 . The method of claim 1 , wherein each member of the set of objects is a consumer packaged good (CPG).

3 . The method of claim 1 , wherein the benefit scoring process excludes from the at least one proposed modification observations associated with an image that is blurry or occluded.

4 . The method of claim 1 , wherein the benefit scoring process implements a deep learning model that learns over time which observations to include in the training dataset utilized in a subsequent iteration of the method.

5 . The method of claim 1 , wherein the benefit scoring process implements an algorithmic process that examines characteristics selected from the group consisting of image quality and class accuracy.

6 . The method of claim 1 , wherein the benefit scoring process implements a human augmentation model that includes human participation in the decision.

7 . The method of claim 1 , wherein the benefit scoring process selects corrected observations for inclusion in the training dataset utilized in a subsequent iteration of the method using at least one selection criteria selected from the group consisting of (a) image quality, and (b) the number of training images already in the training dataset which correspond to the product identified in the corrected observation.

8 . The method of claim 1 , wherein the audit trigger identifies all members of the set of inferences for which the confidence score falls below a predetermined threshold value.

9 . The method of claim 1 , wherein the audit trigger identifies all members of the set of inferences which are missing a classification.

10 . The method of claim 1 , wherein the audit trigger identifies all members of the set of inferences for which the classifier model cannot identify the class to which the object belongs.

11 . The method of claim 1 , wherein the audit trigger identifies all members of the set of inferences for which the difference between the confidence scores of the top two inferences for the same object is below a predetermined threshold value.

12 . The method of claim 1 , wherein the audit trigger identifies all members of the set of inferences whose members have a confidence score that falls below a predetermined threshold value.

13 . The method of claim 1 , wherein the subset of the set of inferences is one of a set of commonly confused classes due to fine-grained differences in packaging.

14 . The method of claim 1 , further comprising:

publishing a set of observations to a group of data consumers, wherein the published set of observations includes (a) members of the set of inferences exclusive of the subset of the set of inferences, and (b) the corrected set of observations.

15 . The method of claim 1 , wherein the at least one classifier model is a set of cooperating models which identify and qualify distinct aspects of images to be classified.

16 . The method of claim 15 , wherein the at least one classifier model is a sequence of models that perform the steps of:

detecting and cropping object images within larger frames or videos;

identifying similarities of the detected object images to existing classes; and

differentiating between the detected object images and other similar classes of objects.

17 . The method of claim 16 , wherein examining the corrected set of observations with a benefit scoring process includes examining the corrected set of observations with a plurality of benefit scoring processes.

18 . The method of claim 17 , wherein incorporating the at least one proposed modification into the training dataset in a subsequent iteration of the method results in the generation of a plurality of training datasets, and wherein each of the plurality of training datasets is used to train the classifier model based on a distinct set of criteria.

19 . The method of claim 1 , wherein the audit trigger is an algorithmic process.

20 . The method of claim 1 , wherein the audit trigger is a rules-based process.

21 . The method of claim 1 , wherein the audit trigger implements a deep-learning approach to adjust its selection criteria based on results achieved in prior iterations of the method.

22 . The method of claim 21 wherein, if at least one of said prior iterations of the method produces a subset containing at least k members for which the inference was correct, and if k>m>0, wherein k and m are integers and m is a predetermined threshold value, then the audit trigger increases the predetermined threshold value for confidence scores in at least one subsequent iteration of the method.

23 . The method of claim 1 , further comprising:

using a proportion of corrected inferences resulting from the human audit process to compute recognition accuracy metrics.

24 . The method of claim 23 , further comprising:

using the computed recognition accuracy metrics to monitor the performance of the at least one classifier model.

25 . The method of claim 23 , further comprising:

using the computed recognition accuracy metrics to train at least one component of the AI system.

26 . The method of claim 25 , wherein the computed accuracy metrics include metrics selected from the group consisting of product categories, geographic areas, and overall summary metrics.

27 . A tangible, non-transient medium containing suitable programming instructions which, when processed by at least one computer processor, perform the method of claim 1 .

28 . A method for training at least one classifier model used by an artificial intelligence (AI) system to recognize each of a set of objects and to assign each of the set of objects to a class, the method comprising:

training the at least one classifier model on a training dataset, thereby producing at least one trained classifier model;

using the at least one trained classifier model to detect and classify each member of a set of objects, thereby generating a set of inferences, wherein each inference includes

(a) a cropped image of a classified object,

(b) the classified object's inferred classification, and

(c) a confidence score associated with the inferred classification;

examining the set of inferences with a machine implemented audit trigger, wherein the audit trigger identifies a subset of the set of inferences whose members have (i) a confidence score that falls below a predetermined threshold value, or (ii) a missing inferred classification; and

if the identified subset has at least one member, subjecting the identified subset to a human audit, thereby yielding a corrected set of observations, wherein, for each member of the corrected set of observations, the inferred classification of the corresponding member of the set of inferences is replaced with a corrected classification;

wherein the audit trigger identifies all members of the set of inferences for which the difference between the confidence scores of the top two inferences for the same object is below a predetermined threshold value.

Assignments (1)
SECURITY INTEREST Recorded Dec 11, 2025
From: PENSA SYSTEMS, INC.
To: LAGO EVERGREEN CREDIT
Reel/Frame 073192/0296 →
Continuity (2)
Provisional Application 63210374 · Jun 14, 2021
Related Publication 20220398829A1 · Dec 15, 2022
References Cited (64)
US 8620078B1 · Chapleau · 2013 [cited by examiner]
US 8706655B1 · Rangarajan · 2014 [cited by examiner]
US 10885395B2 · Iventosch et al. · 2021 [cited by applicant]
US 11314992B2 · Iventosch et al. · 2022 [cited by applicant]
US 11556746B1 · Dasgupta · 2023 [cited by examiner]
US 11636602B1 · Havír · 2023 [cited by examiner]
US 20030147558A1 · Loui · 2003 [cited by examiner]
US 20110170769A1 · Sakimura · 2011 [cited by examiner]
US 20140040173A1 · Sagher · 2014 [cited by examiner]
US 20140198979A1 · Hamarneh · 2014 [cited by examiner]
US 20170255891A1 · Morate · 2017 [cited by examiner]
US 20190080207A1 · Chang · 2019 [cited by examiner]
US 20190385106A1 · Iventosch et al. · 2019 [cited by applicant]
US 20200202257A1 · Lee · 2020 [cited by examiner]
US 20200382527A1 · Mitelman · 2020 [cited by examiner]
US 20220044298A1 · Oshinaike et al. · 2022 [cited by applicant]
US 20220398829A1 · Iventosch · 2022 [cited by examiner]
US 20220405605A1 · Yokoyama · 2022 [cited by examiner]
US 20220415029A1 · Iventosch · 2022 [cited by examiner]
US 20230222779A1 · Hilton · 2023 [cited by examiner]
US 20240177446A1 · Kudo · 2024 [cited by examiner]
US 20240355580A1 · Yim · 2024 [cited by examiner]
US 20240362936A1 · Shen · 2024 [cited by examiner]
Chen, Wei-Yu, et al. “A Closer Look at Few-Shot Classification.” ICLR, 2019, pp. 1-16. [cited by applicant]
Xu, Jiaming, et al. “Convolutional Neural Networks for Text Hashing.” Proceedings of the 24th International Joint Conference on Artificial Intelligence, 2015, pp. 1369-1375. [cited by applicant]
Lai, Hanjiang, et al. “Simultaneous Feature Learning and Hash Coding with Deep Neural Networks.” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Apr. 14, 2015, pp. 3270-3278., doi: 10.1109/cvpr.2… [cited by applicant]
Cao, Yue, et al. “Correlation Hashing Network for Efficient Cross-Modal Retrieval.” Procedings of the British Machine Vision Conference 2017, Feb. 20, 2017, doi:10.5244/c.31.128. [cited by applicant]
Xian, Yongqin, et al. “Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly.” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, No. 9, Jan. 2019, pp. 2251-2265., doi:10.… [cited by applicant]
Wang, Xiaolong, et al. “Zero-Shot Recognition via Semantic Embeddings and Knowledge Graphs.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Apr. 8, 2018, doi:10.1109/cvpr.2018.00717. [cited by applicant]
Zhu, Pengkai, et al. “Generalized Zero-Shot Recognition Based on Visually Semantic Embedding.” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Apr. 9, 2019, doi:10.1109/cvpr.2019.00311. [cited by applicant]
Atzmon, Yuval, and Gal Chechik. “Adaptive Confidence Smoothing for Generalized Zero-Shot Learning.” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), May 13, 2019, doi:10.1109/cvpr.2019.01194. [cited by applicant]
Zhao, An, et al. “Domain-Invariant Projection Learning for Zero-Shot Recognition.” Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018. [cited by applicant]
Liu, Shichen, et al. “Generalized Zero-Shot Learning with Deep Calibration Network.” Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018. [cited by applicant]
Yu, Yunlong, et al. “Stacked Semantics-Guided Attention Model for Fine-Grained Zero-Shot Learning.” Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018. [cited by applicant]
Guo, Yuchen, et al. “Zero-Shot Recognition via Direct Classifier Learning with Transferred Samples and Pseudo Labels.” Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17), 2017. [cited by applicant]
Wang, Wenlin, et al. “Zero-Shot Learning via Class-Conditioned Deep Generative Models.” Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), 2018. [cited by applicant]
Cao, Yue et al. “Collective Deep Quantization for Efficient Cross-Modal Retrieval.” Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17), 2017. [cited by applicant]
Jiang, Qing-Yuan, and Wu-Jun Li. “Deep Cross-Modal Hashing.” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, doi:10.1109/cvpr.2017.348. [cited by applicant]
Chen, Binghui, and Weihong Deng. “Hybrid-Attention Based Decoupled Metric Learning for Zero-Shot Image Retrieval.” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, doi:10.1109/cvpr.2019.… [cited by applicant]
Lin, Kevin, et al. “Deep Learning of Binary Hash Codes for Fast Image Retrieval.” 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2015, doi:10.1109/cvprw.2015.7301269. [cited by applicant]
Cao, Yue, et al. “Deep Cauchy Hashing for Hamming Space Retrieval.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, doi:10.1109/cvpr.2018.00134. [cited by applicant]
Zhu, Han et al. “Deep Hashing Network for Efficient Similarity Retrieval.” Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16), 2016. [cited by applicant]
Cao, Yue et al. “Deep Quantization Network for Efficient Image Retrieval.” Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16), 2016. [cited by applicant]
Cao, Yue, et al. “Deep Visual-Semantic Quantization for Efficient Image Retrieval.” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, doi:10.1109/cvpr.2017.104. [cited by applicant]
Cao, Zhangjie, et al. “HashNet: Deep Learning to Hash by Continuation.” 2017 IEEE International Conference on Computer Vision (ICCV), 2017, doi:10.1109/iccv.2017.598. [cited by applicant]
Jiang, Huajie, et al. “Learning Class Prototypes via Structure Alignment for Zero-Shot Recognition.” Computer Vision—ECCV 2018 Lecture Notes in Computer Science, 2018, pp. 121-138., doi:10.1007/978-3-030-01249-6_8. [cited by applicant]
Li, Wu-Jun, Sheng Wang, and Wang-Cheng Kang. “Feature learning based deep supervised hashing with pairwise labels.” 2015, doi:arXiv:1511.03855. [cited by applicant]
Song, Jie, et al. “Selective Zero-Shot Classification with Augmented Attributes.” Computer Vision—ECCV 2018 Lecture Notes in Computer Science, 2018, pp. 474-490., doi:10.1007/978-3-030-01240-3_29. [cited by applicant]
Lee, Chung-Wei, et al. “Multi-Label Zero-Shot Learning with Structured Knowledge Graphs.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, doi:10.1109/cvpr.2018.00170. [cited by applicant]
Niu, Li, et al. “Webly Supervised Learning Meets Zero-Shot Learning: A Hybrid Approach for Fine-Grained Classification.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, doi:10.1109/cvpr.2018.0… [cited by applicant]
“One-Shot Learning.” Wikipedia, Wikimedia Foundation, Mar. 27, 2020, en.wikipedia.org/wiki/One-shot_learning. [cited by applicant]
Felix, Rafael, et al. “Multi-Modal Cycle-Consistent Generalized Zero-Shot Learning.” Computer Vision—ECCV 2018 Lecture Notes in Computer Science, 2018, pp. 21-37., doi:10.1007/978-3-030-01231-1_2. [cited by applicant]
Cao, Yue, et al. “Deep Visual-Semantic Hashing for Cross-Modal Retrieval.” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining—KDD 16, 2016, doi:10.1145/2939672.2939812. [cited by applicant]
Verma, Vinay Kumar, et al. “Generalized Zero-Shot Learning via Synthesized Examples.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, doi:10.1109/cvpr.2018.00450. [cited by applicant]
Sinha, Smita. “What is Zero-Shot Learning?” Analytics India Magazine, Jun. 18, 2018, analyticsindiamag.com/what-is-zero-shot-learning/. [cited by applicant]
Xian, Yongqin, et al. “Feature Generating Networks for Zero-Shot Learning.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, doi:10.1109/cvpr.2018.00581. [cited by applicant]
Zhao, Fang, et al. “Deep Semantic Ranking Based Hashing for Multi-Label Image Retrieval.” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, doi:10.1109/cvpr.2015.7298763. [cited by applicant]
Grover D, Bauhoff S, Friedman J (2019) Using supervised learning to select audit targets in performance-based financing in health: An example from Zambia. PLoS One 14(1): e0211262. https://doi.org/10.1371/journal.pone.0… [cited by applicant]
Rich, Michael D.; Mills, Robert F.; Dube, Thomas E.; and Rogers, Steven K. (2016) “Evaluating Machine Learning Classifiers for Defensive Cyber Operations,” Military Cyber Affairs: vol. 2 : Iss. 1 , Article 6. https://ww… [cited by applicant]
Nishtha Hooda, Seema Bawa & Prashant Singh Rana (2020) Optimizing Fraudulent Firm Prediction Using Ensemble Machine Learning: A Case Study of an External Audit, Applied Artificial Intelligence, 34:1, 20-30, DOI: 10.1080… [cited by applicant]
Eid, FE., Elmarakeby, H.A., Chan, Y.A. et al. Systematic auditing is essential to debiasing machine learning in biology. Commun Biol 4, 183 (2021). https://doi.org/10.1038/s42003-021-01674-5. [cited by applicant]
Hooda, N., Bawa, S., & Rana, P.S. (2018). Fraudulent Firm Classification: A Case Study of an External Audit. Applied Artificial Intelligence, 32, 48-64. [cited by applicant]
Barredo-Arrieta, A., & Del Ser, J. (2020). Plausible Counterfactuals: Auditing Deep Learning Classifiers with Realistic Adversarial Examples. 2020 International Joint Conference on Neural Networks (IJCNN), 1-7. [cited by applicant]
San Diego Now Intelligence, published by ServiceNow (Jun. 2, 2022). [cited by applicant]