IP Library › Granted Patent US 12,380,368
Granted Patent B2
US 12,380,368 · App. 18/343,444 · Granted Aug 5, 2025

Machine learning model error detection

Inventors: Zhe Liu (San Jose, CA); Yufan Guo (San Jose, CA); Jalal Mahmud (San Jose, CA); Rama Kalyani T. Akkiraju (Cupertino, CA)
Assignee: International Business Machines Corporation
G06N20/00G06N5/02
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,380,368
App. No.
18/343,444
Granted
Aug 5, 2025
Kind
B2
Abstract

A system includes a memory having instructions therein and at least one processor in communication with the memory. The at least one processor is configured to execute the instructions to determine a global-level importance magnitude value for a global-level importance of an explainable feature of a machine learning base model based on a first prediction of the machine learning base model. The at least one processor is also configured to execute the instructions to determine a global-level importance direction label for the global-level importance of the explainable feature based on the first prediction. The at least one processor is also configured to execute the instructions to generate a communication for presentation to a user based on a second prediction of the machine learning base model, based on the global-level importance magnitude value, and based on the global-level importance direction label.

Claims (40)

1. A method for correcting an erroneous prediction of a machine learning base model for a user, the method comprising:

determining a plurality of global-level importance magnitude values representing a global-level importance of a plurality of explainable features of the machine learning base model to the machine learning base model;

determining a plurality of global-level importance direction labels representing the global-level importance of the explainable features of the machine learning base model to the machine learning base model;

ranking the explainable features according to their associated global level importance magnitudes;

generating a communication for presentation to the user comprising one or more highest-ranked explainable features of the ranked explainable features, and the global-level importance direction labels associated with the highest-ranked explainable features; and

calculating a local erroneousness score for one or more new predictions by the machine learning base model based on one or more erroneous global-level importance direction labels associated with the one or more highest-ranked explainable features.

2. The method of claim 1 , further comprising receiving an erroneousness assessment identifying the one or more erroneous global-level importance labels.

3. The method of claim 1 , further comprising correcting the one or more new predictions or revising the machine learning base model based on the local erroneousness score.

4. The method of claim 1 , wherein receiving the erroneousness assessment of the global-level importance direction label comprises receiving a human erroneousness assessment of the global-level importance direction label.

5. A system for correcting an erroneous prediction of a machine learning base model for a user, the system comprising:

a memory having instructions therein, and at least one processor in communication with the memory, wherein the at least one processor is configured to execute the instructions to:

determine a plurality of global-level importance magnitude values representing a global-level importance of a plurality of explainable features of the machine learning base model to the machine learning base model;

determine a plurality of global-level importance direction labels representing the global-level importance of the explainable features of the machine learning base model to the machine learning base model;

rank the explainable features according to their associated global level importance magnitudes;

generate a communication for presentation to the user comprising one or more highest-ranked explainable features of the ranked explainable features, and the global-level importance direction labels associated with the highest-ranked explainable features; and

calculate a local erroneousness score for one or more new predictions by the machine learning base model based on one or more erroneous global-level importance direction labels associated with the one or more highest-ranked explainable features.

6. The system of claim 5 , wherein the at least one processor is further configured to execute the instructions to:

receive an erroneousness assessment identifying the one or more erroneous global-level importance direction labels.

7. The system of claim 5 , wherein the at least one processor is further configured to execute the instructions to correct the one or more new predictions or revising the machine learning base model based on the local erroneousness score.

8. The system of claim 5 , wherein the at least one processor is further configured to execute the instructions to receive a human erroneousness assessment of the global-level importance direction label.

9. The system of claim 5 , wherein the local erroneousness score comprises a normalized version of the accumulated error induced into each of the one or more new predictions by the highest ranked explainable features associated with the one or more erroneous global-level importance direction labels, and wherein the instructions further comprise:

determining an erroneousness designation for one of the one or more new predictions based on whether the local erroneousness score for the new prediction exceeds a threshold value.

10. The system of claim 5 , wherein the at least one processor is further configured to execute the instructions to run the machine learning base model on a first input dataset to generate a pair of baseline predictions by the machine learning base model and to determine a local-level importance of the explainable feature of the machine learning base model to a prediction class of the machine learning base model.

11. The system of claim 10 , wherein running the machine learning base model on the first input dataset to generate the pair of baseline predictions by the machine learning base model and to determine the local-level importance of the explainable feature of the machine learning base model to the prediction class of the machine learning base model comprises using a data perturbation process to generate the pair of baseline predictions by the machine learning base model and to determine the local-level importance of the explainable feature of the machine learning base model to the prediction class of the machine learning base model.

12. The method of claim 1 , wherein the local erroneousness score comprises a normalized version of the accumulated error induced into each of the one or more new predictions by the highest ranked explainable features associated with the one or more erroneous global-level importance direction labels, and wherein the instructions further comprise:

determining an erroneousness designation for one of the one or more new predictions based on whether the local erroneousness score for the new prediction exceeds a threshold value.

13. The method of claim 1 , further comprising running the machine learning base model on a first input dataset to generate a pair of baseline predictions by the machine learning base model and to determine a local-level importance of the explainable feature of the machine learning base model to a prediction class of the machine learning base model.

14. The method of claim 13 , wherein running the machine learning base model on the first input dataset to generate the pair of baseline predictions by the machine learning base model and to determine the local-level importance of the explainable feature of the machine learning base model to the prediction class of the machine learning base model comprises using a data perturbation process to generate the pair of baseline predictions by the machine learning base model and to determine the local-level importance of the explainable feature of the machine learning base model to the prediction class of the machine learning base model.

15. A computer program product comprising computer readable instructions stored on a non-transitory computer readable medium, the instructions executable by a processor to cause the processor to:

determine a plurality of global-level importance magnitude values representing a global-level importance of a plurality of explainable features of the machine learning base model to the machine learning base model;

determine a plurality of global-level importance direction labels representing the global-level importance of the explainable features of the machine learning base model to the machine learning base model;

rank the explainable features according to their associated global level importance magnitudes;

generate a communication for presentation to the user comprising one or more highest-ranked explainable features of the ranked explainable features, and the global-level importance direction labels associated with the highest-ranked explainable features; and

calculate a local erroneousness score for one or more new predictions by the machine learning base model based on one or more erroneous global-level importance direction labels associated with the one or more highest-ranked explainable features.

16. The computer program product of claim 15 , further comprising receiving an erroneousness assessment identifying the one or more erroneous global-level importance labels.

17. The computer program product of claim 15 , wherein executing the instructions are further executable by the processor to cause the processor to correct the one or more new predictions or revising the machine learning base model based on the local erroneousness score.

18. The computer program product of claim 15 , wherein receiving the erroneousness assessment of the global-level importance direction label comprises receiving a human erroneousness assessment of the global-level importance direction label.

19. The computer program product of claim 15 , wherein the local erroneousness score comprises a normalized version of the accumulated error induced into each of the one or more new predictions by the highest ranked explainable features associated with the one or more erroneous global-level importance direction labels, and wherein the instructions further comprise:

determining an erroneousness designation for one of the one or more new predictions based on whether the local erroneousness score for the new prediction exceeds a threshold value.

20. The computer program product of claim 15 , wherein executing the instructions are further executable by the processor to cause the processor to run the machine learning base model on a first input dataset to generate a pair of baseline predictions by the machine learning base model and to determine a local-level importance of the explainable feature of the machine learning base model to a prediction class of the machine learning base model.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 28, 2023
From: LIU, ZHE; GUO, YUFAN; MAHMUD, JALAL; AKKIRAJU, RAMA KALYANI T.
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 064101/0209 →
Continuity (2)
Continuation 16888356 · May 29, 2020
Related Publication 20230334375A1 · Oct 19, 2023
References Cited (56)
US 11720819B2 · Liu et al. · 2023 [cited by applicant]
US 20020157095A1 · Masumitsu · 2002 [cited by examiner]
US 20130311244A1 · Abotchie · 2013 [cited by applicant]
US 20190122135A1 · Parker · 2019 [cited by applicant]
US 20190325333A1 · Chan · 2019 [cited by examiner]
US 20200050932A1 · Iida et al. · 2020 [cited by applicant]
US 20220231981A1 · Patel · 2022 [cited by applicant]
CN 101515313A · 2009 [cited by applicant]
CN 109472318A · 2019 [cited by applicant]
CN 111008898A · 2020 [cited by applicant]
CN 115668238A · 2023 [cited by applicant]
GB 2610775A · 2023 [cited by applicant]
JP 2004086896A · 2004 [cited by applicant]
JP 2023526772A · 2023 [cited by applicant]
WO 2019130974A1 · 2019 [cited by applicant]
WO 2021240300A1 · 2021 [cited by applicant]
Adadi, A., et al., “Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI),” IEEE Access, vol. 6, 2018, 23 pages. [cited by applicant]
Arguello, J., et al., “Classification-Based Resource Selection,” CIKM '09, Nov. 2-6, 2009, Hong Kong, China, 10 pages. [cited by applicant]
Bassil, Y., et al., “OCR Post-Processing Error Correction Algorithm Using Google's Online Spelling Suggestion,” Journal of Emerging Trends in Computing and Information Sciences, ISSN 2079-8407, vol. 3, No. 1, Jan. 2012,… [cited by applicant]
Chander, et al., “Working with Beliefs: AI Transparency in the Enterprise,” ExSS '18, Mar. 11, Tokyo, Japan, 4 pages. [cited by applicant]
Chen, N., et al., “AnchorViz: Facilitating Classifier Error Discovery through Interactive Semantic Data Exploration,” IUI '18, Mar. 7-11, 2018, Tokyo, Japan, ACM ISBN 978-1-4503-4945-1/18/03, DOI: https://doi.org.10, 12… [cited by applicant]
Crawford, K., “The Social and Economic Implications of Artificial Intelligence Technologies in the Near-Term,” The AI Now Report, Version 1.0, Sep. 22, 2016, 25 pages. [cited by applicant]
Fails, J., et al., “Interactive Machine Learning,” Jan. 12-15, 2003, pp. 39-45. [cited by applicant]
Fiebrink, R., “Human Model Evaluation in Interactive Supervised Learning,” May 7-12, 2011, 10 pages. [cited by applicant]
Goldstein, A., “Peeking Inside the Black Box: Visualizing Statistical Learning with Plots of Individual Conditional Expectation,” Mar. 21, 2014, 22 pages. [cited by applicant]
Gunning, D., “Explainable Artificial Intelligence (XAI),” DARPA/120, 18 pages. [cited by applicant]
Harper, F., “Facts or Friends? Distinguishing Informational and Conversational Questions in Social Q&A Sites,” CHI, Apr. 4-9, 2009, 10 pages. [cited by applicant]
Inkpen, K., et al., “Where is the Human? Bridging the Gap Between AI and HCI,” CHI 2019 Workshop Summary, May 4-9, 2019, Glasgow, Scotland, UK, 9 pages. [cited by applicant]
Kim, Y., “Convolutional Neural Networks for Sentence Classification,” Sep. 3, 2014, 6 pages. [cited by applicant]
Klein, T., “Error awareness and the insula: Links to neurological and psychiatric diseases,” Review Article, Frontiers in Human Neuroscience, vol. 7, Article 14, Feb. 4, 2013, 15 pages. [cited by applicant]
Lakkaraju, H., “Interpretable & Explorable Approximations of Black Box Models,” Jul. 4, 2017, 5 pages. [cited by applicant]
Liu, Z., “Seemo: A Computational Approach to See Emotions,” CHI, Montreal, QC, Canda, Apr. 21-26, 2018, 12 pages. [cited by applicant]
Lundberg, S., et al., “A Unified Approach to Interpreting Model Predictions,” 31st Conference on Neural Information Processing Systems, NIPS 2017, Long Beach, CA, USA, Dec. 2017, 10 pages. [cited by applicant]
Malossini, Andrea, et al., “Detecting Potential Labeling Errors in Microarrays by Data Perturbation,” Bioinformatics, v.22, n. 17, 2006, pp. 2114-2121. [cited by applicant]
Mojsilovic, A., “Factsheets for AI Services,” Building Trusted AI, IBM Research, Aug. 22, 2018, 2018, 7 pages. [cited by applicant]
Molnar, C., “Interpretable Machine Learning, A Guide for Making Black Box Models Explainable” Dec. 17, 2019, 3 pages. [cited by applicant]
Nakov , P., “SemEval-2016 Task 4: Sentiment Analysis in Twitter,” Proceedings of SemEval, 2016, 18 pages. [cited by applicant]
Novak, P., et al., “Sentiment of Emojis,” PLOS One, Dec. 7, 2015, 22 pages. [cited by applicant]
Nushi, B., “Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure,” The Sixth AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2018), 2018, pp. 126-135. [cited by applicant]
Pennington, J., et al., “GloVe: Global Vectors for Word Representation,” Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Oct. 25-29, 2014, pp. 1532-1543. [cited by applicant]
Ribeiro, M., et al., “Why Should I Trust You?” Explaining the Predictions of Any Classifier, KDD, San Francisco, CA, USA, 2016, 10 pages. [cited by applicant]
Samek, W., “Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models,” Aug. 28, 2017, 8 pages. [cited by applicant]
Settles, B., “Active Learning Literature Survey,” Computer Sciences Technical Report 1648, University of Wisconsin-Madison, Jan. 26, 2010, 67 pages. [cited by applicant]
Stymne, S., et al., “Blast: A Tool for Error Analysis of Machine Translation Output,” Proceedings of the ACL-HLT, System Demonstrations, Jun. 21, 2011, pp. 56-61. [cited by applicant]
Varma, P., et al., “Flipper: A Systematic Approach to Debugging Training Sets,” HILDA '17, May 14, 2017, Chicago, IL, USA; http://dx.doi.org/10.1145/3077257.3077264, 5 pages. [cited by applicant]
Varma, Paroma, et al., “Snuba: Automating Weak Supervision to Label Training Data,” Proceedings of the VLDB Endowment 12.3 (2018): 223-236. [cited by applicant]
Vaswani, A., “Attention Is All You Need,” 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 2017, 11 pages. [cited by applicant]
Zhang, C., et al., “Methods for Labeling Error Detection in Microarrays Based on the Effect of Data Perturbation on the Regression Model,” Original Paper, vol. 25, No. 20, 2009, pp. 2708-2714; doi:10.1093/bioinformatics… [cited by applicant]
Foreign Communication from a counterpart application, PCT Application PCT/IB2021/054267 filed May 18, 2021, International Search Report and Written Opinion mailed Aug. 24, 2021, 9 pages. [cited by applicant]
IBM: List of IBM Patents or Patent Applications Treated as Related. Filed Herewith. 2 pages. [cited by applicant]
Japan Patent Office, “Decision to Grant a Patent,” Oct. 15, 2024, 5 Pages, JP Application No. 2022-564593. [cited by applicant]
Das et al. “End-user feature labeling: Supervised and semi-supervised approaches based on locally-weighted logistic regression”, Artificial Intelligence, vol. 204, Nov. 2013, pp. 56-74. [cited by applicant]
Jain et al. “A Dynamic Confusion Score for Dependency Arc Labels”, In Proceedings of the Sixth International Joint Conference on Natural Language Processing, 2013, pp. 1237-1242. [cited by applicant]
Letham et al., “Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model”, arXiv:1511.01644v1, Nov. 5, 2015, 23 pages. [cited by applicant]
Wu et al., “Mining With Noise Knowledge: Error-Aware Data Mining”, IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans, vol. 38, No. 4, Jul. 2008, 16 pages. [cited by applicant]
Xiong et al., “Error Detection for Statistical Machine Translation Using Linguistic Features”, Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Jul. 11-16, 2010, pp. 604-611. [cited by applicant]