IP Library › Granted Patent US 12,367,396
Granted Patent B1
US 12,367,396 · App. 17/216,455 · Granted Jul 22, 2025

Automatic failure diagnosis and correction in machine learning models

Inventors: Nathalie Rauschmayr (North Vancouver, CA); Krishnaram Kenthapadi (Sunnyvale, CA); Dylan Slack (San Rafael, CA)
Assignee: Amazon Technologies, Inc.
G06N3/088G06N3/045
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,367,396
App. No.
17/216,455
Granted
Jul 22, 2025
Kind
B1
Abstract

Automatic failure diagnosis and correction may be performed on trained machine learning models. Input data that causes a trained machine learning model may be identified in order to determine different model failures. The model failures may be clustered in order to determine failure scenarios for the trained machine learning model. Examples of the failure scenarios may be generated and truth labels for the example scenarios obtained. The examples and truth labels may then be used to retrain the machine learning model to generate a corrected version of the machine learning model.

Claims (61)

1. A system, comprising:

at least one processor; and

a memory, storing program instructions that when executed by the at least one processor, cause the at least one processor to:

responsive to a request received via an interface for a machine learning system that causes automatic failure diagnosis for a trained machine learning model:

identify different respective input data to the trained machine learning model that cause the trained machine learning model to generate incorrect results corresponding to the different respective input data;

apply a clustering technique to group the different respective input data into one or more failure scenarios for the trained machine learning model, wherein the one or more failure scenarios correspond to respective regions in a latent space of the trained machine learning model;

generate a plurality of respective failure examples for the one or more failure scenarios;

obtain a respective truth label for one of the plurality of respective failure examples via the interface, wherein the respective truth label for the one of the plurality of respective failure examples applies to other ones of the plurality of respective failure examples generated from a same one of the respective regions in the latent space; and

tune the trained machine learning model using the plurality of respective failures examples for the one or more failure scenarios with the respective truth label for the one of the plurality of respective failure examples that applies to other ones of the plurality of respective failure examples to generate a corrected version of the trained machine learning model.

2. The system of claim 1 , wherein to identify the different respective input data to the trained machine learning model that cause the trained machine learning model to generate incorrect results corresponding to the different respective input data, the program instructions cause the at least one processor to:

use an encoder of a Variational Autoencoder (VAE) to generate a latent vector for an input data of a training data set for the trained machine learning model;

generate a set of latent vectors that deviate from the latent vector;

use a decoder of the VAE to generate respective sample data inputs from the set of latent vectors;

compare respective results of the trained machine learning model applied to the respective sample data inputs with a truth label corresponding to the input data of the training data set;

based on the comparing, add those latent vectors of the set of latent vectors with the respective results that do not match the truth label to a failure scenario evaluation set; and

wherein the failure scenario evaluation set is the different respective input data to which the clustering technique is applied.

3. The system of claim 1 , wherein to obtain the respective truth label for the plurality of respective failure examples via the interface, the program instructions cause the at least one processor to:

provide at least one of the plurality of respective failure examples via the interface; and

receive, via the interface, the respective truth label for the at least one of the plurality of respective failure examples.

4. The system of claim 1 , wherein the machine learning system is a machine learning service implemented as part of a provider network and wherein the interface is an integrated development environment (IDE) implemented by the machine learning system.

5. A method, comprising:

identifying different respective input data to a trained machine learning model that cause the trained machine learning model to generate incorrect results corresponding to the different respective input data;

applying a clustering technique to group the different respective input data into one or more failure scenarios for the trained machine learning model, wherein the one or more failure scenarios correspond to respective regions in a latent space of the trained machine learning model;

obtaining a respective truth label for one of a respective plurality of failure examples generated for the one or more failure scenarios, wherein the respective truth label for the one of the plurality of respective failure examples applies to other ones of the plurality of respective failure examples generated from a same one of the respective regions in the latent space; and

retraining the trained machine learning model using the plurality of respective failures examples for the one or more failure scenarios with the respective truth label for the one of the plurality of respective failure examples that applies to other ones of the plurality of respective failure examples to generate a corrected version of the trained machine learning model.

6. The method of claim 5 , wherein obtaining the respective truth labels for the one or more failure examples generated for the one or more failure scenarios comprises:

providing at least one of the plurality of respective failure examples via an interface; and

receiving, via the interface, the respective truth label for the at least one of the plurality of respective failure examples.

7. The method of claim 6 , wherein a respective current label as determined by the trained machine learning model for the at least one of the plurality of respective failure examples is provided with the at least one of the plurality of respective failure examples via the interface.

8. The method of claim 5 , wherein the plurality of respective failure examples are generated using a decoder of a Variable Autoencoder (VAE) applied to respective different respective latent codes sampled from the same region of the latent space.

9. The method of claim 5 , wherein identifying the different respective input data to the trained machine learning model that cause the trained machine learning model to generate incorrect results corresponding to the different respective input data comprises:

using an encoder of a Variational Autoencoder (VAE) to generate a latent vector for an input data of a training data set for the trained machine learning model;

generating a set of latent vectors that deviate from the latent vector;

using a decoder of the VAE to generate respective sample data inputs from the set of latent vectors;

comparing respective results of the trained machine learning model applied to the respective sample data inputs with a truth label corresponding to the input data of the training data set;

based on the comparing, adding those latent vectors of the set of latent vectors with the respective results that do not match the truth label to a failure scenario evaluation set; and

wherein the failure scenario evaluation set is the different respective input data to which the clustering technique is applied.

10. The method of claim 5 , wherein identifying the different respective input data to the trained machine learning model that cause the trained machine learning model to generate incorrect results corresponding to the different respective input data comprises randomly sampling vectors from a latent vector space determined using a trained Variational Autoencoder (VAE).

11. The method of claim 5 , further comprising receiving, via an interface, a request to perform automatic diagnosis and correction for the trained machine learning model and wherein the identifying, the applying, the obtaining, and the retraining are performed in response to the request.

12. The method of claim 5 , wherein retraining the trained machine learning model using the plurality of respective failure examples for the one or more failure scenarios with the respective truth labe to generate the corrected version of the trained machine learning model comprises applying an additional regularization term that is a cross entropy loss between the plurality of respective failure examples and the respective truth label.

13. The method of claim 5 , wherein the trained machine learning model is an image classification model and wherein the one or more failure examples are generated as different respective images.

14. One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement:

identifying different respective input data to a trained machine learning model that cause the trained machine learning model to generate incorrect results corresponding to the different respective input data;

applying a clustering technique to group the different respective input data into one or more failure scenarios for the trained machine learning model, wherein the one or more failure scenarios correspond to respective regions in a latent space of the trained machine learning model;

generating a plurality of respective failure examples for the one or more failure scenarios;

obtaining a respective truth label for one of the plurality of respective failure examples generated for the one or more failure scenarios, wherein the respective truth label for the one of the plurality of respective failure examples applies to other ones of the plurality of respective failure examples generated from a same one of the respective regions in the latent space; and

retraining the trained machine learning model using the plurality of respective failures examples for the one or more failure scenarios with the respective truth label for the one of the plurality of respective failure examples that applies to other ones of the plurality of respective failure examples to generate a corrected version of the trained machine learning model.

15. The one or more non-transitory, computer-readable storage media of claim 14 , wherein, in obtaining the respective truth labels for the one or more failure examples generated for the one or more failure scenarios, the program instructions cause the one or more computing devices to implement:

displaying at least one of the plurality of respective failure examples via an interface; and

receiving, via the interface, the respective truth label for the at least one of the plurality of respective failure examples.

16. The one or more non-transitory, computer-readable storage media of claim 15 , wherein a respective current label as determined by the trained machine learning model is provided with the at least one of the plurality of respective failure examples via the interface.

17. The one or more non-transitory, computer-readable storage media of claim 14 , wherein, in identifying the different respective input data to the trained machine learning model that cause the trained machine learning model to generate incorrect results corresponding to the different respective input data, the program instructions cause the one or more computing devices to implement:

using an encoder of a Variational Autoencoder (VAE) to generate a latent vector for an input data of a training data set for the trained machine learning model;

generating a set of latent vectors that deviate from the latent vector;

using a decoder of the VAE to generate respective sample data inputs from the set of latent vectors;

comparing respective results of the trained machine learning model applied to the respective sample data inputs with a truth label corresponding to the input data of the training data set;

based on the comparing, adding those latent vectors of the set of latent vectors with the respective results that do not match the truth label to a failure scenario evaluation set; and

wherein the failure scenario evaluation set is the different respective input data to which the clustering technique is applied.

18. The one or more non-transitory, computer-readable storage media of claim 14 , storing further instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to implement receiving, via an interface, a request to perform a training job that generates the trained machine learning model and wherein the identifying, the applying, the generating, the obtaining, and the retraining are performed as part of executing the training job.

19. The one or more non-transitory, computer-readable storage media of claim 14 , wherein, in retraining the trained machine learning model using plurality of respective failures examples for the one or more failure scenarios with the respective truth label for the one of the plurality of respective failure examples that applies to other ones of the plurality of respective failure examples to generate a corrected version of the trained machine learning model, the program instructions cause the one or more computing devices to implement applying an additional regularization term that is a cross entropy loss between the plurality of respective failure examples and the respective truth label.

20. The one or more non-transitory, computer-readable storage media of claim 14 , wherein the one or more computing devices are implemented as part of a machine learning service offered by a provider network and wherein the machine learning service hosts the corrected machine learning to perform respective inferences using the corrected machine learning model responsive to one or more requests directed to the corrected machine learning model.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 16, 2024
From: RAUSCHMAYR, NATHALIE; KENTHAPADI, KRISHNARAM; SLACK, DYLAN
To: AMAZON TECHNOLOGIES, INC.
Reel/Frame 067439/0536 →
Continuity (1)
Provisional Application 63086563 · Oct 1, 2020
References Cited (41)
US 20200394461A1 · Perera · 2020 [cited by examiner]
US 20210073671A1 · Puri · 2021 [cited by examiner]
Antoniou, A.; Storkey, A.; and Edwards, H. 2017. “Data Augmentation Generative Adversarial Networks”. International Conference on Artificial Neural Networks and Machine Learning, arXiv preprint arXiv:1711.04340, 2017, p… [cited by applicant]
Booth, S.; Zhou, Y.; Shah, A.; and Shah, J., “Bayes- TrEx: Model Transparency by Example”, eprint arXiv:2002.10248, Feb. 2020, pp. 1-29. [cited by applicant]
Carmon, Y.; Raghunathan, A.; Schmidt, L.; Liang, P.; and Duchi, J, “Unlabeled Data Improves Adversarial Robustness”, arXiv:1905.13736v3 [stat.ML] Dec. 4, 2019, In Advances in Neural Information Processing Systems (NeurI… [cited by applicant]
Dumoulin, V.; Belghazi, I.; Poole, B.; Mastropietro, O.; Lamb, A.; Arjovsky, M.; and Courville, A., “Adversarially Learned Inference”. Published as a conference paper at ICLR 2017, pp. 1-18. [cited by applicant]
Engstrom, L.; Ilyas, A.; Santurkar, S.; Tsipras, D.; Stein-hardt, J.; and Madry, A., “Identifying Statistical Bias in Dataset Replication”, Proceedings of the 37th International Conference on Machine Learning, arXiv:200… [cited by applicant]
Feldman, M.; Friedler, S. A.; Moeller, J.; Scheidegger, C.; and Venkatasubramanian, S., “Certifying and Removing Disparate Impact”, In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery a… [cited by applicant]
Gardner, M.; et al., “Evaluating Models' Local Decision Boundaries via Contrast Sets”, In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1307-1323. Online: Association for Computational Lingu… [cited by applicant]
He, K.; Zhang, X.; Ren, S.; and Sun, J., “Deep Residual Learning for Image Recognition,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 770-778. [cited by applicant]
Higgins, I.; Matthey, L.; Pal, A.; Burgess, C.; Glorot, X.; Botvinick, M. M.; Mohamed, S.; and Lerchner, A., “β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework”. ICLR 2017 Conference Submiss… [cited by applicant]
Ji, D.; Smyth, P.; and Steyvers, M., “Can I Trust My Fairness Metric? Assessing Fairness with Unlabeled Data and Bayesian Inference”, In Advances in Neural Information Processing Systems, arXiv:2010.09851v1 [stat.ML] Oc… [cited by applicant]
Kang, D.; Raghavan, D.; Bailis, P.; and Zaharia, M., “Model Assertions for Debugging Machine Learning,” Debugging Machine Learning Models, In NeurIPS MLSys Workshoppages, 2018 pp. 1-9. [cited by applicant]
Kingma, D. P.; Mohamed, S.; Jimenez Rezende, D.; and Welling, M., “Semi-supervised Learning with Deep Generative Models,” In NIPS'14: Proceedings of the 27th International Conference on Neural Information Processing Sys… [cited by applicant]
Kumar, A.; Sattigeri, P.; and Fletcher, T., “Semi- supervised Learning with GANs: Manifold Invariance with Improved Inference,” In NIPS'17: Proceedings of the 31st International Conference on Neural Information Processi… [cited by applicant]
LeCun, Y.; Cortes, C.; and Burges, C., “Mnist handwritten digit database,” ATT Labs [Online], Available: http://yann.lecun.com/exdb/mnist, 2010, pp. 1-7. [cited by applicant]
Lundberg, S. M.; and Lee, S.-I., :A Unified Ap proach to Interpreting Model Predictions,. In Advances in Neural Information Processing Systems 30, (NIPS 2017) pp. 4765-4774. Curran Associates, Inc. URL http://papers.nip… [cited by applicant]
Milz, S.; Rudiger, T.; and Suss, S., “Aerial GANeration: Towards Realistic Data Augmentation Using Conditional GANs,” In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018, pp. 1-14. [cited by applicant]
Netzer, Y.; Wang, T.; Coates, A.; Bissacco, .; Wu, B.; and Ng, A. Y., “Reading Digits in Natural Images with Unsupervised Feature Learning,” In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011, pp.… [cited by applicant]
Odena, A.; Olsson, C.; Andersen, D.; and Goodfellow, I., “TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing,” Proceedings of the 36th International Conference on Machine Learning, PMLR 97: 2019, pp. 490… [cited by applicant]
Paszke, A.; Gross, S.; Chintala, S.; Chanan, G.; Yang, E.; DeVito, Z.; Lin, Z.; Desmaison, A.; Antiga, L.; and Lerer, A., “MNIST Example Pytorch,” URL https://github.com/pytorch/examples, pp. 1-2, 2019. [cited by applicant]
Patel, K.; Fogarty, J.; Landay, J. A.; and Harrison, B., “Investigating Statistical Machine Learning as a Tool for Software Development,”. In CHI '08: Proceedings of the SIGCHI Conference on Human Factors in Computing S… [cited by applicant]
Pedregosa, F.; et al., “Scikit-learn: Machine Learning in Python”, Journal of Machine Learning Research 12: pp. 2825-2830, 2011. [cited by applicant]
Raghunathan, A.; Xie, S. M.; Yang, F.; Duchi, J.; and Liang, P., et al. “Understanding and Mitigating the Tradeoff between Robustness and Accuracy,”. In Proceedings of the 37th International Conference on Machine Learni… [cited by applicant]
Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P., “Squad: 100,000+ Questions for Machine Comprehension of Text,” In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-23… [cited by applicant]
Recht, B.; Roelofs, R.; Schmidt, L.; and Shankar, V., “Do ImageNet Classifiers Generalize to ImageNet?,” vol. 97 of Proceedings of Machine Learning Research, pp. 5389-5400. Long Beach, California, USA: PMLR. URL:http://… [cited by applicant]
Ribeiro, M. T.; Singh, S.; and Guestrin, C., “Why should i trust you?” Explaining the predictions of any classifier, In KDD '16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Dat… [cited by applicant]
Ribeiro, M. T.; Wu, T.; Guestrin, C.; and Singh, S., “Beyond Accuracy: Behavioral Testing of NLP Models with CheckList,”. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4902-… [cited by applicant]
Russakovsky, O., et al., “ImageNet Large Scale Visual Recognition Challenge,”. International Journal of Computer Vision (IJCV) 115(3): pp. 211-252, doi: 10.1007/s11263-015-0816-y, 2015, preprint arXiv:1409.0575v2 [cs.CV… [cited by applicant]
Sandfort, V.; Yan, K.; Pickhardt, P. J.; and Summers, R. M., “Data augmentation using generative adversarial networks (CycleGAN) to improve generalizability in CT segmentation tasks,”. Scientific Reports 9(1):16884, ISS… [cited by applicant]
Slack, D.; Hilgard, S.; Singh, S.; and Lakkaraju, H, “How Much Should I Trust You? Modeling Uncertainty of Black Box Explanations”. AIES, 2020, pp. 1-14. [cited by applicant]
Song, Y.; Shu, R.; Kushman, N.; and Ermon, S., “Constructing Unrestricted Adversarial Examples with Generative Models”. In Advances in Neural Information Processing Systems 31, 2018, 8312-8323. Curran Associates, Inc. p… [cited by applicant]
Stallkamp, J.; Schlipsing, M.; Salmen, J.; and Igel, C., “The German Traffic Sign Recognition Benchmark: A multi-class classification competition”, In IEEE International Joint Conference on Neural Networks, pp. 1453-146… [cited by applicant]
Sudderth, E. B., “Graphical Models for Visual Object Recognition and Tracking”, Submitted to the Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, pp. 1-301, 2016. [cited by applicant]
Varma, P.; He, B.; Iter, D.; Xu, P.; Yu, R.; Sa, C. D.; and Re', C., 2016, Socratic Learning: Augmenting Generative Models to Incorporate Latent Subsets in Training Data, arXiv:1610.08123v4 [cs.LG], Sep. 28, 2017, pp. 1… [cited by applicant]
Varma, P.; Iter, D.; De Sa, C.; and Re', C., “Flipper: A Systematic Approach to Debugging Training Sets,” In Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics, HILDA'17. New York, NY, USA: Association … [cited by applicant]
Wu, I.; Ribeiro, M. I.; Heer, J.; and Weld, D., “Errudite: Scalable, Reproducible, and Testable Error Analysis,”. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 747-763. … [cited by applicant]
Zhang, H.; Yu, Y.; Jiao, J.; Xing, E.; Ghaoui, L. E.; and Jor-dan, M., “Theoretically Principled Trade-off between Robustness and Accuracy,”. vol. 97 of Proceedings of the 36th International Conference on Machine Learni… [cited by applicant]
Zhang, X.; Zhu, X.; and Wright, S. J., “Training Set Debugging Using Trusted Items,” In vol. 32 No. 1 (2018): Thirty-Second AAAI Conference on Artificial Intelligence, pp. 4482-4489. [cited by applicant]
Zhao, Z.; Dua, D.; and Singh, S., “Generating Natural Adversarial Examples”, In International Conference on Learning Representations (ICLR), 2018, pp. 1-16. [cited by applicant]
Christopher M. Bishop, Pattern Recognition and Machine Learning (Information Science and Statistics) Chapter 9: Mixture Models and EM, Springer-Verlag, ISBN 0387310738, 2006, pp. 423-459. [cited by applicant]
Cited By (3)
US 12,645,563 US 12,664,768 US 12,717,555