IP Library Granted Patent US 12,613,939
Granted Patent B2
US 12,613,939 · App. 17/936,677 · Granted Apr 28, 2026

Correcting a machine learning model

Inventors: Ryutaro Tanno (Cambridge, GB); Aditya Nori (Cambridge, GB); Melanie Fernandez Pradier (Cambridge, GB); Yingzhen Li (Cambridge, GB)
Assignee: Microsoft Technology Licensing, LLC.
G06F18/2193G06F18/211G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,613,939
App. No.
17/936,677
Granted
Apr 28, 2026
Kind
B2
Abstract

A computer-implemented method includes receiving an incorrect prediction output by a trained machine learning model, which has been trained using training data items. The method includes identifying a training data item used to train the model that is a cause of the incorrect prediction, by determining an impact on performance of the trained machine learning model associated with removing the training data item from the plurality of training data. The trained model can then be updated to remove the effect of the identified training data item, allowing the model to be automatically corrected in view of poor quality training data.

Claims (57)

1 . A system for saving computational resource and memory usage while ensuring that a previous property of a trained machine learning model is retained, the system comprising a processor and a memory, the memory storing computer-readable instructions that, when executed by the processor, cause the system to:

receive an incorrect prediction of the trained machine learning model, the trained machine learning model based on a plurality of training data items;

select a training data item of the plurality of training data items;

calculate an estimate of an impact on performance of the trained machine learning model in respect of the incorrect prediction, the impact caused by removing the selected training data item from the plurality of training data items, wherein the estimate of the impact is calculated by:

calculating a first posterior probability distribution corresponding to the trained machine learning model;

calculating a second posterior distribution corresponding to the trained machine learning model with the selected training data item removed; and

comparing the first posterior probability distribution and second posterior probability distribution to determine the estimate of the impact;

determine that the impact on performance indicates an improvement in performance of the trained machine learning model;

identify the selected training data item as a cause of the incorrect prediction based on the calculated estimate of the impact on performance indicating improvement in performance of the trained machine learning model; and

automatically correct the trained machine learning model to remove an effect of the selected training data item without requiring entire retraining of the trained machine learning model.

2 . The system of claim 1 , wherein the computer-readable instructions that cause the system to calculate the estimate of the impact on performance of the trained machine learning model comprises instructions that, when executed by the processor, determine an impact on the first posterior probability distribution of the trained machine learning model.

3 . The system of claim 1 , wherein the trained machine learning model is corrected by unlearning datapoints in the selected training data item.

4 . The system of claim 1 , comprising instructions that, when executed by the processor, cause the system to:

determine a detrimental impact on performance of the trained machine learning model of each training data item in the plurality of training data items, and

select one or more of the plurality of training data items with greatest detrimental impact as the cause of the incorrect prediction.

5 . The system of claim 4 , wherein the computer-readable instructions that cause the system to determine the detrimental impact of each training data item comprises instructions that, when executed by the processor, cause the system to:

calculate a third posterior distribution corresponding to the trained machine learning model with the incorrect prediction added to the plurality of training data items,

wherein the third posterior distribution is estimated from the first posterior probability distribution corresponding to the trained machine learning model.

6 . The system of claim 5 , comprising instructions that, when executed by the processor, cause the system to estimate the third posterior distribution using a linear influence function.

7 . The system of claim 3 , wherein the datapoints in the selected training data item are unlearned while using the first posterior probability distribution as a prior.

8 . A computer-implemented method for saving computational resource and memory usage while ensuring that a previous property of a trained machine learning model is retained, the method comprising:

receiving an incorrect prediction of the trained machine learning model, the trained machine learning model based on a plurality of training data items;

selecting a training data item of the plurality of training data items;

calculating an estimate of an impact on performance of the trained machine learning model in respect of the incorrect prediction, the impact caused by removing the selected training data item from the plurality of training data items, wherein the estimate of the impact is calculated by:

calculating a first posterior probability distribution corresponding to the trained machine learning model;

calculating a second posterior distribution corresponding to the trained machine learning model with the selected training data item removed; and

comparing the first posterior probability distribution and second posterior probability distribution to determine the estimate of the impact;

determining that the impact on performance indicates an improvement in performance of the trained machine learning model;

identifying the selected training data item as a cause of the incorrect prediction based on the calculated estimate of the impact on performance indicating improvement in performance of the trained machine learning model; and

automatically correcting the trained machine learning model to remove an effect of the selected training data item without requiring entire retraining of the trained machine learning model.

9 . The method of claim 8 , wherein the trained machine learning model is one of an image analysis model, an audio analysis model, or a text processing model.

10 . The method of claim 8 , wherein calculating the estimate of the impact on performance of the trained machine learning model comprises determining an impact on the first posterior probability distribution of the trained machine learning model.

11 . The method of claim 8 , wherein the trained machine learning model is corrected by unlearning datapoints in the selected training data item.

12 . The method of claim 11 , wherein the datapoints in the selected training data item are unlearned while using the first posterior probability distribution as a prior.

13 . The method of claim 8 , wherein correcting the trained machine learning model comprises fine tuning the trained machine learning model on the identified selected training data item.

14 . The method of claim 8 , wherein correcting the trained machine learning model comprises:

calculating a third posterior distribution corresponding to the trained machine learning model with the identified selected training data item removed from the plurality of training data items,

wherein the third posterior distribution is estimated from the first posterior probability distribution corresponding to the trained machine learning model.

15 . The method of claim 8 , comprising:

receiving an input data item, and

using the corrected trained machine learning model to generate a prediction from the input data item.

16 . A tangible non-transient computer-readable storage medium for saving computational resource and memory usage while ensuring that a previous property of a trained machine learning model is retained, the computer-readable storage medium having recorded thereon instructions that, when executed by a computer device cause the computer device to:

receive an incorrect prediction of the trained machine learning model, the trained machine learning model based on a plurality of training data items;

select a training data item of the plurality of training data items;

calculate an estimate of an impact on performance of the trained machine learning model in respect of the incorrect prediction, the impact caused by removing the selected training data item from the plurality of training data items, wherein the estimate of the impact is calculated by:

calculating a first posterior probability distribution corresponding to the trained machine learning model;

calculating a second posterior distribution corresponding to the trained machine learning model with the selected training data item removed; and

comparing the first posterior probability distribution and second posterior probability distribution to determine the estimate of the impact;

determine that the impact on performance indicates an improvement in performance of the trained machine learning model;

identify the selected training data item as a cause of the incorrect prediction based on the calculated estimate of the impact on performance indicating improvement in performance of the trained machine learning model; and

automatically correct the trained machine learning model to remove an effect of the selected training data item without requiring entire retraining of the trained machine learning model.

17 . The tangible non-transient computer-readable storage medium of claim 16 , wherein the trained machine learning model is one of an image analysis model, an audio analysis model, or a text processing model.

18 . The tangible non-transient computer-readable storage medium of claim 16 , wherein calculating the estimate of the impact on performance of the trained machine learning model comprises determining an impact on the first posterior probability distribution of the trained machine learning model.

19 . The tangible non-transient computer-readable storage medium of claim 16 , wherein the trained machine learning model is corrected by unlearning datapoints in the selected training data item.

20 . The tangible non-transient computer-readable storage medium of claim 16 , comprising instructions to:

determine a detrimental impact on performance of the trained machine learning model of each training data item in the plurality of training data items, and

select one or more of the plurality of training data items with greatest detrimental impact as the cause of the incorrect prediction.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 4, 2023
From: TANNO, RYUTARO; NORI, ADITYA; PRADIER, MELANIE FERNANDEZ; LI, YINGZHEN
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 063212/0912 →
Continuity (2)
Provisional Application 63353520 · Jun 17, 2022
Related Publication 20230409674A1 · Dec 21, 2023
References Cited (63)
US 10990901B2 · Deo et al. · 2021 [cited by applicant]
US 11132584B2 · Challis et al. · 2021 [cited by applicant]
US 11288115B1 · Katsis et al. · 2022 [cited by applicant]
US 11887003B1 · Bopardikar · 2024 [cited by examiner]
US 20190236482A1 · Desjardins · 2019 [cited by examiner]
US 20200005148A1 · Nori et al. · 2020 [cited by applicant]
US 20210064700A1 · Gopalan · 2021 [cited by examiner]
US 20210256420A1 · Elisha et al. · 2021 [cited by applicant]
US 20210357779A1 · Kabra et al. · 2021 [cited by applicant]
US 20220269991A1 · Kurokawa · 2022 [cited by examiner]
US 20220383152A1 · Kallur Palli Kumar · 2022 [cited by examiner]
US 20240403715A1 · Wang · 2024 [cited by examiner]
“Amazon SageMaker Debugger”, Retrieved from: https://web.archive.org/web/20221005195418/http://aws.amazon.com/sagemaker/debugger/, Oct. 5, 2022, 8 Pages. [cited by applicant]
“AutoML”, Retrieved from: https://web.archive.org/web/20220830122953/https://www.microsoft.com/en-us/research/project/automl/, Aug. 30, 2022, 2 Pages. [cited by applicant]
Agarwal, et al., “Second Order Stochastic Optimization in Linear Time”, In Journal of Stat, vol. 1050, Feb. 15, 2016, 27 Pages. [cited by applicant]
Amari, Shun-Ichi, “Natural Gradient Works Efficiently in Learning”, In Journal of Neural Computation, vol. 10, Issue 2, Feb. 15, 1998, pp. 251-276. [cited by applicant]
Ancona, et al., “Explaining Deep Neural Networks with A Polynomial Time Algorithm for Shapley Value Approximation”, In Proceedings of International Conference on Machine Learning, May 24, 2019, 10 Pages. [cited by applicant]
Barshan, et al., “Relatif: Identifying Explanatory Training Examples Via Relative Influence”, In Repository of arXiv:2003.11630v1, Mar. 25, 2020, 18 Pages. [cited by applicant]
Bourtoule, et al., “Machine Unlearning”, In Repository of arXiv:1912.03817v1, Dec. 9, 2019, 16 Pages. [cited by applicant]
Cao, et al., “Editing Factual Knowledge in Language Models”, In Repository of arXiv:2104.08164v2, Sep. 8, 2021, 16 Pages. [cited by applicant]
Chakarov, et al., “Debugging Machine Learning Tasks”, In Repository of arXiv:1603.07292v1, Mar. 23, 2016, 29 Pages. [cited by applicant]
Chen, et al., “Why is My Classifier Discriminatory?”, In Proceedings of Advances in Neural Information Processing Systems, Dec. 3, 2018, 12 Pages. [cited by applicant]
Farajtabar, et al., “Orthogonal Gradient Descent for Continual Learning”, In Proceedings of International Conference on Artificial Intelligence and Statistics, Aug. 26, 2020, 11 Pages. [cited by applicant]
Finn, et al., “Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks”, In Proceedings of International conference on machine learning, Aug. 6, 2017, 10 Pages. [cited by applicant]
Frenay, et al., “Classification in the Presence of Label Noise: A Survey”, In Journal of IEEE Transactions on Neural Networks and Learning Systems vol. 25, Issue 5, May 2014, pp. 845-869. [cited by applicant]
Ghorbani, et al., “Data Shapley: Equitable Valuation of Data For Machine Learning”, In Proceedings of 36th International Conference on Machine Learning, Jun. 9, 2019, 10 Pages. [cited by applicant]
Ginart, et al., “Making AI Forget You: Data Deletion in Machine Learning”, In Proceedings of Advances in Neural Information Processing Systems, Dec. 8, 2019, 14 Pages. [cited by applicant]
Giordano, et al., “A Swiss Army Infinitesimal Jackknife”, In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, Apr. 16, 2019, 9 Pages. [cited by applicant]
Gish, et al., “Towards a New Understanding of the Training of Neural Networks with Mislabeled Training Data”, In Repository of arXiv:1909.09136v1, Sep. 18, 2019, 13 Pages. [cited by applicant]
Goodfellow, et al., “Explaining and Harnessing Adversarial Examples”, In Repository of arXiv:1412.6572v1, Dec. 20, 2014, 10 Pages. [cited by applicant]
Guo, et al., “Certified Data Removal from Machine Learning Models”, In Repository of arXiv:1911.03030v1, Nov. 8, 2019, 14 Pages. [cited by applicant]
Gupta, et al., “Adaptive Machine Unlearning”, In Repository of arXiv:2106.04378v1, Jun. 8, 2021, 25 Pages. [cited by applicant]
Hara, et al., “Data Cleansing for Models Trained with SGD”, In Proceedings of Advances in Neural Information Processing Systems, vol. 32, Dec. 8, 2019, 10 Pages. [cited by applicant]
Hara, Satoshi, “Sgd-Influence”, Retrieved from: https://github.com/sato9hara/sgd-influence, Sep. 28, 2020, 2 Pages. [cited by applicant]
He, et al., “Deep Residual Learning for Image Recognition”, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 27, 2016, pp. 770-778. [cited by applicant]
Izzo, et al., “Approximate Data Deletion from Machine Learning Models: Algorithms and Evaluations”, In Repository of arXiv:2002.10077v1, Feb. 24, 2020, 20 Pages. [cited by applicant]
Jia, et al., “Towards Efficient Data Valuation Based on the Shapley Value”, In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, Apr. 16, 2019, 10 Pages. [cited by applicant]
Khanna, et al., “Interpreting Black Box Predictions using Fisher Kernels”, In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistic, Apr. 11, 2019, 9 Pages. [cited by applicant]
Kingma, et al., “Adam: A Method for Stochastic Optimization”, In Repository of arXiv:1412.6980v1, Dec. 22, 2014, 9 Pages. [cited by applicant]
Kirkpatricka, et al., “Overcoming Catastrophic Forgetting in Neural Networks”, In Proceedings of the National Academy of Sciences, vol. 114, Issue 13, Mar. 14, 2017, pp. 3521-3526. [cited by applicant]
Koh, et al., “On The Accuracy of Influence Functions for Measuring Group Effects”, In Proceedings of Advances in Neural Information Processing Systems, May 2019, 11 Pages. [cited by applicant]
Koh, et al., “Understanding Black-Box Predictions via Influence Functions”, In Proceedings of the 34th International Conference on Machine Learning, vol. 70, Aug. 6, 2017, 10 Pages. [cited by applicant]
Koh, et al., “Wilds: A Benchmark of in-the-Wild Distribution Shifts”, In Proceedings of the 38th International Conference on Machine Learning, Jul. 1, 2021, 28 Pages. [cited by applicant]
Loo, et al., “Generalized Variational Continual Learning”, In Proceedings of International Conference on Learning Representations, Mar. 17, 2021, 46 Pages. [cited by applicant]
Mitchell, et al., “Fast Model Editing at Scale”, In Repository of arXiv:2110.11309v1, Oct. 21, 2021, 17 Pages. [cited by applicant]
Nag, Avishek, “Bayesian Approach and Model Evaluation”, Retrieved from: https://towardsdatascience.com/bayesian-approach-and-model-evaluation-371ad669cf2c, Feb. 3, 2022, 13 Pages. [cited by applicant]
Neel, et al., “Descent-To-Delete: Gradient-Based Methods for Machine Unlearning”, In Repository of arXiv:2007.02923v1, Jul. 6, 2020, 30 Pages. [cited by applicant]
Nguyen, et al., “Variational Bayesian Unlearning”, In Repository of arXiv:2010.12883v1, Oct. 24, 2020, 22 Pages. [cited by applicant]
Nguyen, et al., “Variational Continual Learning”, In Repository of arXiv:1710.10628v1, Oct. 29, 2017, 17 Pages. [cited by applicant]
Northcutt, et al., “Confident Learning: Estimating Uncertainty in Dataset Labels”, In Journal of Artificial Intelligence Research, vol. 70, Apr. 14, 2021, pp. 1373-1411. [cited by applicant]
Nushi, Besmira, “Responsible Machine Learning with Error Analysis”, Retrieved from: https://techcommunity.microsoft.com/t5/ai-machine-learning-blog/responsible-machine-learning-with-error-analysis/ba-p/2141774, Feb. 18,… [cited by applicant]
Pan, et al., “Continual Deep Learning by Functional Regularisation of Memorable Past”, In Proceedings of the 34th International Conference on Neural Information Processing Systems, Dec. 6, 2020, 12 Pages. [cited by applicant]
Parisi, et al., “Continual Lifelong Learning with Neural Networks: A Review”, In Journal of Neural Networks, vol. 113, May 2019, pp. 54-71. [cited by applicant]
Sagawa, et al., “Distributionally Robust Neural Networks for Group Shifts: On The Importance of Regularization for Worst-Case Generalization”, In Repository of arXiv:1911.08731v1, Nov. 20, 2019, 19 Pages. [cited by applicant]
Santurkar, et al., “Editing a Classifier by Rewriting its Prediction Rules”, In Proceedings of Advances in Neural Information Processing Systems, Dec. 6, 2021, 15 Pages. [cited by applicant]
Schwarz, et al., “Progress & Compress: A Scalable Framework for Continual Learning”, In Proceedings of International Conference on Machine Learning, Jul. 3, 2018, 10 Pages. [cited by applicant]
Sinitsin, et al., “Editable Neural Networks”, In Repository of arXiv:2004.00345v1, Apr. 1, 2020, 12 Pages. [cited by applicant]
Slack, et al., “Amazon Presents New Method for “Debugging” Machine Learning Models”, Retrieved from: https://www.amazon.science/blog/amazon-presents-new-method-for-debugging-machine-learning-models, Dec. 7, 2021, 10 Pag… [cited by applicant]
Tanno, et al., “Repairing Neural Networks by Leaving the Right Past Behind”, In Repository of arXiv:2207.04806v1, Jul. 11, 2022, 19 Pages. [cited by applicant]
Torralba, et al., “Unbiased Look at Dataset Bias”, In Proceedings of 24th IEEE Conference on Computer Vision and Pattern Recognition, Jun. 20, 2011, pp. 1521-1528. [cited by applicant]
Xu, et al., “Adversarial Attacks and Defenses in Images, Graphs and Text: A Review”, In International Journal of Automation and Computing, vol. 17, issue 2, Apr. 2020, pp. 151-178. [cited by applicant]
Zenke, et al., “Continual Learning Through Synaptic Intelligence”, In Proceedings of International Conference on Machine Learning, Jul. 17, 2017, 9 Pages. [cited by applicant]
Zhu, et al., “Modifying Memories in Transformer Models”, In Repository of arXiv:2012.00363v1, Dec. 1, 2020, 21 Pages. [cited by applicant]