IP Library › Granted Patent US 12,499,309
Granted Patent B1
US 12,499,309 · App. 17/740,871 · Granted Dec 16, 2025

Programmatically updating machine learning models

Inventors: Elizabeth Dekeyser (Arlington, MA); Nicholas Comment (Cambridge, MA); Shermin Pei (Somerville, MA)
Assignee: Amazon Technologies, Inc.
G06F40/279G06F18/214G06F18/217G06N20/20
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,499,309
App. No.
17/740,871
Filed
May 10, 2022
Granted
Dec 16, 2025
Kind
B1
Art Unit
2658
USPC
704/9
Abstract

The present disclosure provide techniques for automated machine learning (ML) training based on updating training data based on model performance. A system can determine a sample pattern from input data to be used for training a model. The system determines a training dataset and a test dataset including multiple samples corresponding to the sample pattern. After training a model using the training dataset, the system tests the model using the test dataset. The system determines that the trained model's performance with respect to a sample pattern is unsatisfactory, and retrains the model using an updated training dataset that includes additional samples corresponding to the sample pattern.

Claims (106)

1 . A computer-implemented method, comprising:

receiving input data to be used to train a machine learning model, the input data representing at least a first sample and a second sample different than the first sample;

determining that first words represented in the first sample are arranged in accordance with a first natural language phrase pattern;

determining that second words represented in the second sample are arranged in accordance with to a second natural language phrase pattern, the second natural language phrase pattern being different than the first natural language phrase pattern;

determining a first training dataset including a first plurality of samples corresponding to the first natural language phrase pattern and a second plurality of samples corresponding to the second natural language phrase pattern;

determining a first test dataset including a third plurality of samples corresponding to the first natural language phrase pattern and a fourth plurality of samples corresponding to the second natural language phrase pattern;

determining a first additional dataset including a fifth plurality of samples corresponding to the first natural language phrase pattern, the fifth plurality of samples being different than the first plurality of samples, the first additional dataset to be used to update the first training dataset based on model performance;

determining a first trained model using the first training dataset, the first trained model configured to perform a prediction task;

determining, using the first test dataset, that processing by the first trained model with respect to the first natural language phrase pattern is unsatisfactory and that processing by the first trained model with respect to the second natural language phrase pattern is satisfactory;

based at least in part on the processing by the first trained model with respect to the first natural language phrase pattern being unsatisfactory and the processing by the first trained model with respect to the second natural language phrase pattern being satisfactory, determining a second training dataset including the first plurality of samples, the second plurality of samples, and at least a first additional sample from the first additional dataset; and

determining a second trained model using the second training dataset, the second trained model configured to perform the prediction task.

2 . The computer-implemented method of claim 1 , further comprising:

receiving a prediction accuracy condition associated with the first natural language phrase pattern, the prediction accuracy condition indicative of a minimum prediction accuracy to be satisfied by a trained model with respect to generating predictions for the first natural language phrase pattern; and

processing the first test dataset using the first trained model to determine first prediction data, the first prediction data including at least a first prediction corresponding to a first sample of the third plurality of samples,

wherein determining that the processing by the first trained model with respect to the first natural language phrase pattern is unsatisfactory comprises processing the first prediction data with respect to the prediction accuracy condition.

3 . The computer-implemented method of claim 1 , further comprising:

receiving a second test dataset including a sixth plurality of samples that the second trained model is previously configured to process;

determining, using the second test dataset, that processing by the second trained model with respect to the sixth plurality of samples is satisfactory; and

based at least in part on the processing by the second trained model with respect to the sixth plurality of samples being satisfactory, using the second trained model to perform the prediction task on subsequently received inputs.

4 . The computer-implemented method of claim 1 , further comprising:

receiving a second test dataset including a sixth plurality of samples that the second trained model is previously configured to process;

determining a third natural language phrase pattern of data represented in the second test dataset;

determining, using the second test dataset, that processing by the second trained model with respect to the third natural language phrase pattern is unsatisfactory;

based at least in part on the processing by the second trained model with respect to the third natural language phrase pattern being unsatisfactory, determining a third training dataset including the second training dataset and at least one additional sample corresponding to the third natural language phrase pattern; and

determining a third trained model using the third training dataset, the third trained model configured to perform the prediction task.

5 . The computer-implemented method of claim 1 , further comprising:

determining a third training dataset including the first plurality of samples, the second plurality of samples, and a plurality of additional samples from the first additional dataset, wherein the third training dataset includes more samples than the second training dataset;

determining a third trained model using the third training dataset, the third trained model configured to perform the prediction task;

determining, using the first test dataset, that processing by the second trained model with respect to the first natural language phrase pattern is satisfactory;

determining, using the first test dataset, that processing by the third trained model with respect to the first natural language phrase pattern is satisfactory; and

based at least in part on the second training dataset including fewer samples than the third training dataset, selecting the second trained model, instead of the third trained model, for further processing.

6 . The computer-implemented method of claim 1 , further comprising:

receiving a second test dataset including a sixth plurality of samples that the second trained model is previously configured to process;

determining a third natural language phrase pattern of data represented in the second test dataset;

determining, using the second test dataset, that processing by the second trained model with respect to the third natural language phrase pattern is unsatisfactory;

based at least in part on the processing by the second trained model with respect to the third natural language phrase pattern being unsatisfactory, determining a rule to be used to process subsequently received inputs corresponding to the third natural language phrase pattern; and

using the second trained model and the rule to perform the prediction task on subsequently received inputs.

7 . The computer-implemented method of claim 1 , further comprising:

determining a third training dataset including a sixth plurality of samples corresponding to the first natural language phrase pattern and a seventh plurality of samples corresponding to the second natural language phrase pattern, wherein the first training dataset includes fewer samples than the third training dataset;

determining a third trained model using the third training dataset, the third trained model configured to perform the prediction task;

determining that the first trained model satisfies a performance metric;

determining that the third trained model satisfies the performance metric; and

selecting the first trained model, instead of the third trained model, based at least in part on the first training dataset including fewer samples than the third training dataset, the first trained model being selected for further processing with respect to the first test dataset.

8 . The computer-implemented method of claim 1 , wherein the input data includes a first natural language input, a second natural language input, and an entity catalog including a plurality of entities, and the method further comprises:

determining a first portion of the first natural language input representing a first entity;

determining a first plurality of natural language inputs using the first natural language input and replacing the first portion with individual entities from the entity catalog, wherein the first plurality of natural language inputs includes at least a third natural language input including a second entity from the plurality of entities;

determining a second portion of the second natural language input representing a third entity;

determining a second plurality of natural language inputs using the second natural language input and replacing the second portion with individual entities from the entity catalog, wherein the second plurality of natural language inputs includes at least a fourth natural language input including a fourth entity from the plurality of entities;

determining the first plurality of samples using the first plurality of natural language inputs;

determining the second plurality of samples using the second plurality of natural language inputs;

determining the third plurality of samples using the first plurality of natural language inputs; and

determining the fourth plurality of samples using the second plurality of natural language inputs.

9 . A system, comprising:

at least one processor; and

at least one memory including instructions that, when executed by the at least one processor, cause the system to:

receive input data to be used to train a machine learning model, the input data representing at least a first sample and a second sample different than the first sample;

determine that first words represented in the first sample are arranged in accordance with to a first natural language phrase pattern;

determine that second words represented in the second sample are arranged in accordance with to a second natural language phrase pattern, the second natural language phrase pattern being different than the first natural language phrase pattern;

determine a first training dataset including a first plurality of samples corresponding to the first natural language phrase pattern and a second plurality of samples corresponding to the second natural language phrase pattern;

determine a first test dataset including a third plurality of samples corresponding to the first natural language phrase pattern and a fourth plurality of samples corresponding to the second natural language phrase pattern;

determine a first additional dataset including a fifth plurality of samples corresponding to the first natural language phrase pattern, the fifth plurality of samples being different than the first plurality of samples, the first additional dataset to be used to update the first training dataset based on model performance;

determine a first trained model using the first training dataset, the first trained model configured to perform a prediction task;

determine, using the first test dataset, that processing by the first trained model with respect to the first natural language phrase pattern is unsatisfactory and that processing by the first trained model with respect to the second natural language phrase pattern is satisfactory;

based at least in part on the processing by the first trained model with respect to the first natural language phrase pattern being unsatisfactory and the processing by the first trained model with respect to the second natural language phrase pattern being satisfactory, determine a second training dataset including the first plurality of samples and at least a first additional sample from the first additional dataset; and

determine a second trained model using the second training dataset, the second trained model configured to perform the prediction task.

10 . The system of claim 9 , wherein the at least one memory includes further instructions that, when executed by the at least one processor, further cause the system to:

receive a second test dataset including a sixth plurality of samples that the second trained model is previously configured to process;

determine, using the second test dataset, that processing by the second trained model with respect to the sixth plurality of samples is satisfactory; and

based at least in part on the processing by the second trained model with respect to the sixth plurality of samples being satisfactory, use the second trained model to perform the prediction task on subsequently received inputs.

11 . The system of claim 9 , wherein the at least one memory includes further instructions that, when executed by the at least one processor, further cause the system to:

receive a second test dataset including a sixth plurality of samples that the second trained model is previously configured to process;

determine a third natural language phrase pattern of data represented in the second test dataset;

determine, using the second test dataset, that processing by the second trained model with respect to the third natural language phrase pattern is unsatisfactory;

based at least in part on the processing by the second trained model with respect to the third natural language phrase pattern being unsatisfactory, determine a third training dataset including the second training dataset and at least one additional sample corresponding to the third natural language phrase pattern; and

determine a third trained model using the third training dataset, the third trained model configured to perform the prediction task.

12 . The system of claim 9 , wherein the at least one memory includes further instructions that, when executed by the at least one processor, further cause the system to:

determine a third training dataset including the first plurality of samples, the second plurality of samples, and a plurality of additional samples from the first additional dataset, wherein the third training dataset includes more samples than the second training dataset;

determine a third trained model using the third training dataset, the third trained model configured to perform the prediction task;

determine, using the first test dataset, that processing by the second trained model with respect to the first natural language phrase pattern is satisfactory;

determine, using the first test dataset, that processing by the third trained model with respect to the first natural language phrase pattern is satisfactory; and

based at least in part on the second training dataset including fewer samples than the third training dataset, select the second trained model, instead of the third trained model, for further processing.

13 . The system of claim 9 , wherein the at least one memory includes further instructions that, when executed by the at least one processor, further cause the system to:

receive a second test dataset including a sixth plurality of samples that the second trained model is previously configured to process;

determine a third natural language phrase pattern of data represented in the second test dataset;

determine, using the second test dataset, that processing by the second trained model with respect to the third natural language phrase pattern is unsatisfactory;

based at least in part on the processing by the second trained model with respect to the third natural language phrase pattern being unsatisfactory, determine a rule to be used to process subsequently received inputs corresponding to the third natural language phrase pattern; and

use the second trained model and the rule to perform the prediction task on subsequently received inputs.

14 . The system of claim 9 , wherein the at least one memory includes further instructions that, when executed by the at least one processor, further cause the system to:

determine a third training dataset including a sixth plurality of samples corresponding to the first natural language phrase pattern and a seventh plurality of samples corresponding to the second natural language phrase pattern, wherein the first training dataset includes fewer samples than the third training dataset;

determine a third trained model using the third training dataset, the third trained model configured to perform the prediction task;

determine that the first trained model satisfies a performance metric;

determine that the third trained model satisfies the performance metric; and

select the first trained model, instead of the third trained model, based at least in part on the first training dataset including fewer samples than the third training dataset, the first trained model being selected for further processing with respect to the first test dataset.

15 . The system of claim 9 , wherein the input data includes a first natural language input, a second natural language input and an entity catalog including a plurality of entities, and wherein the at least one memory includes further instructions that, when executed by the at least one processor, further cause the system to:

determine a first portion of the first natural language input representing a first entity;

determine a first plurality of natural language inputs using the first natural language input and replacing the first portion with individual entities from the entity catalog, wherein the first plurality of natural language inputs includes at least a third natural language input including a second entity from the plurality of entities;

determine a second portion of the second natural language input representing a third entity;

determine a second plurality of natural language inputs using the second natural language input and replacing the second portion with individual entities from the entity catalog, wherein the second plurality of natural language inputs includes at least a fourth natural language input including a fourth entity from the plurality of entities;

determine the first plurality of samples using the first plurality of natural language inputs;

determine the second plurality of samples using the second plurality of natural language inputs;

determine the third plurality of samples using the first plurality of natural language inputs; and

determine the fourth plurality of samples using the second plurality of natural language inputs.

16 . The system of claim 9 , wherein the at least one memory includes further instructions that, when executed by the at least one processor, further cause the system to:

receive a prediction accuracy condition associated with the first natural language phrase pattern, the prediction accuracy condition indicative of a minimum prediction accuracy to be satisfied by a trained model with respect to generating predictions for the first natural language phrase pattern;

process the first test dataset using the first trained model to determine first prediction data, the first prediction data including at least a first prediction corresponding to a first sample of the third plurality of samples; and

determine that the processing by the first trained model with respect to the first natural language phrase pattern is unsatisfactory at least in part by processing the first prediction data with respect to the prediction accuracy condition.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 10, 2022
From: DEKEYSER, ELIZABETH; COMMENT, NICHOLAS; PEI, SHERMIN
To: AMAZON TECHNOLOGIES, INC.
Reel/Frame 059883/0202 →
References Cited (32)
US 11955134B2 · Amid · 2024 [cited by examiner]
US 12242820B2 · Salaam · 2025 [cited by examiner]
US 20100049590A1 · Anshul · 2010 [cited by examiner]
US 20200302234A1 · Walters · 2020 [cited by examiner]
US 20210224486A1 · Stabler · 2021 [cited by examiner]
US 20210406369A1 · Healy · 2021 [cited by examiner]
US 20210406533A1 · Arroyo · 2021 [cited by examiner]
US 20220108222A1 · Brannon · 2022 [cited by examiner]
US 20230169394A1 · Niu · 2023 [cited by examiner]
US 20230237384A1 · Dowdell · 2023 [cited by examiner]
Amittai Axelrod et al., “Domain Adaptation via Pseudo In-Domain Data Selection.” In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, (ACL), 2011, pp. 355-362. [cited by applicant]
Mikhail Belkin et al., “Reconciling Modern Machine-Learning Practice and the Classical Bias-Variance Trade-Off.” Proceedings of the National Academy of Sciences, vol. 116, No. 32, https://www.pnas.org/doi/epdf/10.1073/p… [cited by applicant]
David A. Cohn et al., “Active Learning with Statistical Models.” Journal of Artificial Intelligence Research 4, 1996, pp. 129-145. AI Foundation and Morgan Kaufmann Publishers. [cited by applicant]
Cody Coleman et al., “Selection Via Proxy: Efficient Data Selection for Deep Learning.” Published as a Conference Paper at ICLR 2020. arXiv preprint arXiv:1906.11829v4, 2020, 25 pages. [cited by applicant]
Matthias Feurer et al., “Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning.” arXiv preprint arXiv:2007.04074v2, 2021, 56 pages. [cited by applicant]
Chelsea Finn et al., “Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks.” In 34th International Conference on Machine Learning, PMLR 70. arXiv preprint arXiv:1703.034003v3, 2017, 13 pages. [cited by applicant]
Yarin Gal et al., “Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning.” In Proceedings of the 33rd International Conference on Machine Learning, (retrieved from https://proceedings.mlr.… [cited by applicant]
Kin He et al., “Automl: A Survey of the State-of-the-Art.” arXiv preprint arXiv:1908.00709v6, 2021, 37 pages. [cited by applicant]
Rishabh K. Iyer et al., “Submodular Optimization with Submodular Cover and Submodular Knapsack Constraints.” In Advances in Neural Information Processing Systems, (retrieved from https://papers.nips.cc/paper/2013/file/a… [cited by applicant]
Christoph Kading et al., “Fine-tuning Deep Neural Networks in Continuous Learning Scenarios.” In ACCV 2016 Workshop on Interpretation and Visualization of Deep Neural Nets, 2016, pp. 588-605. Springer. [cited by applicant]
Zhuoming Liu et al., “Influence Selection for Active Learning.” In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), arXiv preprint arXiv:2108.09331, 2021, 18 pages. [cited by applicant]
Richard C. Moore et al., “Intelligent Selection of Language Model Training Data.” In Proceedings of the ACL 2010 conference short papers, 2010, pp. 220-224. [cited by applicant]
Hung Viet Pham et al., “Problems and Opportunities in Training Deep Learning Software Systems: An Analysis of Variance.” In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, (r… [cited by applicant]
Jacob M. Schreiber et al., “apricot: Submodular selection for data summarization in Python.” In Journal of Machine Learning Research 21, 2020, 6 pages. [cited by applicant]
Burr Settles, “Active Learning Literature Survey.” Computer Sciences Technical Report 1648, University of Wisconsin—Madison, https://research.cs.wisc.edu/techreports/2009/TR1648.pdf, 2009, 46 pages. [cited by applicant]
Noam Shazeer et al., “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.” arXiv preprint arXiv:1701.06538v1, 2017, 19 pages. [cited by applicant]
Anh Truong et al., “Towards Automated Machine Learning: Evaluation and Comparison of AutoML Approaches and Tools.” arXiv preprint arXiv:1908.05557v2, 2019, 9 pages. [cited by applicant]
Zhiqiang Wan et al., “Variational Autoencoder Based Synthetic Data Generation for Imbalanced Learning.” In 2017 IEEE Symposium Series on Computational Intelligence (SSCI), (doi: 10.1109/SSCI.2017.8285168), 2017, pp. 1-7. [cited by applicant]
Kai Wei et al., “Submodular Subset Selection for Large-Scale Speech Training Data.” In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (retrieved from https://citeseerx.ist.psu.ed… [cited by applicant]
Yue Yu et al., “Fine-Tuning Pre-Trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach.” arXiv preprint arXiv:2010.07835v3, 2021, 15 pages. [cited by applicant]
Seniha Esen Yuksel et al., “Twenty Years of Mixture of Experts.” IEEE Transactions on Neural Networks and Learning Systems, vol. 23, No. 8, 2012, pp. 1177-1193. [cited by applicant]
Tianyi Zhang et al., “BERTScore: Evaluating Text Generation with BERT.” arXiv preprint arXiv:1904.09675v3, 2020, 43 pages. [cited by applicant]