Curating training data for incremental re-training of a predictive model

1. A computer-implemented method for adaptively improving the performance of a current predictive model by curating training data used to derive the current predictive model, the method comprising:

receiving the training data and the current predictive model derived using the training data;

selecting a set of labeled data instances from a labeled data reservoir, wherein the labeled data reservoir includes a pool of possible training data, wherein the set of labeled data instances are not included in the training data, wherein each labeled data instance is associated with a true label representing the instance, and wherein selecting the set of labeled data instances is based on a determination that re-training the model with updated training data likely will result in improved model performance;

generating at least one candidate training data set by updating the training data using the set of labeled data instances;

deriving a candidate model using the candidate training data set;

generating, by a training data manager component, an assessment of whether the candidate model performance is improved from the current model performance; and

instantiating the candidate training data set and the candidate model in an instance in which the candidate model performance is improved from the current model performance.

2. The method of claim 1 , wherein the labeled data reservoir includes data that have been collected continuously over time from input data being processed by the current predictive model.

3. The method of claim 1 , wherein the determination is based at least in part on analyzing the distribution and quality of the training data.

4. The method of claim 3 , wherein the current model is a classifier predicting to which of a set of predictive categories an input data instance belongs, wherein a true label associated with a labeled data instance identifies the predictive category to which the labeled data instance belongs, and wherein selecting the set of labeled data instances from the labeled data reservoir is based at least in part on maintaining a class balance within the training data.

5. The method of claim 1 , wherein generating the candidate training data comprises:

identifying and removing outlier instances.

6. The method of claim 5 , wherein the current model is a classifier predicting to which of a set of predictive categories an input data instance belongs, and wherein selecting the set of labeled data instances from the labeled data reservoir comprises:

identifying and removing outlier instances in one predictive category.

7. The method of claim 1 , wherein the labeled data reservoir includes labeled data instances that are received from multiple sources, and wherein selecting a labeled data instance from the set of labeled data instances comprises:

comparing a source of the labeled data instance with a pre-determined source; and

selecting the labeled data instance in an instance in which the source of the labeled data instance matches the pre-determined source.

8. The method of claim 1 , wherein generating at least one candidate training data set is based on a greedy algorithm, the generating comprising:

generating a first candidate training data set by adding a first subset of the labeled data instances to the training data; and

generating a second candidate training data set by adding a second subset of the labeled data instances to the first candidate training data set.

9. The method of claim 1 , wherein generating at least one candidate training data set is based on a non-greedy algorithm, the generating comprising:

replacing the training data with a subset of the labeled data instances.

10. The method of claim 1 , wherein generating an assessment of whether the candidate model performance is improved from the current model performance includes A/B testing.

11. The method of claim 10 , wherein generating the assessment comprises calculating a cross-validation between the candidate model performance and the current model performance.

12. The method of claim 10 , wherein there are multiple candidate models, and wherein generating the assessment for each of the multiple candidate models is implemented in parallel.

13. A computer program product, stored on a non-transitory computer readable medium, comprising instructions that when executed on one or more computers cause the one or more computers to perform operations comprising:

receiving the training data and the current predictive model derived using the training data;

selecting a set of labeled data instances from a labeled data reservoir, wherein the labeled data reservoir includes a pool of possible training data, wherein the set of labeled data instances are not included in the training data, wherein each labeled data instance is associated with a true label representing the instance, and wherein selecting the set of labeled data instances is based on a determination that re-training the model with updated training data likely will result in improved model performance;

generating at least one candidate training data set by updating the training data using the set of labeled data instances;

deriving a candidate model using the candidate training data set;

generating, by a training data manager component, an assessment of whether the candidate model performance is improved from the current model performance; and

instantiating the candidate training data set and the candidate model in an instance in which the candidate model performance is improved from the current model performance.

14. The computer program product of claim 13 , wherein the labeled data reservoir includes data that have been collected continuously over time from input data being processed by the current predictive model.

15. The computer program product of claim 13 , wherein the determination is based at least in part on analyzing the distribution and quality of the training data.

16. The computer program product of claim 15 , wherein the current model is a classifier predicting to which of a set of predictive categories an input data instance belongs, wherein a true label associated with a labeled data instance identifies the predictive category to which the labeled data instance belongs, and wherein selecting the set of labeled data instances from the labeled data reservoir is based at least in part on maintaining a class balance within the training data.

17. The computer program product of claim 13 , wherein generating the candidate training data comprises:

identifying and removing outlier instances.

18. The computer program product of claim 17 , wherein the current model is a classifier predicting to which of a set of predictive categories an input data instance belongs, and wherein selecting the set of labeled data instances from the labeled data reservoir comprises:

identifying and removing outlier instances in one predictive category.

19. The computer program product of claim 13 , wherein the labeled data reservoir includes labeled data instances that are received from multiple sources, and wherein selecting a labeled data instance from the set of labeled data instances comprises:

comparing a source of the labeled data instance with a pre-determined source; and

selecting the labeled data instance in an instance in which the source of the labeled data instance matches the pre-determined source.

20. The computer program product of claim 13 , wherein generating at least one candidate training data set is based on a greedy algorithm, the generating comprising:

generating a first candidate training data set by adding a first subset of the labeled data instances to the training data; and

generating a second candidate training data set by adding a second subset of the labeled data instances to the first candidate training data set.

21. The computer program product of claim 13 , wherein generating at least one candidate training data set is based on a non-greedy algorithm, the generating comprising:

replacing the training data with a subset of the labeled data instances.

22. The computer program product of claim 13 , wherein generating an assessment of whether the candidate model performance is improved from the current model performance includes A/B testing.

23. The computer program product of claim 22 , wherein generating the assessment comprises calculating a cross-validation between the candidate model performance and the current model performance.

24. The computer program product of claim 22 , wherein there are multiple candidate models, and wherein generating the assessment for each of the multiple candidate models is implemented in parallel.

25. A system, comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

receiving the training data and the current predictive model derived using the training data;

selecting a set of labeled data instances from a labeled data reservoir, wherein the labeled data reservoir includes a pool of possible training data, wherein the set of labeled data instances are not included in the training data, wherein each labeled data instance is associated with a true label representing the instance, and wherein selecting the set of labeled data instances is based on a determination that re-training the model with updated training data likely will result in improved model performance;

generating at least one candidate training data set by updating the training data using the set of labeled data instances;

deriving a candidate model using the candidate training data set;

generating, by a training data manager component, an assessment of whether the candidate model performance is improved from the current model performance; and

instantiating the candidate training data set and the candidate model in an instance in which the candidate model performance is improved from the current model performance.

26. The system of claim 25 , wherein the labeled data reservoir includes data that have been collected continuously over time from input data being processed by the current predictive model.

27. The system of claim 25 wherein the determination is based at least in part on analyzing the distribution and quality of the training data.

28. The system of claim 27 , wherein the current model is a classifier predicting to which of a set of predictive categories an input data instance belongs, wherein a true label associated with a labeled data instance identifies the predictive category to which the labeled data instance belongs, and wherein selecting the set of labeled data instances from the labeled data reservoir is based at least in part on maintaining a class balance within the training data.

29. The system of claim 25 , wherein generating the candidate training data comprises:

identifying and removing outlier instances.

30. The system of claim 29 , wherein the current model is a classifier predicting to which of a set of predictive categories an input data instance belongs, and wherein selecting the set of labeled data instances from the labeled data reservoir comprises:

identifying and removing outlier instances in one predictive category.

31. The system of claim 25 , wherein the labeled data reservoir includes labeled data instances that are received from multiple sources, and wherein selecting a labeled data instance from the set of labeled data instances comprises:

comparing a source of the labeled data instance with a pre-determined source; and

selecting the labeled data instance in an instance in which the source of the labeled data instance matches the pre-determined source.

32. The system of claim 25 , wherein generating at least one candidate training data set is based on a greedy algorithm, the generating comprising:

generating a first candidate training data set by adding a first subset of the labeled data instances to the training data; and

generating a second candidate training data set by adding a second subset of the labeled data instances to the first candidate training data set.

33. The system of claim 25 , wherein generating at least one candidate training data set is based on a non-greedy algorithm, the generating comprising:

replacing the training data with a subset of the labeled data instances.

34. The system of claim 25 , wherein generating an assessment of whether the candidate model performance is improved from the current model performance includes A/B testing.

35. The system of claim 34 , wherein generating the assessment comprises calculating a cross-validation between the candidate model performance and the current model performance.

36. The system of claim 34 , wherein there are multiple candidate models, and wherein generating the assessment for each of the multiple candidate models is implemented in parallel.