Automatic selection of high quality training data using an adaptive oracle-trained learning framework
In general, embodiments of the present invention provide systems, methods and computer readable media for an adaptive oracle-trained learning framework for automatically building and maintaining models that are developed using machine learning algorithms. In embodiments, the framework leverages at least one oracle (e.g., a crowd) for automatic generation of high-quality training data to use in deriving a model. Once a model is trained, the framework monitors the performance of the model and, in embodiments, leverages active learning and the oracle to generate feedback about the changing data for modifying training data sets while maintaining data quality to enable incremental adaptation of the model.
1. A computer-implemented method, comprising:
receiving, by a processor, a feature representation of a multi-dimensional data instance having a particular data type and k features, wherein the feature representation includes the k features, each feature of the k features representing a value of a corresponding attribute of the multi-dimensional data instance;
calculating, by a processor and for each feature of the k features, a per operator estimate confidence value to produce a set of k per operator estimate confidence values, each per operator estimate confidence value associated with a unique operator estimate and representing a probability of the feature representation belonging to a specific distribution, each unique operator estimate associated with an operator powered by a statistical model and configured to clean data;
calculating, by a processor, an operator estimation score for the multi-dimensional data instance using the feature representation, the operator estimation score calculated based on the set of k per operator estimate confidence values and calculated using a trained per operator estimator;
calculating, by a processor and for each global data set of a set of n global data sets, a per data set global estimate confidence value to produce a set of n per data set global estimate confidence values, each per data set global estimate confidence value representing a probability of the feature representation belonging to a corresponding global distribution represented by the global data set;
calculating, by a processor, a global estimation score using the feature representation, the global estimation score calculated based on the set of n per data set global estimate confidence values;
updating, by the processor, at least one initial training data set with the multi-dimensional data instance upon determining that the multi-dimensional data instance will be included in the at least one initial training data set based on the operator estimation score and the global estimation score; and
deriving, by the processor, at least one model from the at least one initial training data set.
2. The method of claim 1 , wherein an operator represents a single data cleaning manipulation action based on a particular statistical model.
3. The method of claim 1 , wherein an operator estimate includes multiple operators chained together.
4. The method of claim 1 , wherein calculating the global estimation score comprises:
assigning the multi-dimensional data instance to each of a group of n global data sets, wherein each global data set includes data instances having the particular type, and wherein each of the global data sets respectively represents a set of data instances that were collected during a corresponding time window within a group of pre-defined time windows;
generating a set of n per data set global estimate confidence values, each per data set global estimate confidence value being associated with a corresponding global data set; and
calculating the global estimation score using the set of n per data set global estimate confidence values.
5. The method of claim 4 , wherein generating a data set global estimate confidence value for a dataset comprises:
generating, using a classifier trained on the data set distribution, a per data set global estimate confidence value by estimating a probability that the multi-dimensional data instance belongs to the global distribution of the data set.
6. The method of claim 1 , wherein the feature representation is generated automatically from a pool of unlabeled data instances using a distribution-based feature analysis.
7. The method of claim 6 , wherein the distribution-based feature analysis is dynamic clustering.
8. A computer program product, stored on a non-transitory computer readable medium, comprising instructions that when executed on one or more computers cause the one or more computers to:
receive a feature representation of a multi-dimensional data instance having a particular data type and k features, wherein the feature representation includes the k features, each feature of the k features representing a value of a corresponding attribute of the multi-dimensional data instance;
calculate, for each feature of the k features, a per operator estimate confidence value to produce a set of k per operator estimate confidence values, each per operator estimate confidence value associated with a unique operator estimate and representing a probability of the feature representation belonging to a specific distribution, each unique operator estimate associated with an operator powered by a statistical model and configured to clean data;
calculate an operator estimation score for the multi-dimensional data instance using the feature representation, the operator estimation score calculated based on the set of k per operator estimate confidence values and calculated using a trained operator estimator;
calculate, for each global data set of a set of n global data sets, a per data set global estimate confidence value to produce a set of n per data set global estimate confidence values, each per data set global estimate confidence value representing a probability of the feature representation belonging to a corresponding global distribution represented by the global data set;
calculate a global estimation score for the multi-dimensional data instance using the feature representation, wherein calculating the global estimation score is based on the set of n per data set global estimate confidence values;
update at least one initial training data set with the multi-dimensional data instance upon determining that the multi-dimensional data instance will be included in the at least one initial training data set based on the operator estimation score and the global estimation score; and
derive at least one model from the at least one initial training data set.
9. The computer program product of claim 8 , wherein an operator represents a single data cleaning manipulation action based on a particular statistical model.
10. The computer program product of claim 8 , wherein an operator estimate includes multiple operators chained together.
11. The computer program product of claim 8 , wherein calculating the global estimation score comprises:
assigning the multi-dimensional data instance to each of a group of n global data sets, wherein each global data set includes data instances having the particular type, and wherein each of the global data sets respectively represents a set of data instances that were collected during a corresponding time window within a group of pre-defined time windows;
generating a set of n per data set global estimate confidence values, each per data set global estimate confidence value being associated with a corresponding global data set; and
calculating the global estimation score using the set of n per data set global estimate confidence values.
12. The computer program product of claim 11 , wherein generating a data set global estimate confidence value for a dataset comprises:
generating, using a classifier trained on the data set distribution, a per data set global estimate confidence value by estimating a probability that the multi-dimensional data instance belongs to the global distribution of the data set.
13. The computer program product of claim 8 , wherein the feature representation is generated automatically from a pool of unlabeled data instances using a distribution-based feature analysis.
14. The computer program product of claim 13 , wherein the distribution-based feature analysis is dynamic clustering.
15. An apparatus comprising one or more processors and one or more non-transitory storage mediums comprising instructions that, when executed by the one or more processors, cause the apparatus to:
receive a feature representation of a multi-dimensional data instance having a particular data type and k features, wherein the feature representation includes the k features, each feature of the k features representing a value of a corresponding attribute of the multi-dimensional data instance;
calculate, for each feature of the k features, a per operator estimate confidence value to produce a set of k per operator estimate confidence values, each per operator estimate confidence value associated with a unique operator estimate and representing a probability of the feature representation belonging to a specific distribution, each unique operator estimate associated with an operator powered by a statistical model and configured to clean data;
calculate an operator estimation score for the multi-dimensional data instance using the feature representation, the operator estimation score calculated based on the set of k per operator estimate confidence values and calculated using a trained operator estimator;
calculate, for each global data set of a set of n global data sets, a per data set global estimate confidence value to produce a set of n per data set global estimate confidence values, each per data set global estimate confidence value representing a probability of the feature representation belonging to a corresponding global distribution represented by the global data set;
calculate a global estimation score for the multi-dimensional data instance using the feature representation, wherein calculating the global estimation score is based on the set of n per data set global estimate confidence values;
update at least one initial training data set with the multi-dimensional data instance upon determining that the multi-dimensional data instance will be included in the at least one initial training data set based on the operator estimation score and the global estimation score; and
derive at least one model from the at least one initial training data set.
16. The apparatus of claim 15 , wherein an operator performs a single data cleaning manipulation action based on a particular statistical model.
17. The apparatus of claim 15 , wherein an operator estimate includes multiple operators chained together.
18. The apparatus of claim 15 , wherein calculating the global estimation score comprises:
assigning the multi-dimensional data instance to each of a group of n global data sets, wherein each global data set includes data instances having the particular type, and wherein each of the global data sets respectively represents a set of data instances that were collected during a corresponding time window within a group of pre-defined time windows;
generating a set of n per data set global estimate confidence values, each per data set global estimate confidence value being associated with a corresponding global data set; and
calculating the global estimation score using the set of n per data set global estimate confidence values.
19. The apparatus of claim 18 , wherein generating a data set global estimate confidence value for a dataset comprises:
generating, using a classifier trained on the data set distribution, a per data set global estimate confidence value by estimating a probability that the multi-dimensional data instance belongs to the global distribution of the data set.
20. The apparatus of claim 15 , wherein the feature representation is generated automatically from a pool of unlabeled data instances using a distribution-based feature analysis.
21. The apparatus of claim 20 , wherein the distribution-based feature analysis is dynamic clustering.