Processing dynamic data within an adaptive oracle-trained learning system using curated training data for incremental re-training of a predictive model

1. A system comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the system to implement an adaptive learning framework for automatically building and maintaining a predictive model for processing dynamic data, wherein the adaptive learning framework is configured to include:

the predictive model, wherein the model is configured to, using the processor, generate model output from processing a multi-dimensional input data instance received by the adaptive learning framework, the multi-dimensional input data instance having k attributes and represented by a feature vector having k elements, wherein each element in the feature vector represents a value of a corresponding attribute, wherein each element in the feature vector is assigned to a particular cluster of the corresponding attribute using a clustering algorithm wherein the model output includes a judgment and a confidence value representing, a probability that the feature vector belongs to a cluster T of multi-dimensional data instance feature vectors to which it has been assigned according to the judgment;

a training data set from which the predictive model is derived, by the processor, using machine learning; and

a training data manager, wherein the training data manager is configured for, using the processor, curating the training data set; and

a labeled data reservoir configured to, using the processor, store labeled data instances that have been processed as model output by the predictive model, wherein the labeled data reservoir includes a pool of possible training data, wherein a first set of labeled data instances in the labeled data reservoir is not included in the training data; and wherein each labeled data instance is associated with a true label representing the data instance; and

wherein the training data manager is configured to perform, using the processor, operations comprising:

determining whether to update the training data set based at least in part on an accuracy assessment of performance of the predictive model;

in an instance in which the training data set is to be updated,

selecting the first set of labeled data instances from the labeled data reservoir; and

updating the training data set using the first set of labeled data instances.

2. The system of claim 1 , wherein determining whether to update the training data set is based at least in part on analyzing a distribution and quality of the training data set.

3. The system of claim 1 , wherein the accuracy assessment is based on determining whether the confidence value satisfies a confidence threshold value.

4. The system of claim 2 , wherein the predictive model is a classifier predicting to which of a set of predictive categories an input data instance belongs, wherein a true label associated with a labeled data instance identifies the predictive category to which the labeled data instance belongs, and wherein selecting the first set of labeled data instances from the labeled data reservoir is based at least in part on maintaining a class balance within the training data set.

5. The system of claim 1 , wherein the labeled data reservoir includes labeled data instances that are received from multiple sources, and wherein selecting a labeled data instance from the set of labeled data instances comprises:

comparing a source of the labeled data instance with a pre-determined source; and

selecting the labeled data instance in an instance in which the source of the labeled data instance matches the pre-determined source.

6. The system of claim 1 , wherein the operations performed by the processor further comprise:

in response to updating the training data set, determining whether to re-train the predictive model;

in an instance in which the predictive model is re-trained,

generating at least one candidate training data set using the updated training data;

deriving a candidate model using the candidate training data set;

generating an assessment of whether the candidate model performance is improved in comparison to the predictive model performance; and

instantiating the candidate training data set and the candidate model in the adaptive learning framework in an instance in which the candidate model performance is improved in comparison to the predictive model performance.

7. The system of claim 6 , wherein generating the assessment of whether the candidate model performance is improved in comparison to the predictive performance includes A/B testing.

8. The system of claim 7 , wherein generating the assessment comprises calculating a cross-validation between the candidate model performance and the predictive model performance.

9. The system of claim 7 , wherein there are multiple candidate models, and wherein generating the assessment respectively for each of the multiple candidate models is implemented in parallel.

10. A computer program product, stored on a non-transitory computer readable medium, comprising instructions that when executed on one or more computers each having at least one processor cause the one or more computers to implement an adaptive learning framework for automatically building and maintaining a predictive model for processing dynamic data, wherein the adaptive learning framework is configured to include:

the predictive model, wherein the model is configured to, using the processor, generate model output from processing a multi-dimensional input data instance received by the adaptive learning framework, the multi-dimensional input data instance having k attributes and represented by a feature vector having k elements, wherein each element in the feature vector represents a value of a corresponding attribute, wherein each element in the feature vector is assigned to a particular cluster of the corresponding attribute using a clustering algorithm wherein the model output includes a judgment and a confidence value representing, a probability that the feature vector belongs to a cluster T of multi-dimensional data instance feature vectors to which it has been assigned according to the judgment;

a training data set from which the predictive model is derived using machine learning; and

a training data manager, wherein the training data manager is configured for curating the training data set; and

a labeled data reservoir configured to store labeled data instances that have been processed as model output by the predictive model, wherein the labeled data reservoir includes a pool of possible training data, a first set of labeled data instances in the labeled data reservoir is not included in the training data; and wherein each labeled data instance is associated with a true label representing the data; and

wherein the training data manager is configured to perform operations comprising:

determining whether to update the training data set based at least in part on an accuracy assessment of performance of the predictive model;

in an instance in which the training data set is to be updated,

selecting the first set of labeled data instances from the labeled data reservoir; and

updating the training data set using the first set of labeled data instances.

11. The computer program product of claim 10 , wherein determining whether to update the training data set is based at least in part on analyzing a distribution and quality of the training data sets.

12. The computer program product of claim 10 , wherein the accuracy assessment is based on determining whether the confidence value satisfies a confidence threshold value.

13. The computer program product of claim 10 , wherein the predictive model is a classifier predicting to which of a set of predictive categories an input data instance belongs, wherein a true label associated with a labeled data instance identifies the predictive category to which the labeled data instance belongs, and wherein selecting the set of labeled data instances from the labeled data reservoir is based at least in part on maintaining a class balance within the training data.

14. The computer program product of claim 10 , wherein the labeled data reservoir includes labeled data instances that are received from multiple sources, and wherein selecting a labeled data instance from the set of labeled data instances comprises:

comparing a source of the labeled data instance with a pre-determined source; and

selecting the labeled data instance in an instance in which the source of the labeled data instance matches the pre-determined source.

15. The computer program product of claim 10 , wherein the operations further comprise:

in response to updating the training data, determining whether to re-train the predictive model;

in an instance in which the predictive model is re-trained,

generating at least one candidate training data set using the updated training data;

deriving a candidate model using the candidate training data set;

generating an assessment of whether the candidate model performance is improved in comparison to the predictive performance; and

instantiating the candidate training data set and the candidate model in the adaptive learning framework in an instance in which the candidate model performance is improved in comparison to the predictive performance.

16. The computer program product of claim 15 , wherein generating the assessment of whether the candidate model performance is improved in comparison to the predictive performance includes A/B testing.

17. The computer program product of claim 15 , wherein generating the assessment comprises calculating a cross-validation between the candidate model performance and the predictive model performance.

18. The computer program product of claim 15 , wherein there are multiple candidate models, and wherein generating the assessment respectively for each of the multiple candidate models is implemented in parallel.