Detecting suitability of machine learning models for datasets
Apparatuses, systems, program products, and method are disclosed for detecting suitability of machine learning models for datasets. An apparatus includes a training evaluation module configured to calculate a first statistical data signature for a training data set of a machine learning system using one or more predefined statistical algorithms. An apparatus includes an inference evaluation module configured to calculate a second statistical data signature for an inference data set of a machine learning system using one or more predefined statistical algorithms. An apparatus includes a score module configured to calculate a suitability score describing the suitability of a training data set to an inference data set as a function of a first and a second statistical data signature. An apparatus includes an action module configured to perform an action related to a machine learning system in response to a suitability score satisfying an unsuitability threshold.
1. A system comprising:
a memory device; and
one or more processors configured to execute instructions stored in the memory device to cause the system to perform operations including:
generating, by the one or more processors based on one or more statistical models, a first signature including a first score based on a distribution of values of one or more features of a training data set, the training data set provided as input to train a first model using machine learning;
generating, by the one or more processors based on the one or more statistical models, a second signature including a second score based on a distribution of values of one or more features of an inference data set, the inference data set provided as input to the first model trained based on the training data set using machine learning;
generating, by the one or more processors, a score indicating a suitability between the training data set and the inference data set based on the first score of the first signature and the second score of the second signature;
in response to a determination that the suitability score satisfies a threshold indicating unsuitability of the first model trained using the training data set for making predictions based on the inference data set, generating, by the one or more processors, a message an alert, message, or notification including an indication that the first model trained using the training data set is unsuitable for the inference data set and a recommendation regarding remedial action, and providing the message to a user via a computer network and/or a display device; and
in response to receiving user input indicating approval of the remedial action, replacing the first model with a second machine learning model,
wherein the distribution of values of the one or more features of the inference data set corresponds to a probability distribution of values of the one or more features of the inference data set.
2. The system of claim 1 , wherein the operations further include generating the suitability score on an ongoing basis during processing using machine learning.
3. The system of claim 1 , the suitability score indicating accuracy or predictive performance of the first model.
4. The system of claim 1 , one or more of the training data set or the inference data set comprising feature data sets free of labels.
5. The system of claim 1 , wherein the operations further include generating the first score before analyzing the inference data set.
6. The system of claim 1 , the distribution of values of the one or more features of the training data set and the distribution of values of the one or more features of the inference data set each based on a data structure corresponding to a forest of trees.
7. The system of claim 1 , wherein at least one feature of the one or more features of the training data set includes image data, string data, text data, or graph data, and at least one feature of the one or more features of the inference data set includes image data, string data, text data, or graph data.
8. The system of claim 1 , wherein the first model comprises one or more learning layers of a deep learning system, the training data set comprises data output from a learning layer of the deep learning system preceding the one or more learning layers of the first model during training of the first model, and the inference data set comprises data output from the learning layer of the deep learning system during use of the first model.
9. The system of claim 1 , wherein the remedial action comprises (i) retraining the first model on a second training data set, (ii) replacing the first model with the second machine learning model, or (iii) generating the second machine learning model.
10. The system of claim 9 , wherein generating the second machine learning model comprises performing a hyper-parameter search to choose values of one or more hyper-parameters for a machine learning algorithm and training the second machine learning model using the machine learning algorithm.
11. The system of claim 1 , wherein the remedial action comprises:
determining, based on a policy, that user approval is required to replace the first model with another machine learning model.
12. The system of claim 1 , wherein the probability distribution of values of the one or more features of the inference data set is based on an assignment of the values of the one or more features of the inference data set to a plurality of bins.
13. A method comprising:
performing, by one or more processors, operations including:
generating, by the one or more processors based on one or more statistical models, a first signature including a first score based on a distribution of values of one or more features of a training data set, the training data set provided as input to train a first model using machine learning;
generating, by the one or more processors based on the one or more statistical models, a second signature including a second score based on a distribution of values of one or more features of an inference data set, the inference data set provided as input to the first model trained based on the training data set using machine learning;
generating by the one or more processors a score indicating a suitability between the training data set and the inference data set based on the first score of the first signature and the second score of the second signature;
in response to a determination that the suitability score satisfies a threshold indicating unsuitability of the first model trained using the training data set for making predictions based on the inference data set, generating by the one or more processors a message including an indication that the first model trained using the training data set is unsuitable for the inference data set and a recommendation regarding remedial action, and providing the message to a user via a computer network and/or a display device; and
in response to receiving user input indicating approval of the remedial action, replacing the first model with a second machine learning model,
wherein the distribution of values of the one or more features of the inference data set corresponds to a probability distribution of values of the one or more features of the inference data set.
14. The method of claim 13 , further comprising: generating the suitability score on an ongoing basis during processing using machine learning.
15. The method of claim 13 , the suitability score indicating accuracy or predictive performance of the first model.
16. The method of claim 13 , one or more of the training data set or the inference data set comprising feature data sets free of labels.
17. The method of claim 13 , further comprising: generating the first score before analyzing the inference data set.
18. The method of claim 13 , the distribution of values of the one or more features of the training data set and the distribution of values of the one or more features of the inference data set each based on a data structure corresponding to a forest of trees.
19. The method of claim 13 , wherein at least one feature of the one or more features of the training data set includes image data, string data, text data, or graph data, and at least one feature of the one or more features of the inference data set includes image data, string data, text data, or graph data.
20. The method of claim 13 , wherein the first model comprises one or more learning layers of a deep learning system, the training data set comprises data output from a learning layer of the deep learning system preceding the one or more learning layers of the first model during training of the first model, and the inference data set comprises data output from the learning layer of the deep learning system during use of the first model.
21. The method of claim 13 , wherein the remedial action comprises (i) retraining the first model on a second training data set, (ii) replacing the first model with the second machine learning model, or (iii) generating the second machine learning model.
22. The method of claim 21 , wherein generating the second machine learning model comprises performing a hyper-parameter search to choose values of one or more hyper-parameters for a machine learning algorithm and training the second machine learning model using the machine learning algorithm.
23. The method of claim 13 , wherein the remedial action comprises:
determining, based on a policy, that user approval is required to replace the model with another machine learning model.
24. The method of claim 13 , wherein the probability distribution of values of the one or more features of the inference data set is based on an assignment of the values of the one or more features of the inference data set to a plurality of bins.
25. A non-transitory computer readable medium including one or more instructions stored thereon and executable by one or more processors to perform operations including:
generating, by the one or more processors and based on one or more statistical models, a first signature including a first score based on a distribution of values of one or more features of a training data set, the training data set provided as input to train a first model using machine learning;
generating, by the one or more processors and based on the one or more statistical models, a second signature including a second score based on a distribution of values of one or more features of an inference data set provided as input to the first model trained based on the training data set, the inference data set using machine learning;
generating, by the one or more processors, a score indicating a suitability between the training data set and the inference data set based on the first score of the first signature and the second score of the second signature; and
in response to a determination that the suitability score satisfies a threshold indicating unsuitability of the first model trained using the training data set for making predictions based on the inference data set, generating, by the one or more processors, a message including an indication that the first model trained using the training data set is unsuitable for the inference data set and a recommendation regarding remedial action, and providing the message to a user via a computer network and/or a display device; and
in response to receiving user input indicating approval of the remedial action, replacing the first model with a second machine learning model,
wherein the distribution of values of the one or more features of the inference data set corresponds to a probability distribution of values of the one or more features of the inference data set.
26. The non-transitory computer readable medium of claim 25 , wherein the operations further include generating the suitability score on an ongoing basis during processing using machine learning.