Target variable distribution-based acceptance of machine learning test data sets
Respective statistical distributions of a target variable within a proposed training data set and a proposed test data set for a machine learning model are obtained. A metric indicative of the difference between the two statistical distributions is computed. The difference metric is used to determine whether the proposed test data set is acceptable to evaluate the machine learning model.
1. A computer-implemented method, comprising:
obtaining an indication of a first data set;
determining available computational resources for one or more computing devices to perform an algorithm to be used to determine acceptability of the first data set as an input data set for a machine learning model;
identifying, based at least in part on a model type of the machine learning model, the determined available computational resources and the first data set, the algorithm to be used to determine acceptability of the first data set as the input data set for the machine learning model; and
causing to be presented, via one or more programmatic interfaces, one or more results obtained using the algorithm, wherein the one or more results indicate a difference in one or more statistical properties between the first data set and a second data set.
2. The computer-implemented method as recited in claim 1 , wherein the first data set is a test data set, and wherein the second data is a training data set.
3. The computer-implemented method as recited in claim 1 , wherein the model type comprises one or more of: (a) a linear regression model type or (b) a logistic regression model type.
4. The computer-implemented method as recited in claim 1 , wherein the algorithm comprises one or more of: (a) a chi-squared test, (b) Welch's t-test, (c) a Kullback-Leibler divergence based algorithm, or (d) an algorithm based on a Kolmogorov-Smirnoff statistic.
5. The computer-implemented method as recited in claim 1 , wherein said identifying the algorithm is based at least in part on a resource constraint indicated via a programmatic interface.
6. The computer-implemented method as recited in claim 1 , further comprising:
causing to be presented, via the one or more programmatic interfaces, an indication of a threshold for determining acceptability of the first data set.
7. The computer-implemented method as recited in claim 1 , further comprising:
causing to be presented, via the one or more programmatic interfaces, a recommendation of an algorithm to obtain a different data set to be used as input for the machine learning model.
8. A system, comprising:
one or more computing devices;
wherein the one or more computing devices include instructions that upon execution on or across the one or more computing devices cause the one or more computing devices to:
obtain an indication of a first data set;
determine available computational resources for a further one or more computing devices to perform an algorithm to be used to determine acceptability of the first data set as an input data set for a machine learning model;
identify, based at least in part on a model type of the machine learning model, the determined available computational resources and the first data set, the algorithm to be used to determine acceptability of the first data set as the input data set for the machine learning model; and
cause to be presented, via one or more programmatic interfaces, one or more results obtained using the algorithm, wherein the one or more results indicate a difference in one or more statistical properties between the first data set and a second data set.
9. The system as recited in claim 8 , wherein the first data set is a data set to be used to evaluate the machine learning model.
10. The system as recited in claim 8 , wherein the model type comprises one or more of: (a) a linear regression model type or (b) a logistic regression model type.
11. The system as recited in claim 8 , wherein the algorithm comprises one or more of: (a) a chi-squared test, (b) Welch's t-test, (c) a Kullback-Leibler divergence based algorithm, or (d) an algorithm based on a Kolmogorov-Smirnoff statistic.
12. The system as recited in claim 8 , wherein the one or more computing devices include further instructions that upon execution on or across the one or more computing devices further cause the one or more computing devices to:
initiate a modification of the first data set in response to a determination that the first data set does not satisfy an acceptability criterion for use as an input data set for the machine learning model.
13. The system as recited in claim 8 , wherein the one or more computing devices include further instructions that upon execution on or across the one or more computing devices further cause the one or more computing devices to:
obtain, via the one or more programmatic interfaces, an indication of a threshold for determining acceptability of the first data set.
14. The system as recited in claim 8 , wherein the one or more computing devices include further instructions that upon execution on or across the one or more computing devices further cause the one or more computing devices to:
cause to be presented, via the one or more programmatic interfaces, an indication of (a) a type of difference metric used to determine acceptability of the first data set and (b) a value of the difference metric obtained using the algorithm.
15. One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more processors cause the one or more processors to:
obtain an indication of a first data set;
determine available computational resources for one or more computing devices to perform an algorithm to be used to determine acceptability of the first data set as an input data set for a machine learning model;
identify, based at least in part on a model type of a machine learning model, the determined available computational resources and the first data set, the algorithm to be used to determine acceptability of the first data set as the input data set for a machine learning model; and
cause to be presented, via one or more programmatic interfaces, one or more results obtained using the algorithm, wherein the one or more results indicate a difference in one or more statistical properties between the first data set and a second data set.
16. The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein the first data set is a data set to be used to train the machine learning model.
17. The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein the model type comprises one or more of: (a) a linear regression model type or (b) a logistic regression model type.
18. The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein the algorithm comprises one or more of: (a) a chi-squared test, (b) Welch's t-test, (c) a Kullback-Leibler divergence based algorithm, or (d) an algorithm based on a Kolmogorov-Smirnoff statistic.
19. The one or more non-transitory computer-accessible storage media as recited in claim 15 , storing further instructions that when executed on or across the one or more processors further cause the one or more processors to:
identify the algorithm based at least in part on an assumed statistical distribution of values of a variable included in the first data set.
20. The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein the one or more results comprise one or more of: (a) a histogram or (b) a cumulative distribution frequency.