Generating an error policy for a machine learning engine
A computer hardware system includes a slice generator and a policy generator and performs the following. The slice generator slices a first dataset including true values and predicted values of a class variable into a plurality of slices each defining a plurality of observations within the first dataset. A first one and another one of the plurality of slices are selected, and a union of observations is generated by adding observations within the selected another one to observations within the selected first one of the plurality of slices. The selecting another one of the plurality of slices and the generating the union is repeated until a number of observations within the union reaches a predetermined value. Using the policy generator and after the number of observations within the union reaches the predetermined value, an error policy is generated. The predicted values were generated by a machine learning engine.
1 . A computer-implemented method within a computer hardware system including a slice generator and a policy generator, comprising:
slicing, using the slice generator, a first dataset including true values and predicted values of a class variable into a plurality of slices each defining a plurality of observations within the first dataset;
generating a union of observations by iteratively adding observations within one or more slices of the plurality of slices to observations within the union of observations until a number of observations within the union of observations reaches a predetermined value;
generating, using the policy generator and after the number of observations within the union of observations reaches the predetermined value, an error policy, wherein the predicted values were generated by a machine learning engine, and wherein the error policy is applied to a second dataset different than the first dataset;
determining performance of the error policy based on an error proportion for the first dataset and an error proportion for the second dataset; and
utilizing the error policy to improve the machine learning engine based on the performance of the error policy.
2 . The method of claim 1 , wherein the error policy includes all feature value conditions used to generate each of the one or more slices.
3 . The method of claim 1 , wherein the machine learning engine is modified based upon the error policy.
4 . The method of claim 1 , wherein the one or more slices are selected based upon a ratio of observations having predicted values not equaling true values for each of the one or more slices.
5 . The method of claim 1 , wherein the predetermined value is an absolute number of observations in the union of observations.
6 . The method of claim 1 , wherein the predetermined value is a percentage of observations in the first dataset.
7 . The method of claim 1 , wherein the error policy is applied to the second dataset different than the first dataset to generate a second union of observations for the second dataset.
8 . A computer hardware system comprising at least one hardware processor configured to perform the following executable operations:
slicing a first dataset including true values and predicted values of a class variable into a plurality of slices each defining a plurality of observations within the first dataset;
generating a union of observations by iteratively adding observations within one or more slices of the plurality of slices to observations within the union of observations until a number of observations within the union of observations reaches a predetermined value;
generating, after the number of observations within the union of observations reaches the predetermined value, an error policy, wherein the predicted values were generated by a machine learning engine, and wherein the error policy is applied to a second dataset different than the first dataset;
determining performance of the error policy based on an error proportion for the first dataset and an error proportion for the second dataset; and
utilizing the error policy to improve the machine learning engine based on the performance of the error policy.
9 . The system of claim 8 , wherein the error policy includes all feature value conditions used to generate each of the one or more slices.
10 . The system of claim 8 , wherein the machine learning engine is modified based upon the error policy.
11 . The system of claim 8 , wherein the one or more slices are selected based upon a ratio of observations having predicted values not equaling true values for each of the one or more slices.
12 . The system of claim 8 , wherein the predetermined value is an absolute number of observations in the union of observations.
13 . The system of claim 8 , wherein the predetermined value is a percentage of observations in the first dataset.
14 . The system of claim 8 , wherein the error policy is applied to the second dataset different than the first dataset to generate a second union of observations for the second dataset.
15 . A computer program product, comprising:
a computer readable storage medium having stored therein program code for training a training dataset,
the program code, which when executed by a computer hardware system including a slice generator and a policy generator, causes the computer hardware system to perform:
slicing, using the slice generator, a first dataset including true values and predicted values of a class variable into a plurality of slices each defining a plurality of observations within the first dataset;
generating a union of observations by iteratively adding observations within one or more slices of the plurality of slices to observations within the union of observations until a number of observations within the union of observations reaches a predetermined value;
generating, using the policy generator and after the number of observations within the union of observations reaches the predetermined value, an error policy, wherein the predicted values were generated by a machine learning engine, and wherein the error policy is applied to a second dataset different than the first dataset;
determining performance of the error policy based on an error proportion for the first dataset and an error proportion for the second dataset; and
utilizing the error policy to improve the machine learning engine based on the performance of the error policy.
16 . The computer program product of claim 15 , wherein the error policy includes all feature value conditions used to generate each of the one or more slices.
17 . The computer program product of claim 15 , wherein the machine learning engine is modified based upon the error policy.
18 . The computer program product of claim 15 , wherein the one or more slices are selected based upon a ratio of observations having predicted values not equaling true values for each of the one or more slices.
19 . The computer program product of claim 15 , wherein the predetermined value is one of an absolute number of observations in the union of observations or as a percentage of observations in the first dataset.
20 . The computer program product of claim 15 , wherein the error policy is applied to the second dataset different than the first dataset to generate a second union of observations for the second dataset.