System for automatic error estimate correction for a machine learning model
A computing device computes curve descriptive values to correct an error estimate of a prediction. A predefined number of times, an input dataset is split into a training dataset and a validation dataset, a predictive model and a domain model are trained, the trained predictive model and the trained domain model are validated, a predictive error value, a residual value, and a domain error value are computed, and each value is stored in output data. A domain threshold value is computed from the stored domain error values. Each predictive error value and each residual value stored in the output data is stored in in-domain output data when a respective domain error value is less than or equal to the computed domain threshold value. Curve descriptive values are computed to describe a relationship between the residual values as a function of the prediction error values stored in the in-domain output data.
1. A non-transitory computer-readable medium having stored thereon computer-readable instructions that when executed by a computing device cause the computing device to:
(A) split an input dataset received from a distributed computer system into a training dataset and a validation dataset, wherein the input dataset includes a plurality of observation vectors;
(B) define a predictive model description and a domain model description based on the input dataset;
(C) train a predictive model associated with the predictive model description with the training dataset subsequent to defining the predictive model description;
(D) train a domain model associated with the domain model description with the training dataset subsequent to defining the domain model description;
(E) validate the trained predictive model with the validation dataset;
(F) validate the trained domain model with the validation dataset;
(G) compute a predictive error value and a residual value from the validated predictive model for each observation vector of the plurality of observation vectors included in the validation dataset;
(H) compute a domain error value from the validated domain model for each observation vector of the plurality of observation vectors included in the validation dataset;
(I) store the computed predictive error value, the computed residual value, and the computed domain error value in output data for each observation vector of the plurality of observation vectors included in the validation dataset;
(J) repeat (A) through (I) a predefined number of times;
(K) compute a domain threshold value using the stored domain error values;
(L) store each predictive error value and each residual value stored in the output data in in-domain output data when a respective stored domain error value is less than or equal to the computed domain threshold value;
(M) compute curve descriptive values based on a type of curve, wherein the curve describes a relationship between the residual values stored in the in-domain output data as a function of the prediction error values stored in the in-domain output data;
(N) output the curve descriptive values to correct an error estimate value of a predicted value of a new observation vector; and
(O) provide an indication of the error estimate value of the predicted value of the new observation vector to the distributed computing system.
2. The non-transitory computer-readable medium of claim 1 , wherein after (L) and before (M), the computer-readable instructions further cause the computing device to:
compute a standard deviation value from a total predictive error value and each predictive error values stored in the in-domain output data;
divide each predictive error value stored in the in-domain output data by the computed standard deviation value to compute a scaled, predictive error value, wherein the predictive error value in (M) is the scaled, predictive error value; and
divide each residual value stored in the in-domain output data by the computed standard deviation value to compute a scaled, residual value, wherein the residual value in (M) is the scaled, residual value.
3. The non-transitory computer-readable medium of claim 2 , wherein the computer-readable instructions further cause the computing device to:
(P) train a second predictive model using the plurality of observation vectors included in the input dataset;
(Q) train a second domain model using the plurality of observation vectors included in the input dataset;
output a first description of the trained second predictive model; and
output a second description of the trained second domain model.
4. The non-transitory computer-readable medium of claim 3 , wherein an average domain threshold value is computed from each domain threshold value computed in (J).
5. The non-transitory computer-readable medium of claim 4 , wherein after (P), the computer-readable instructions further cause the computing device to:
compute a predicted domain error value for a new observation vector using the trained second domain model, wherein the new observation vector is not included in the input dataset; and
when the computed predicted domain error value is less than the computed domain threshold value,
compute a predicted value of the new observation vector and an error estimate value associated with the computed predicted value using the trained second predictive model; and
correct the error estimate value of the computed predicted value using the computed curve descriptive values; and
output the computed predicted value and the corrected error estimate value of the new observation vector.
6. The non-transitory computer-readable medium of claim 5 , wherein, when the computed predicted domain error value is greater than the computed domain threshold value, the computer-readable instructions further cause the computing device to output an out of domain indicator for the new observation vector.
7. The non-transitory computer-readable medium of claim 1 , wherein the computer-readable instructions further cause the computing device to:
(P) train a second predictive model using the plurality of observation vectors included in the input dataset; and
(Q) output a description of the trained second predictive model.
8. The non-transitory computer-readable medium of claim 7 , wherein
after (P), the computer-readable instructions further cause the computing device to:
compute a predicted value of a new observation vector and an error estimate value associated with the computed predicted value using the trained second predictive model, wherein the new observation vector is not included in the input dataset;
correct the error estimate value of the computed predicted value using the computed curve descriptive values; and
output the computed predicted value and the corrected error estimate value of the new observation vector.
9. The non-transitory computer-readable medium of claim 1 , wherein the computer-readable instructions further cause the computing device to:
(P) train a second predictive model using the plurality of observation vectors included in the input dataset;
(Q) train a second domain model using the plurality of observation vectors included in the input dataset;
output a first description of the trained second predictive model; and
output a second description of the trained second domain model.
10. The non-transitory computer-readable medium of claim 9 , wherein an average domain threshold value is computed from each domain threshold value computed in (J).
11. The non-transitory computer-readable medium of claim 10 , wherein after (P), the computer-readable instructions further cause the computing device to:
compute a predicted domain error value for a new observation vector using the trained second domain model, wherein the new observation vector is not included in the input dataset; and
when the computed predicted domain error value is less than the computed domain threshold value,
compute a predicted value of the new observation vector and an error estimate value associated with the computed predicted value using the trained second predictive model; and
correct the error estimate value of the computed predicted value using the computed curve descriptive values; and
output the computed predicted value and the corrected error estimate value of the new observation vector.
12. The non-transitory computer-readable medium of claim 11 , wherein, when the computed predicted domain error value is greater than the computed domain threshold value, the computer-readable instructions further cause the computing device to output an out of domain indicator for the new observation vector.
13. The non-transitory computer-readable medium of claim 1 , wherein the computed curve descriptive values include a slope and a y-intercept value.
14. The non-transitory computer-readable medium of claim 1 , wherein the computed curve descriptive values include a plurality of polynomial coefficients.
15. The non-transitory computer-readable medium of claim 1 , wherein the domain threshold value is a mean value computed from each domain error value stored in the output data.
16. The non-transitory computer-readable medium of claim 1 , wherein the domain threshold value is a predefined percentage value computed from each domain error value stored in the output data.
17. The non-transitory computer-readable medium of claim 1 , wherein a model type of the domain model is a Gaussian process regression model type.
18. The non-transitory computer-readable medium of claim 1 , wherein the domain error value is a distance value computed between observation vectors included in the validation dataset.
19. A computing device comprising:
a processor; and
a non-transitory computer-readable medium operably coupled to the processor, the computer-readable medium having computer-readable instructions stored thereon that, when executed by the processor, cause the computing device to
(A) split an input dataset received from a distributed computer system into a training dataset and a validation dataset, wherein the input dataset includes a plurality of observation vectors;
(B) define a predictive model description and a domain model description based on the input dataset;
(C) train a predictive model associated with the predictive model description with the training dataset subsequent to defining the predictive model description;
(D) train a domain model associated with the domain model description with the training dataset subsequent to defining the domain model description;
(E) validate the trained predictive model with the validation dataset;
(F) validate the trained domain model with the validation dataset;
(G) compute a predictive error value and a residual value from the validated predictive model for each observation vector of the plurality of observation vectors included in the validation dataset;
(H) compute a domain error value from the validated domain model for each observation vector of the plurality of observation vectors included in the validation dataset;
(I) store the computed predictive error value, the computed residual value, and the computed domain error value in output data for each observation vector of the plurality of observation vectors included in the validation dataset;
(J) repeat (A) through (I) a predefined number of times;
(K) compute a domain threshold value using the stored domain error values;
(L) store each predictive error value and each residual value stored in the output data in in-domain output data when a respective stored domain error value is less than or equal to the computed domain threshold value;
(M) compute curve descriptive values based on a type of curve, wherein the curve describes a relationship between the residual values stored in the in-domain output data as a function of the prediction error values stored in the in-domain output data;
(N) output the curve descriptive values to correct an error estimate value of a predicted value of a new observation vector; and
(O) provide an indication of the error estimate value of the predicted value of the new observation vector to the distributed computing system.
20. A method of computing an error estimate correction for a machine learning model, the method comprising:
(A) splitting, by a computing device, an input dataset received from a distributed computer system into a training dataset and a validation dataset, wherein the input dataset includes a plurality of observation vectors;
(B) defining, by the computing device, a predictive model description and a domain model description based on the input dataset;
(C) training, by the computing device, a predictive model with the training dataset;
(D) training, by the computing device, a domain model associated with the predictive model description with the training dataset subsequent to defining the predictive model description;
(E) validating, by the computing device, the trained predictive model with the validation dataset;
(F) validating, by the computing device, the trained domain model with the validation dataset;
(G) computing, by the computing device, a predictive error value and a residual value from the validated predictive model for each observation vector of the plurality of observation vectors included in the validation dataset;
(H) computing, by the computing device, a domain error value from the validated domain model for each observation vector of the plurality of observation vectors included in the validation dataset;
(I) storing, by the computing device, the computed predictive error value, the computed residual value, and the computed domain error value in output data for each observation vector of the plurality of observation vectors included in the validation dataset;
(J) repeating, by the computing device, (A) through (I) a predefined number of times;
(K) computing, by the computing device, a domain threshold value using the stored domain error values;
(L) storing, by the computing device, each predictive error value and each residual value stored in the output data in in-domain output data when a respective stored domain error value is less than or equal to the computed domain threshold value;
(M) computing, by the computing device, curve descriptive values based on a type of curve, wherein the curve describes a relationship between the residual values stored in the in-domain output data as a function of the prediction error values stored in the in-domain output data;
(N) outputting, by the computing device, the curve descriptive values to correct an error estimate value of a predicted value of a new observation vector; and
(O) providing, by the computing device, an indication of the error estimate value of the predicted value of the new observation vector to the distributed computing system.