Evaluation of modeling algorithms with continuous outputs
Certain aspects involve evaluating modeling algorithms whose outputs can impact machine-implemented operating environments. For instance, a computing system generates, from a comparison of a set of estimated attribute values of an attribute to a set of validation attribute values of the attribute, a discretized evaluation dataset with data values in multiple categories. The computing system computes, for a modeling algorithm used to generate the estimated attribute values, an evaluation metric. The computing system provides a host computing system with access to the evaluation metric, one or more modeling outputs generated with the modeling algorithm, or both. Providing one or more of these outputs to the host computing system can facilitate modifying one or more machine-implemented operations.
1 . A system comprising:
a data repository storing data samples having values of variables for input to a machine-learning model for risk assessment for an entity,
an external-facing subsystem configured for preventing a host server system from accessing the data repository via a data network, and
an evaluation system configured for:
accessing (a) an estimated dataset having a set of estimated values of an attribute that is a continuous variable, the estimated dataset generated by applying the machine-learning model to an input dataset of the data samples and (b) a validation dataset having a set of validation values of the attribute, the set of validation values respectively being known values corresponding to the set of estimated values generated by the machine-learning model,
generating, from a comparison of the estimated dataset and the validation dataset to an outcome of interest, a discretized evaluation dataset with data values in multiple categories, the discretized evaluation dataset comprising a set of categories in a classification matrix and a number of instances in each category, the set of categories including a true positive category, a true negative category, a false positive category, and a false negative category,
computing, for the machine-learning model, an evaluation metric based on a comparison of data values from different categories of the discretized evaluation dataset, the evaluation metric indicating an accuracy of the machine-learning model, and
providing the host server system with access to (a) the evaluation metric or (b) a modeling output generated with the machine-learning model which indicates a risk level associated with the entity, causing the host server system to allow or prevent the entity to access to a restricted function of a computing environment, based on the modeling output, wherein generating the discretized evaluation dataset comprises:
identifying a first category for the discretized evaluation dataset indicating a match between estimated attribute values and validation attribute values with respect to the outcome of interest;
identifying a second category for the discretized evaluation dataset indicating a mismatch between estimated attribute values and validation attribute values with respect to the outcome of interest;
determining, from the comparison of the estimated dataset and the validation dataset to the outcome of interest, a number of matches in the first category and a number of mismatches in the second category; and
outputting the discretized evaluation dataset having the first category with the number of matches and the second category with the number of mismatches.
2 . The system of claim 1 , wherein:
the outcome of interest comprises the attribute having a value greater than a threshold attribute value,
the match comprises both a first estimated attribute value and a first validation attribute value being greater than the threshold attribute value, the first validation attribute value corresponding to the first estimated attribute value,
the mismatch comprises one of a second estimated attribute value and a second validation attribute value being greater than the threshold attribute value and another of the second estimated attribute value and the second validation attribute value being less than the threshold attribute value, the second validation attribute value corresponding to the second estimated attribute value.
3 . The system of claim 1 , wherein:
the outcome of interest comprises the attribute having a value less than a threshold attribute value,
the match comprises both a first estimated attribute value and a first validation attribute value being less than the threshold attribute value, the first validation attribute value corresponding to the first estimated attribute value,
the mismatch comprises one of a second estimated attribute value and a second validation attribute value being greater than the threshold attribute value and another of the second estimated attribute value and the second validation attribute value being less than the threshold attribute value, the second validation attribute value corresponding to the second estimated attribute value.
4 . The system of claim 1 , wherein:
the first category comprises the true positive category and the true negative category, and
the second category comprises the false positive category and the false negative category.
5 . The system of claim 1 , wherein computing the evaluation metric comprises computing a percentage of matches within a sum of the matches in the first category and the mismatches in the second category.
6 . The system of claim 1 , wherein the allowing or preventing the access comprises modifying a functionality of an online interface provided to a user device associated with the entity.
7 . A method comprising:
generating, by a server system and from a comparison of an estimated dataset and a validation dataset to an outcome of interest, a discretized evaluation dataset with data values in multiple categories, the discretized evaluation dataset comprising a set of categories in a classification matrix and a number of instances in each category, the set of categories including a true positive category, a true negative category, a false positive category, and a false negative category, the estimated dataset having a set of estimated attribute values of an attribute and the validation dataset having a set of validation attribute values of the attribute, the set of validation attribute values respectively corresponding to the set of estimated attribute values;
computing, by the server system, an evaluation metric based on a comparison of data values from different categories of the discretized evaluation dataset, the evaluation metric indicating an accuracy of a modeling algorithm; and
providing a host computing system with access to (a) the evaluation metric or (b) a modeling output generated with the modeling algorithm, wherein providing the host computing system with access to the one or more of (a) the evaluation metric and (b) the modeling output causes the host computing system to modify a host system operation which indicates a risk level associated with an entity, allowing or preventing the entity to access to a restricted function of a computing environment, based the modeling output, wherein generating the discretized evaluation dataset comprises:
identifying a first category for the discretized evaluation dataset indicating a match between estimated attribute values and validation attribute values with respect to the outcome of interest;
identifying a second category for the discretized evaluation dataset indicating a mismatch between estimated attribute values and validation attribute values with respect to the outcome of interest;
determining, from the comparison of the estimated dataset and the validation dataset to the outcome of interest, a number of matches in the first category and a number of mismatches in the second category; and
outputting the discretized evaluation dataset having the first category with the number of matches and the second category with the number of mismatches.
8 . The method of claim 7 , wherein:
the outcome of interest comprises the attribute having a value greater than a threshold attribute value,
the match comprises both a first estimated attribute value and a first validation attribute value being greater than the threshold attribute value, the first validation attribute value corresponding to the first estimated attribute value,
the mismatch comprises one of a second estimated attribute value and a second validation attribute value being greater than the threshold attribute value and another of the second estimated attribute value and the second validation attribute value being less than the threshold attribute value, the second validation attribute value corresponding to the second estimated attribute value.
9 . The method of claim 7 , wherein:
the outcome of interest comprises the attribute having a value less than a threshold attribute value,
the match comprises both a first estimated attribute value and a first validation attribute value being less than the threshold attribute value, the first validation attribute value corresponding to the first estimated attribute value,
the mismatch comprises one of a second estimated attribute value and a second validation attribute value being greater than the threshold attribute value and another of the second estimated attribute value and the second validation attribute value being less than the threshold attribute value, the second validation attribute value corresponding to the second estimated attribute value.
10 . The method of claim 7 , wherein:
the first category comprises the true positive category and the true negative category, and
the second category comprises the false positive category and the false negative category.
11 . The method of claim 10 , wherein computing the evaluation metric comprises computing a percentage of matches within a sum of the matches in the first category and the mismatches in the second category.
12 . The method of claim 7 , wherein:
the evaluation metric comprises a one-tail accuracy or a classification accuracy.
13 . The method of claim 7 , wherein the allowing or preventing the access comprises modifying a functionality of an online interface provided to a user device associated with the entity.
14 . A non-transitory computer-readable medium having program code stored thereon, wherein the program code, when executed by one or more processing devices, configures the one or more processing devices to perform operations comprising:
generating, from a comparison of an estimated dataset and a validation dataset to an outcome of interest, a discretized evaluation dataset with data values in multiple categories, the discretized evaluation dataset comprising a set of categories in a classification matrix and a number of instances in each category, the set of categories including a true positive category, a true negative category, a false positive category, and a false negative category, the estimated dataset having a set of estimated attribute values of an attribute and the validation dataset having a set of validation attribute values of the attribute, the set of validation attribute values respectively corresponding to the set of estimated attribute values;
computing, for a modeling algorithm, an evaluation metric based on a comparison of data values from different categories of the discretized evaluation dataset, the evaluation metric indicating an accuracy of the modeling algorithm; and
providing a host computing system with access to (a) the evaluation metric or (b) a modeling output generated with the modeling algorithm, wherein the one or more of (a) the evaluation metric and (b) the modeling output is usable by the host computing system for modifying a host system operation which indicates a risk level associated with an entity, allowing or preventing the entity to access to a restricted function of a computing environment, based on the modeling output, wherein generating the discretized evaluation dataset comprises:
identifying a first category for the discretized evaluation dataset indicating a match between estimated attribute values and validation attribute values with respect to the outcome of interest;
identifying a second category for the discretized evaluation dataset indicating a mismatch between estimated attribute values and validation attribute values with respect to the outcome of interest;
determining, from the comparison of the estimated dataset and the validation dataset to the outcome of interest, a number of matches in the first category and a number of mismatches in the second category; and
outputting the discretized evaluation dataset having the first category with the number of matches and the second category with the number of mismatches.
15 . The non-transitory computer-readable medium of claim 14 , wherein:
the first category comprises the true positive category and the true negative category,
the second category comprises the false positive category and the false negative category, and
computing the evaluation metric comprises computing a percentage of matches within a sum of the matches in the first category and the mismatches in the second category.
16 . The non-transitory computer-readable medium of claim 14 , wherein:
the outcome of interest comprises the attribute having a value less than a threshold attribute value,
the match comprises both a first estimated attribute value and a first validation attribute value being less than the threshold attribute value, the first validation attribute value corresponding to the first estimated attribute value,
the mismatch comprises one of a second estimated attribute value and a second validation attribute value being greater than the threshold attribute value and another of the second estimated attribute value and the second validation attribute value being less than the threshold attribute value, the second validation attribute value corresponding to the second estimated attribute value.
17 . The non-transitory computer-readable medium of claim 14 , wherein:
the first category comprises the true positive category and the true negative category, and
the second category comprises the false positive category and the false negative category.