Automatic thresholding for classification models

1 . A system for selecting a classification threshold value usable to control a classification decision rendered by a trained machine learning model, comprising:

at least one processor circuit; and

at least one memory that stores program code to be executed by the at least one processor circuit, the program code comprising:

a threshold determiner that:

executes the trained machine learning model against an input data set;

for each of a plurality of predetermined classification threshold values usable to control the classification decision rendered by the trained machine learning model:

determines at least one accuracy metric of the trained machine learning model based on said executing;

models a precision of the trained machine learning model by determining a first probability density function of a beta distribution based on a number of true positives and a number of false positives to generate a modeled precision value;

models a recall of the trained machine learning model by determining a second probability density function of a beta distribution based on the number of true positives and a number of false negatives to generate a modeled recall value;

determines a probability that the modeled recall value has a first predetermined relationship with a target recall value;

determines that the probability has a second predetermined relationship with a predetermined confidence level; and

responsive to determining that the probability has the second predetermined relationship with the predetermined confidence level, includes the predetermined classification threshold value in a set of candidate classification threshold values;

selects the classification threshold value from among the set of candidate classification threshold values by selecting the candidate classification threshold value associated with the largest modeled precision value; and

configures the trained machine learning model to render the classification decision based on a comparison of an output of the trained machine learning model and the selected classification threshold value.

2 . The system of claim 1 , wherein the input data set is a validation data set utilized to validate the machine learning model during training thereof.

3 . The system of claim 1 , wherein at least one of the target recall value or the predetermined confidence level is specified via a user interface.

4 . The system of claim 1 , wherein the at least one accuracy metric comprises one or more of:

the number of true positives of the trained machine learning model resulting during execution against the input data set;

the number of false positives of the trained machine learning model resulting during execution against the input data set; or

the number of false negatives of the trained machine learning model resulting during execution against the input data set.

5 . The system of claim 1 , wherein the threshold determiner determines the probability that the modeled recall value has the first predetermined relationship with the target recall value by performing one of:

determining the probability that the modeled recall value meets the target recall value; or

determining the probability that the modeled recall value exceeds the target recall value.

6 . The system of claim 1 , wherein the threshold determiner determines that the probability has the second predetermined relationship with the predetermined confidence level by performing one of:

determining that the probability meets the predetermined confidence level; or

determining that the probability exceeds the predetermined confidence level.

7 . The system of claim 1 , wherein the first modeled precision value is determined by dividing the number of true positives by a summation of the number of true positives and the number of false positives, and the first modeled recall value is determined by dividing the number of true positives by a summation of the number of true positives and the number of false negatives.

8 . A computer-implemented method for selecting a classification threshold value usable to control a classification decision rendered by a trained machine learning model, comprising:

executing the trained machine learning model against an input data set;

for each of a plurality of predetermined classification threshold values usable to control the classification decision rendered by the trained machine learning model:

determining at least one accuracy metric of the trained machine learning model based on said executing;

modeling a precision of the trained machine learning model by determining a first probability density function of a beta distribution based on a number of true positives and a number of false positives to generate a modeled precision value;

modeling a recall of the trained machine learning model by determining a second probability density function of a beta distribution based on the number of true positives and a number of false negatives to generate a modeled recall value;

determining a probability that the modeled precision value has a first predetermined relationship with a target precision value;

determining that the probability has a second predetermined relationship with a predetermined confidence level; and

responsive to determining that the probability has the second predetermined relationship with the predetermined confidence level, including the predetermined classification threshold value in a set of candidate classification threshold values;

selecting the classification threshold value from among the set of candidate classification threshold values by selecting the candidate classification threshold value associated with the largest modeled recall value; and

configures the trained machine learning model to render the classification decision based on a comparison of an output of the trained machine learning model and the selected classification threshold value.

9 . The method of claim 8 , wherein the input data set is a validation data set utilized to validate the machine learning model during training thereof.

10 . The method of claim 8 , wherein at least one of the target precision value or the predetermined confidence level is specified via a user interface.

11 . The method of claim 8 , wherein the at least one accuracy metric comprises one or more of:

the number of true positives of the trained machine learning model resulting during execution against the input data set;

the number of false positives of the trained machine learning model resulting during execution against the input data set; or

the number of false negatives of the trained machine learning model resulting during execution against the input data set.

12 . The method of claim 8 , wherein determining the probability that the modeled precision value has the first predetermined relationship with the target precision value comprises one of:

determining the probability that the modeled precision value meets the target precision value; or

determining the probability that the modeled precision value exceeds the target precision value.

13 . The method of claim 8 , wherein determining that the probability has the second predetermined relationship with the predetermined confidence level comprises one of:

determining that the probability meets the predetermined confidence level; or

determining that the probability exceeds the predetermined confidence level.

14 . The method of claim 8 , wherein the first modeled precision value is determined by dividing the number of true positives by a summation of the number of true positives and the number of false positives, and the first modeled recall value is determined by dividing the number of true positives by a summation of the number of true positives and the number of false negatives.

15 . A computer-readable storage medium having program instructions recorded thereon that, when executed by at least one processor, perform a method for selecting a classification threshold value usable to control a classification decision rendered by a trained machine learning model, the method comprising:

executing the trained machine learning model against an input data set;

for each of a plurality of predetermined classification threshold values usable to control the classification decision rendered by the trained machine learning model:

determining at least one accuracy metric of the trained machine learning model based on said executing;

modeling a first evaluation metric of the trained machine learning model by determining a first probability density function of a beta distribution based on a number of true positives and a number of false positives to generate a first modeled evaluation metric value;

modeling a second evaluation metric of the trained machine learning model by determining a second probability density function of a beta distribution based on the number of true positives and a number of false negatives to generate a second modeled evaluation metric value;

determining a probability that the first modeled evaluation metric value has a first predetermined relationship with a target evaluation metric value;

determining that the probability has a second predetermined relationship with a predetermined confidence level; and

responsive to determining that the probability has the second predetermined relationship with the predetermined confidence level, including the predetermined classification threshold value in a set of candidate classification threshold values;

selecting the classification threshold value from among the set of candidate classification threshold values by selecting the candidate classification threshold value associated with the largest second modeled evaluation metric value; and

configures the trained machine learning model to render the classification decision based on a comparison of an output of the trained machine learning model and the selected classification threshold value.

16 . The computer-readable storage medium of claim 15 , wherein the first evaluation metric is one of a precision metric or a recall metric and the second evaluation metric is the other of the precision metric or the recall metric.

17 . The computer-readable storage medium of claim 15 , wherein the input data set is a validation data set utilized to validate the machine learning model during training thereof.

18 . The computer-readable storage medium of claim 15 , wherein the at least one accuracy metric comprises one or more of:

the number of true positives of the trained machine learning model resulting during execution against the input data set;

the number of false positives of the trained machine learning model resulting during execution against the input data set; or

the number of false negatives of the trained machine learning model resulting during execution against the input data set.

19 . The computer-readable storage medium of claim 15 , wherein determining the probability that the first modeled evaluation metric value has the first predetermined relationship with the target precision value comprises one of:

determining the probability that the first modeled evaluation metric value meets the target evaluation metric value; or

determining the probability that the first modeled evaluation metric value exceeds the target evaluation metric value.

20 . The computer-readable storage medium of claim 15 , wherein the first modeled precision value is determined by dividing the number of true positives by a summation of the number of true positives and the number of false positives, and the first modeled recall value is determined by dividing the number of true positives by a summation of the number of true positives and the number of false negatives.