Methods and servers for determining metric-specific thresholds to be used with a plurality of nested metrics for binary classification of a digital object
Method and server for determining a target combination of metric-specific thresholds to be used with a plurality of nested metrics for performing binary classification of a digital object are disclosed. The method includes acquiring object-specific validation datasets, and a plurality of nested metrics thereon, thereby generating a plurality of prediction values. During a first iteration, the server compares predictions values against a first combination of metric-specific thresholds and generates first precision parameters and first recall parameters for the first iteration. During a second iteration, the server adjusts one of the first combination thereby generating a second combination, compares the predictions values against the second combination, and generates second precision parameters and second recall parameters for the second iteration. The method includes selecting, by the one of the first combination and the second combination as the target combination of metric-specific thresholds.
1. A method of determining a target combination of metric-specific thresholds to be used with a plurality of nested metrics for performing binary classification of a digital object into a first class or a second class, the object being associated with past object events an indication of which is stored in a storage, the method executable by a server configured to access the storage, the method comprising:
acquiring, by the server, a plurality of object-specific validation datasets, a given one of the plurality of object-specific validation datasets comprising an indication of a plurality of past object events associated with a respective validation object and a ground-truth class of the respective validation object being one of the first class and the second class;
applying, by the server, a plurality of nested metrics onto the plurality of object-specific validation datasets, thereby generating a plurality of prediction values,
a given prediction value being indicative of a respective probability of the respective validation object belonging to one of the first class and the second class;
during a first iteration:
comparing, by the server, the plurality of predictions values against respective ones from a first combination of metric-specific thresholds for determining predicted classes of the respective validation objects for the first iteration;
generating, by the server, first precision parameters and first recall parameters for the plurality of nested metrics for the first iteration by comparing the ground-truth classes against the respective predicted classes of respective validation objects of the first iteration;
during a second iteration:
adjusting, by the server, one of the first combination of metric-specific thresholds thereby generating a second combination of metric-specific thresholds;
comparing, by the server, the plurality of predictions values against respective ones from the second combination of metric-specific thresholds for determining predicted classes of the respective validation objects for the second iteration;
generating, by the server, second precision parameters and second recall parameters for the plurality of nested metrics for the second iteration by comparing the ground-truth classes against the respective predicted classes of respective validation objects of the second iteration; and
selecting, by the server, one of the first combination of metric-specific thresholds and the second combination of metric-specific thresholds as the target combination of metric-specific thresholds by:
comparing at least one of (i) the first precision parameters and the second precision parameters against a precision threshold, and (ii) the first recall parameters and the second recall parameters against a recall threshold, and
the target combination of metric-specific thresholds to be used with the plurality of nested metrics in an in-use mode for performing binary classification of the digital object,
such that in response to an in-use predicted value of at least one of the plurality of nested metrics for the digital object being above a respective one of the target combination of metric-specific thresholds, determining the digital object to be of the first class.
2. The method of claim 1 , wherein the method further comprises:
performing, by the server, a plurality of iterations until at least one of (i) precision parameters for a given iteration are above the precision threshold, and (ii) recall parameters for the given iteration are above the recall threshold; and
selecting, by the server, a given combination of metric-specific thresholds from the given iteration as the target combination of metric-specific thresholds.
3. The method of claim 1 , wherein the method further comprises:
during the in-use mode:
acquiring, by the server, an object-specific dataset comprising an indication of a plurality of past object events associated with the object;
applying, by the server, a plurality of nested metrics onto the object-specific dataset, thereby generating one or more prediction values indicative of a respective probability of the object belonging to one of the first class and the second class;
comparing, by the server, the one or more prediction values against the respective ones from the target combination of metric-specific thresholds,
in response to at least one of the one or more prediction values being above the respective one from the target combination of metric-specific thresholds, determining, by the server, the object to be of the first class; and
in response to none of the one or more prediction values being above the respective one from the target combination of metric-specific thresholds, determining, by the server, the object to be of the second class.
4. The method of claim 1 , wherein a first metric from the plurality of nested metrics is based on a first type of object events, and wherein a second metric from the plurality of nested metrics is based on a second type of object events, the second type of object events occurring only if the first type of object events have occurred.
5. The method of claim 4 , wherein the second type of object events is a subset of the first type of object events.
6. The method of claim 1 , wherein the target combination of metric-specific thresholds is selected simultaneously for the plurality of nested metrics during a single iteration.
7. The method of claim 1 , wherein the object is a given email and the first class is spam and the second class is non-spam.
8. The method of claim 1 , wherein the object is a given user of an e-market platform and the first class is a fraudulent class and the second class is a non-fraudulent class.
9. The method of claim 1 , wherein the object is a given document and the first class is a relevant class and the second class is a non-relevant class.
10. A server for determining a target combination of metric-specific thresholds to be used with a plurality of nested metrics for performing binary classification of a digital object into a first class or a second class, the object being associated with past object events an indication of which is stored in a storage, the server configured to access the storage, the server being configured to:
acquire a plurality of object-specific validation datasets, a given one of the plurality of object-specific validation datasets comprising an indication of a plurality of past object events associated with a respective validation object and a ground-truth class of the respective validation object being one of the first class and the second class;
apply a plurality of nested metrics onto the plurality of object-specific validation datasets, thereby generating a plurality of prediction values,
a given prediction value being indicative of a respective probability of the respective validation object belonging to one of the first class and the second class;
during a first iteration:
compare the plurality of predictions values against respective ones from a first combination of metric-specific thresholds for determining predicted classes of the respective validation objects for the first iteration;
generate first precision parameters and first recall parameters for the plurality of nested metrics for the first iteration by comparing the ground-truth classes against the respective predicted classes of respective validation objects of the first iteration;
during a second iteration:
adjust one of the first combination of metric-specific thresholds thereby generating a second combination of metric-specific thresholds;
compare the plurality of predictions values against respective ones from the second combination of metric-specific thresholds for determining predicted classes of the respective validation objects for the second iteration;
generate second precision parameters and second recall parameters for the plurality of nested metrics for the second iteration by comparing the ground-truth classes against the respective predicted classes of respective validation objects of the second iteration; and
select one of the first combination of metric-specific thresholds and the second combination of metric-specific thresholds as the target combination of metric-specific thresholds by:
comparing at least one of (i) the first precision parameters and the second precision parameters against a precision threshold, and (ii) the first recall parameters and the second recall parameters against a recall threshold, and
the target combination of metric-specific thresholds to be used with the plurality of nested metrics in an in-use mode for performing binary classification of the digital object,
such that in response to an in-use predicted value of at least one of the plurality of nested metrics for the digital object being above a respective one of the target combination of metric-specific thresholds, determining the digital object to be of the first class.
11. The server of claim 10 , wherein the server is further configured to:
perform a plurality of iterations until at least one of (i) precision parameters for a given iteration are above the precision threshold, and (ii) recall parameters for the given iteration are above the recall threshold; and
select a given combination of metric-specific thresholds from the given iteration as the target combination of metric-specific thresholds.
12. The server of claim 10 , wherein the server is further configured to:
during the in-use mode:
acquire an object-specific dataset comprising an indication of a plurality of past object events associated with the object;
apply a plurality of nested metrics onto the object-specific dataset, thereby generating one or more prediction values indicative of a respective probability of the object belonging to one of the first class and the second class;
compare the one or more prediction values against the respective ones from the target combination of metric-specific thresholds,
in response to at least one of the one or more prediction values being above the respective one from the target combination of metric-specific thresholds, determine the object to be of the first class; and
in response to none of the one or more prediction values being above the respective one from the target combination of metric-specific thresholds, determine the object to be of the second class.
13. The server of claim 10 , wherein a first metric from the plurality of nested metrics is based on a first type of object events, and wherein a second metric from the plurality of nested metrics is based on a second type of object events, the second type of object events occurring only if the first type of object events have occurred.
14. The server of claim 13 , wherein the second type of object events is a subset of the first type of object events.
15. The server of claim 10 , wherein the target combination of metric-specific thresholds is selected simultaneously for the plurality of nested metrics during a single iteration.
16. The server of claim 10 , wherein the object is a given email and the first class is spam and the second class is non-spam.
17. The server of claim 10 , wherein the object is a given user of an e-market platform and the first class is a fraudulent class and the second class is a non-fraudulent class.
18. The server of claim 10 , wherein the object is a given document and the first class is a relevant class and the second class is a non-relevant class.