Systems, apparatus, and methods of programmatically determining unique contacts based on crowdsourced error correction
Systems, apparatus, and methods for determining unique contacts from a collection or pool of merchant data are discussed herein. Some embodiments may provide for an apparatus including circuitry configured to determine programmatic match results indicating whether different instances of merchant data match (e.g., describe the same contact). The circuitry may further determine probabilities of precision or recall errors with the programmatic match results. Programmatic match results having a high probability of error may be annotated by a user to generate user match results. The user match results may be used to generate a more reliable contacts database including unique contacts, as well as to train and/or update the match scoring algorithm. As such, the accuracy of machine-implemented binary classification is improved.
1. An apparatus, comprising:
circuitry configured to:
determine a programmatic match result indicating whether first merchant data is a match with second merchant data by applying a match score algorithm to the first merchant data and second merchant data, the match score algorithm including a match score threshold;
in response to determining that the first merchant data is a match with the second merchant data:
determine a precision error score indicating whether the programmatic match result includes a precision error;
in response to determining that the precision error score indicates the programmatic match result includes a precision error, determine a user match result based on received user input;
determine a second programmatic match result indicating whether third merchant data is a match with fourth merchant data by applying the match score algorithm to the third merchant data and fourth merchant data;
in response to determining that the third merchant data fails to be a match with the fourth merchant data:
determine a recall error score indicating whether the second programmatic match result includes a recall error;
in response to determining that the precision error score indicates the second programmatic match result includes a recall error, determine a second user match result based on received second user input;
determine a precision error rate based at least in part on the programmatic match result and the user match result;
determine a recall error rate based at least in part on the second programmatic match result and the second user match result;
determine an estimated precision/recall error ratio based on the precision error rate and the recall error rate;
determine an expected precision/recall error ratio;
determine an updated match score threshold such that the estimated precision/recall error ratio matches the expected precision/recall error ratio; and
adjust the match score threshold of the match algorithm to the updated match score threshold, thereby improving accuracy of machine-implemented binary classification.
2. The apparatus of claim 1 , wherein the circuitry is further configured to update a contacts database based on the user match result.
3. The apparatus of claim 1 , wherein the circuitry being configured to determine the updated match score threshold includes the circuitry being configured to adjust the match score threshold a predetermined amount.
4. The apparatus of claim 1 , wherein the circuitry being configured to determine the updated match score threshold such that the estimated precision/recall error ratio matches the expected precision/recall error ratio includes the circuitry being configured to incrementally adjust the match score threshold by a predetermined amount.
5. The apparatus of claim 4 , wherein the circuitry is configured to incrementally increase the match score threshold in response to determining that the expected precision/recall error ratio exceeds the estimated precision/recall error ratio.
6. The apparatus of claim 4 , wherein the circuitry is configured to incrementally decrease the match score threshold in response to determining that the estimated precision/recall error ratio exceeds the expected precision/recall error ratio.
7. The apparatus of claim 1 , wherein the circuitry being configured to determine the updated match score threshold such that the estimated precision/recall error ratio matches the expected precision/recall error ratio includes the circuitry being configured to use one or more of: gradient descent, simulated annealing, or Newton's method.
8. The apparatus of claim 1 , wherein the circuitry being configured to determine the updated match score threshold such that the estimated precision/recall error ratio matches the expected precision/recall error ratio includes the circuitry being configured to utilize an optimization procedure to minimize a difference between the estimated p/r error ratio and the expected p/r error ratio as a function of the match score threshold.
9. The apparatus of claim 1 , wherein the circuitry being configured to determine the expected precision/recall error ratio includes the circuitry being configured to minimize a cost value as function of the precision error rate and recall error rate.
10. A method for machine-implemented binary classification, comprising, by circuitry of an apparatus:
determining a programmatic match result indicating whether first merchant data is a match with second merchant data by applying a match score algorithm to the first merchant data and second merchant data, the match score algorithm including a match score threshold;
in response to determining that the first merchant data is a match with the second merchant data:
determining a precision error score indicating whether the programmatic match result includes a precision error;
in response to determining that the precision error score indicates the programmatic match result includes a precision error, determining a user match result based on received user input;
determining a second programmatic match result indicating whether third merchant data is a match with fourth merchant data by applying the match score algorithm to the third merchant data and fourth merchant data;
in response to determining that the third merchant data fails to be a match with the fourth merchant data:
determining a recall error score indicating whether the second programmatic match result includes a recall error;
in response to determining that the precision error score indicates the second programmatic match result includes a recall error, determining a second user match result based on received second user input;
determining a precision error rate based at least in part on the programmatic match result and the user match result;
determining a recall error rate based at least in part on the second programmatic match result and the second user match result;
determining an estimated precision/recall error ratio based on the precision error rate and the recall error rate;
determining an expected precision/recall error ratio;
determining an updated match score threshold such that the estimated precision/recall error ratio matches the expected precision/recall error ratio; and
adjusting the match score threshold of the match algorithm to the updated match score threshold, thereby improving accuracy of machine-implemented binary classification.
11. The method of claim 10 further comprising, by the circuitry, updating a contacts database based on the user match result.
12. The method of claim 10 , wherein determining the updated match score threshold includes adjusting the match score threshold a predetermined amount.
13. The method of claim 10 , wherein determining the updated match score threshold such that the estimated precision/recall error ratio matches the expected precision/recall error ratio includes incrementally adjusting the match score threshold by a predetermined amount.
14. The method of claim 13 further comprising, by the circuitry, incrementally increasing the match score threshold in response to determining that the expected precision/recall error ratio exceeds the estimated precision/recall error ratio.
15. The method of claim 13 further comprising, by the circuitry, incrementally decreasing the match score threshold in response to determining that the estimated precision/recall error ratio exceeds the expected precision/recall error ratio.
16. The method of claim 10 , wherein determining the updated match score threshold such that the estimated precision/recall error ratio matches the expected precision/recall error ratio includes using one or more of: gradient descent, simulated annealing, or Newton's method.
17. The method of claim 10 , wherein determining the updated match score threshold such that the estimated precision/recall error ratio matches the expected precision/recall error ratio includes utilizing an optimization procedure to minimize a difference between the estimated p/r error ratio and the expected p/r error ratio as a function of the match score threshold.
18. The method of claim 10 , wherein determining the expected precision/recall error ratio includes minimizing a cost value as function of the precision error rate and recall error rate.