Machine learning to tune probabilistic matching in entity resolution systems

1. A method, comprising:

receiving a plurality of data records;

generating a first comparison vector by comparing a first and a second data records of the plurality of data records, wherein the first and second data records comprise values for a plurality of attributes related to an entity, wherein the first comparison vector indicates differences between the first and second data records in the plurality of attributes;

generating, by a probabilistic matching engine, a matching score based at least in part on the first comparison vector;

training a machine learning (ML) model based at least in part on the first comparison vector and the matching score, wherein the training the ML model comprises:

determining a match status of the first and second data records based at least in part on comparing the matching score with a threshold,

training the machine learning model using the first comparison vector as input and the match status as target output, and

upon determining that accuracy of the machine learning model satisfies one or more accuracy criteria, deploying the machine learning model;

evaluating, based on one or more generated comparison vectors, the plurality of data records using the trained machine learning model; and

linking at least two data records of the plurality of data records based on the evaluation.

2. The method of claim 1 , the method further comprising:

generating a second comparison vector by comparing third and fourth data records of the plurality of data records;

labeling the third and fourth data records as not matching, based on evaluating the second comparison vector using trained the machine learning model;

receiving an indication that the third and fourth data records are matching; and

refining the trained machine learning model based on the indication.

3. The method of claim 1 , the method further comprising:

generating a second comparison vector by comparing third and fourth data records of the plurality of data records;

labeling the third and fourth data records as matching, based on evaluating the second comparison vector using the trained machine learning model;

receiving an indication that the third and fourth data records are not matching; and

refining the trained machine learning model based on the indication.

4. The method of claim 3 , the method further comprising:

determining that the third and fourth data records represent a false positive;

evaluating the second comparison vector to identify a feature pattern indicative of the false positive; and

evaluating a set of matching records using the identified feature pattern.

5. The method of claim 1 , wherein the plurality of attributes includes at least one of:

(i) a name of a corresponding person;

(ii) a numeric identifier of the corresponding person;

(iii) a date of birth of the corresponding person;

(iv) an email address of the corresponding person;

(v) a mailing address of the corresponding person; or

(vi) a phone number of the corresponding person.

6. The method of claim 1 , the method further comprising:

providing a subset of the plurality of data records for manual review;

iteratively refining the trained machine learning model based on the manual review; and

upon determining that the trained machine learning model is sufficiently accurate, deploying the trained machine learning model.

7. The method of claim 1 , wherein generating the first comparison vector comprises:

identifying differences between the first and second data records;

generating one or more scores based on the identified differences using a predefined default configuration; and

aggregating the identified differences and the one or more scores.

8. The method of claim 7 , wherein the training the machine learning model based at least in part on the first comparison vector further comprises:

determining a match status of the first and second data records; and

training the machine learning model based further on the match status.

9. A computer program product comprising one or more computer-readable storage media collectively containing computer-readable program code that, when executed by operation of one or more computer processors, performs an operation comprising:

receiving a plurality of data records;

generating a first comparison vector by comparing a first and a second data records of the plurality of data records, wherein the first and second data records comprise values for a plurality of attributes related to an entity, wherein the first comparison vector indicates differences between the first and second data records in the plurality of attributes;

generating, by a probabilistic matching engine, a matching score based at least in part on the first comparison vector;

training a machine learning (ML) model based at least in part on the first comparison vector and the matching score, wherein the training the ML model comprises:

determining a match status of the first and second data records based at least in part on comparing the matching score with a threshold,

training the machine learning model using the first comparison vector as input and the match status as target output, and

upon determining that accuracy of the machine learning model satisfies one or more accuracy criteria, deploying the machine learning model;

evaluating, based on one or more generate comparison vectors, the plurality of data records using the trained machine learning model; and

linking at least two data records of the plurality of data records based on the evaluation.

10. The computer program product of claim 9 , the operation further comprising:

generating a second comparison vector by comparing third and fourth data records of the plurality of data records;

labeling the third and fourth data records as matching, based on evaluating the second comparison vector using the trained machine learning model;

receiving an indication that the third and fourth data records are not matching; and

refining the trained machine learning model based on the indication.

11. The computer program product of claim 10 , the operation further comprising:

determining that the third and fourth data records represent a false positive;

evaluating the second comparison vector to identify a feature pattern indicative of the false positive; and

evaluating a set of matching records using the identified feature pattern.

12. The computer program product of claim 9 , wherein the plurality of attributes includes at least one of:

(i) a name of a corresponding person;

(ii) a numeric identifier of the corresponding person;

(iii) a date of birth of the corresponding person;

(iv) an email address of the corresponding person;

(v) a mailing address of the corresponding person; or

(vi) a phone number of the corresponding person.

13. The computer program product of claim 9 , wherein generating the first comparison vector comprises:

identifying differences between the first and second data records;

generating one or more scores based on the identified differences using a predefined default configuration; and

aggregating the identified differences and the one or more scores.

14. The computer program product of claim 13 , wherein the training the machine learning model based at least in part on the first comparison vector further comprises:

determining a match status of the first and second data records; and

training the machine learning model based further on the match status.

15. A system comprising:

one or more computer processors; and

one or more memories collectively containing one or more programs which when executed by the one or more computer processors performs an operation, the operation comprising:

receiving a plurality of data records;

generating a first comparison vector by comparing a first and a second data records of the plurality of data records, wherein the first and second data records comprise values for a plurality of attributes related to an entity, wherein the first comparison vector indicates differences between the first and second data records in the plurality of attributes;

generating, by a probabilistic matching engine, a matching score based at least in part on the first comparison vector;

training a machine learning (ML) model based at least in part on the first comparison vector and the matching score, wherein the training the ML model comprises:

determining a match status of the first and second data records based at least in part on comparing the matching score with a threshold, and

training the machine learning model using the first comparison vector as input and the match status as target output, and

upon determining that accuracy of the machine learning model satisfies one or more accuracy criteria, deploying the machine learning model;

evaluating, based on one or more generated comparison vector, the plurality of data records using the trained machine learning model; and

linking at least two data records of the plurality of data records based on the evaluation.

16. The system of claim 15 , the operation further comprising:

generating a second comparison vector by comparing third and fourth data records of the plurality of data records;

labeling the third and fourth data records as matching, based on evaluating the second comparison vector using the trained machine learning model;

receiving an indication that the third and fourth data records are not matching; and

refining the trained machine learning model based on the indication.

17. The system of claim 16 , the operation further comprising:

determining that the third and fourth data records represent a false positive;

evaluating the second comparison vector to identify a feature pattern indicative of the false positive; and

evaluating a set of matching records using the identified feature pattern.

18. The system of claim 15 , wherein the plurality of attributes includes at least one of:

(i) a name of a corresponding person;

(ii) a numeric identifier of the corresponding person;

(iii) a date of birth of the corresponding person;

(iv) an email address of the corresponding person;

(v) a mailing address of the corresponding person; or

(vi) a phone number of the corresponding person.

19. The system of claim 15 , wherein generating the first comparison vector comprises:

identifying differences between the first and second data records;

generating one or more scores based on the identified differences using a predefined default configuration; and

aggregating the identified differences and the one or more scores.

20. The system of claim 19 , wherein the training the machine learning model based at least in part on the first comparison vector further comprises:

determining a match status of the first and second data records; and

training the machine learning model based further on the match status.