Machine-learning system and method for identifying same person in genealogical databases
Systems and methods for determining whether two tree persons in a genealogical database correspond to the same real-life individual. Embodiments include identifying two tree persons in a genealogical database and extracting a plurality of features from both tree persons to generate two vectors. Embodiments also include calculating a plurality of metrics between the two vectors to generate a metric function. Embodiments further include generating feature weights using a recursive process based on training data input by external users, and generating a score by calculating a weighted sum of the metric function being weighted by the feature weights. The generated score may then be compared to a threshold value.
1. A computer-implemented method comprising:
identifying a first tree person from a first genealogical tree and a second tree person from a second genealogical tree, wherein both the first genealogical tree and the second genealogical tree comprise a plurality of interconnected tree persons corresponding to individuals that are related to each other;
extracting, from first tree data of the first genealogical tree, a first set of features for the first tree person and, from second tree data of the second genealogical tree, a second set of features for the second tree person;
based on extracting the first set of features for the first tree person and the second set of features for the second tree person, generating a metric function, by comparing like features from the first set of features for the first tree person with corresponding features from the second set of features for the second tree person;
generating a plurality of feature weights for similarity metrics of the metric function using a machine learning model configured to output the plurality of feature weights based on receiving an input comprising the first set of features and the second set of features, wherein the machine learning model is trained by:
providing training data comprising pairs of tree persons to the machine learning model; and
modifying the machine learning model using an error computed based on an output of the machine learning model when provided with the training data;
generating a plurality of weighted similarity metrics by multiplying similarity metrics of the metric function with corresponding feature weights from the plurality of feature weights;
generating a similarity score indicating a likelihood of the first tree person and the second tree person being duplicates by calculating a sum of the plurality of weighted similarity metrics; and
modifying a cluster in a genealogical database based on the likelihood of the first tree person and the second tree person being duplicates.
2. The computer-implemented method of claim 1 , wherein comparing the like features from the first set of features for the first tree person with corresponding features from the second set of features for the second tree person further comprises generating the similarity metrics of the metric function, wherein the similarity metrics indicate similarities between features of the first set of features and corresponding features of the second set of features.
3. The computer-implemented method of claim 1 , further comprising:
determining that a feature of the first set of features or the second set of features comprises an error; and
based on determining that the feature of the first set of features or the second set of features comprises an error, modifying the feature to correct the error.
4. The computer-implemented method of claim 3 , wherein modifying the feature to correct the error further comprises normalizing a format of the feature.
5. The computer-implemented method of claim 4 , wherein normalizing the format of the feature further comprises modifying an address feature into a predetermined format.
6. The computer-implemented method of claim 1 , wherein at least one feature of the first set of features is a race feature.
7. The computer-implemented method of claim 1 , further comprising:
determining that the similarity score satisfies a similarity threshold; and
modifying the cluster in the genealogical database based on determining that similarity score satisfies the similarity threshold.
8. The computer-implemented method of claim 7 , further comprising:
determining a number of available features from the first set of features and the second set of features; and
modifying a value of the similarity threshold based on the number of available features from the first set of features and the second set of features.
9. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause a computer system to:
identify a first tree person from a first genealogical tree and a second tree person from a second genealogical tree, wherein both the first genealogical tree and the second genealogical tree comprise a plurality of interconnected tree persons corresponding to individuals that are related to each other;
extract, from first tree data of the first genealogical tree, a first set of features for the first tree person and, from second tree data of the second genealogical tree, a second set of features for the second tree person;
based on extracting the first set of features for the first tree person and the second set of features for the second tree person, generate a metric function, by comparing like features from the first set of features for the first tree person with corresponding features from the second set of features for the second tree person;
generate a plurality of feature weights for the metric function using a machine learning model configured to output the plurality of feature weights based on receiving an input comprising the first set of features and the second set of features, wherein the machine learning model is trained by:
providing training data comprising pairs of tree persons to the machine learning model; and
modifying the machine learning model using an error computed based on an output of the machine learning model when provided with the training data;
generate a plurality of weighted similarity metrics by multiplying similarity metrics of the metric function with corresponding feature weights from the plurality of feature weights;
generate a similarity score indicating a likelihood of the first tree person and the second tree person being duplicates by calculating a sum of the plurality of weighted similarity metrics; and
modify a cluster in a genealogical database based on the likelihood of the first tree person and the second tree person being duplicates.
10. The non-transitory computer-readable medium of claim 9 , further comprising instructions that, when executed by the at least one processor, cause the computer system to compare the like features from the first set of features for the first tree person with corresponding features from the second set of features for the second tree person by generating the similarity metrics of the metric function, wherein the similarity metrics indicate similarities between features of the first set of features and corresponding features of the second set of features.
11. The non-transitory computer-readable medium of claim 9 , further comprising instructions that, when executed by the at least one processor, cause the computer system to:
determine that a feature of the first set of features or the second set of features comprises an error; and
based on determining that the feature of the first set of features or the second set of features comprises an error, modify the feature to correct the error.
12. The non-transitory computer-readable medium of claim 11 , further comprising instructions that, when executed by the at least one processor cause the computer system to correct the error by normalizing a format of the feature.
13. The non-transitory computer-readable medium of claim 12 , further comprising instructions that, when executed by the at least one processor, cause the computer system to normalize the format of the feature by transforming an address feature into a predetermined format.
14. The non-transitory computer-readable medium of claim 9 , wherein at least one feature of the first set of features is a race feature.
15. The non-transitory computer-readable medium of claim 9 , further comprising instructions that, when executed by the at least one processor, cause the computer system to:
determine that the similarity score satisfies a similarity threshold; and
modify the cluster in the genealogical database based on determining that similarity score satisfies the similarity threshold.
16. The non-transitory computer-readable medium of claim 15 , further comprising instructions that, when executed by the at least one processor, cause the computer system to:
determine a number of available features from the first set of features and the second set of features; and
modify a value of the similarity threshold based on the number of available features from the first set of features and the second set of features.
17. A system comprising:
at least one processor; and
at least one non-transitory computer-readable storage medium storing instructions that, when executed by the at least one processor, cause the system to:
identify a first tree person from a first genealogical tree and a second tree person from a second genealogical tree, wherein both the first genealogical tree and the second genealogical tree comprise a plurality of interconnected tree persons corresponding to individuals that are related to each other;
extract, from first tree data of the first genealogical tree, a first set of features for the first tree person and, from second tree data of the second genealogical tree, a second set of features for the second tree person;
based on extracting the first set of features for the first tree person and the second set of features for the second tree person, generate a metric function, by comparing like features from the first set of features for the first tree person with corresponding features from the second set of features for the second tree person;
generate a plurality of feature weights for similarity metrics of the metric function using a machine learning model configured to output the plurality of feature weights based on receiving an input comprising the first set of features and the second set of features, wherein the machine learning model is trained by:
providing training data comprising pairs of tree persons to the machine learning model; and
modifying the machine learning model using an error computed based on an output of the machine learning model when provided with the training data;
generate a plurality of weighted similarity metrics by multiplying similarity metrics of the metric function with corresponding feature weights from the plurality of feature weights;
generate a similarity score indicating a likelihood of the first tree person and the second tree person being duplicates by calculating a sum of the plurality of weighted similarity metrics; and
modify a cluster in a genealogical database based on the likelihood of the first tree person and the second tree person being duplicates.
18. The system of claim 17 , further comprising instructions that, when executed by the at least one processor, cause the system to:
generating a similarity threshold based on a number of available features;
determine that the similarity score satisfies the similarity threshold; and
modify the cluster in the genealogical database based on determining that similarity score satisfies the similarity threshold.
19. The system of claim 17 , further comprising instructions that, when executed by the at least one processor, cause the system to:
determine that a feature of the first set of features or the second set of features comprises an error; and
based on determining that the feature of the first set of features or the second set of features comprises an error, modify the feature to normalize a format of the feature.
20. The system of claim 19 , wherein modifying the feature to normalize the format of the feature comprises modifying an address feature into a predetermined format.