Data normalization system
A data normalization system receives a first string and a second string that are ordered according to an initial string ordering. The data normalization system analyzes, the first string and the second string based on a list of known character sets included in surnames, yielding an analysis, and determines, based on the analysis, that a set of characters in the second string matches a known character set included in the list of known character sets included in surnames. In response to determining that the set of characters in the second string matches a known character set included in the list of known character sets included in surname, the data normalization system orders the first string and the second string according to an updated string ordering.
1. A data normalization system comprising:
one or more computer processors; and
one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, causes the data normalization system to perform operations comprising:
receiving a list of string pairs, the list of string pairs including at least a first string pair, the first string pair including a first string and a second string being ordered according to an initial string ordering;
converting, using a metaphone algorithm, the first string into a first metaphone string, and the second string into a second metaphone string;
searching, based on the first metaphone string and the second metaphone string, a name index including a listing of metaphone strings representing common names, each respective metaphone string included in the name index including a first probability score of a probability that a respective common name represented by the respective metaphone string is a given name and a second probability score of a probability that the respective common name represented by the respective metaphone string is a surname;
determining, based on searching the name index, a confidence score indicating a confidence level that the first string represents the given name and that the second string represents the surname, the confidence score based on the first probability score and the second probability score;
comparing the confidence score to a threshold confidence score;
performing, on the basis of determining the confidence score meets or exceeds the threshold confidence score, operations comprising:
performing, on the basis of determining that the first string represents the given name and the second string represents the surname, operations comprising;
ordering the first string and the second string according to an updated string ordering; and
performing, on the basis of determining the confidence score does not meet the threshold confidence score, operations comprising:
performing, on the basis of determining that the first string represents the surname and the second string represents the given name, operations comprising:
ordering the first string and the second string according to an updated string ordering.
2. The data normalization system of claim 1 , the operations further comprising:
generating a normalized list of string pairs.
3. The data normalization system of claim 1 , the operations further comprising:
determining that the second string includes a set of characters or patterns indicating that the second string is the surname.
4. The data normalization system of claim 3 , wherein determining that the second string includes the set of characters or patterns indicating that the second string is a surname comprises:
analyzing the second string based on a list of known character sets or patterns included in surnames; and
determining that the set of characters or patterns included in the second string matches a known character set included in the list of known character sets or patterns included in surnames.
5. The data normalization system of claim 3 , wherein determining the confidence score is further based on determining that the second string includes the set of characters or patterns indicating that the second string is the surname.
6. The data normalization system of claim 1 , wherein the name index includes at least a first set of metaphone strings corresponding to a first language set, and a second set of metaphone strings corresponding to a second language set.
7. The data normalization system of claim 1 , the operations further comprising:
accessing a second string pair from the list of string pairs, the second string pair including a third string and a fourth string being ordered according to the initial string ordering;
converting, using the metaphone algorithm, the third string into a third metaphone string, and the fourth string into a fourth metaphone string;
searching the name index based on the third metaphone string and the fourth metaphone string;
determining a second confidence score indicating a confidence level that the third string represents the given name and that the fourth string represents the surname; and
determining that the third string represents the surname and the fourth string represents the given name on the basis of the second confidence score not meeting the threshold confidence score.
8. A method comprising:
receiving a list of string pairs, the list of string pairs including at least a first string pair, the first string pair including a first string and a second string being ordered according to an initial string ordering;
converting, using a metaphone algorithm, the first string into a first metaphone string, and the second string into a second metaphone string;
searching, based on the first metaphone string and the second metaphone string, a name index including a listing of metaphone strings representing common names, each respective metaphone string included in the name index including a first probability score of a probability that a respective common name represented by the respective metaphone string is a given name and a second probability score of a probability that the respective common name represented by the respective metaphone string is a surname;
determining, based on searching the name index, a confidence score indicating a confidence level that the first string represents the given name and that the second string represents the surname, the confidence score based on the first probability score and the second probability score;
comparing the confidence score to a threshold confidence score;
performing, on the basis of determining the confidence score meets or exceeds the threshold confidence score, operations comprising:
performing, on the basis of determining that the first string represents the given name and the second string represents the surname, operations comprising:
ordering the first string and the second string according to an updated string ordering; and
performing, on the basis of determining the confidence score does not meet the threshold confidence score, operations comprising:
performing, on the basis of determining that the first string represents the surname and the second string represents the given name, operations comprising:
ordering the first string and the second string according to an updated string ordering.
9. The method of claim 8 , further comprising:
generating a normalized list of string pairs.
10. The method of claim 8 , further comprising:
determining that the second string includes a set of characters or patterns indicating that the second string is the surname.
11. The method of claim 10 , wherein determining that the second string includes the set of characters or patterns indicating that the second string is a surname comprises:
analyzing the second string based on a list of known character sets or patterns included in surnames; and
determining that the set of characters or patterns included in the second string matches a known character set included in the list of known character sets or patterns included in surnames.
12. The method of claim 10 , wherein determining the confidence score is further based on determining that the second string includes the set of characters or patterns indicating that the second string is the surname.
13. The method of claim 8 , wherein the name index includes at least a first set of metaphone strings corresponding to a first language set, and a second set of metaphone strings corresponding to a second language set.
14. The method of claim 8 , further comprising:
accessing a second string pair from the list of string pairs, the second string pair including a third string and a fourth string being ordered according to the initial string ordering;
converting, using the metaphone algorithm, the third string into a third metaphone string, and the fourth string into a fourth metaphone string;
searching the name index based on the third metaphone string and the fourth metaphone string;
determining a second confidence score indicating a confidence level that the third string represents the given name and that the fourth string represents the surname; and
determining that the third string represents the surname and the fourth string represents the given name on the basis of the second confidence score not meeting the threshold confidence score.
15. A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors of one or more computing devices, cause the one or more computer processor to perform operations comprising:
receiving a list of string pairs, the list of string pairs including at least a first string pair, the first string pair including a first string and a second string being ordered according to an initial string ordering;
converting, using a metaphone algorithm, the first string into a first metaphone string, and the second string into a second metaphone string;
searching, based on the first metaphone string and the second metaphone string, a name index including a listing of metaphone strings representing common names, each respective metaphone string included in the name index including a first probability score of a probability that a respective common name represented by the respective metaphone string is a given name and a second probability score of a probability that the respective common name represented by the respective metaphone string is a surname;
determining, based on searching the name index, a confidence score indicating a confidence level that the first string represents the given name and that the second string represents the surname, the confidence score based on the first probability score and the second probability score;
comparing the confidence score to a threshold confidence score;
performing, on the basis of determining the confidence score meets or exceeds the threshold confidence score, operations comprising:
on the basis of determining that the first string represents the given name and the second string represents the surname, performing operations comprising:
ordering the first string and the second string according to an updated string ordering; and
performing, on the basis of determining the confidence score does not meet the threshold confidence score, operations comprising:
on the basis of determining that the first string represents the surname and the second string represents the given name, performing operations comprising:
ordering the first string and the second string according to an updated string ordering.
16. The non-transitory computer-readable medium of claim 15 , the operations further comprising:
generating a normalized list of string pairs.
17. The non-transitory computer-readable medium of claim 15 , the operations further comprising:
determining that the second string includes a set of characters or patterns indicating that the second string is the surname.
18. The non-transitory computer-readable medium of claim 17 , wherein determining that the second string includes the set of characters or patterns indicating that the second string is a surname comprises:
analyzing the second string based on a list of known character sets or patterns included in surnames; and
determining that the set of characters or patterns included in the second string matches a known character set included in the list of known character sets or patterns included in surnames.
19. The non-transitory computer-readable medium of claim 17 , wherein determining the confidence score is further based on determining that the second string includes the set of characters or patterns indicating that the second string is the surname.
20. The non-transitory computer-readable medium of claim 15 , wherein the name index includes at least a first set of metaphone strings corresponding to a first language set, and a second set of metaphone strings corresponding to a second language set.