Data normalization system
A data normalization system that normalizes data that includes user names is disclosed herein. The data normalization system converts, using a metaphone algorithm, a first string into a first metaphone string, and a second string into a second metaphone string. The data normalization system searches, based on the first metaphone string and the second metaphone string, a name index including a listing of metaphone strings representing common names and probabilities that the common names are either a given name or a surname. The data normalization system determines, based on searching the name index, a confidence score indicating a confidence level that the first string represents the given name and the second string represents the surname. The data normalization system determines that the confidence score meets or exceeds a threshold confidence score, and in response, determines that the first string represents the given name and the second string represents the surname.
1. A data normalization system comprising:
one or more computer processors; and
one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, causes the data normalization system to perform operations comprising:
converting, using a metaphone algorithm, a first string into a first metaphone string, and a second string into a second metaphone string, the first string being positioned before the second string in an initial string ordering;
searching, based on the first metaphone string and the second metaphone string, a name index including a listing of metaphone strings representing common names and probabilities that the common names are either a given name or a surname;
determining, based on searching the name index, a signal strength that the first metaphone string is a given name, a signal strength that the first metaphone string is a surname, a signal strength that the second metaphone string is a given name, and a signal strength that the second metaphone string is a surname;
determining an absolute difference between the signal strength that the first metaphone string is a surname and the signal strength that the first metaphone string is a given name, yielding an absolute signal difference for the first string;
determining an absolute difference between the signal strength that the second metaphone string is a surname and the signal strength that the second metaphone string is a given name, yielding an absolute signal difference for the second string;
determining that the absolute signal difference for the first string is greater than the absolute signal difference for the second string;
determining, based on the absolute difference for the first string, a confidence score indicating a confidence level that the first string represents the given name and that the second string represents the surname;
determining that the confidence score meets or exceeds a threshold confidence score; and
in response to determining that the confidence score meets or exceeds the threshold confidence score, ordering the first string and the second string according to an updated string ordering based on the absolute signal difference of the first string.
2. The data normalization system of claim 1 , the operations further comprising:
determining that the second string includes a set of characters or patterns indicating that the second string is the surname.
3. The data normalization system of claim 2 , wherein determining that the second string includes the set of characters indicating that the second string is a surname comprises:
analyzing the second string based on a list of known character sets or patterns included in surnames; and
determining that the set of characters or patterns included in the second string matches a known character set included in the list of known character sets or patterns included in surnames.
4. The data normalization system of claim 2 , wherein determining the confidence score is further based on determining that the second string includes the set of characters or patterns indicating that the second string is the surname.
5. The data normalization system of claim 1 , wherein the name index includes at least a first set of metaphone strings corresponding to a first language set, and a second set of metaphone strings corresponding to a second language set.
6. The data normalization system of claim 5 , wherein a metaphone string from the first set of metaphone strings and a metaphone string from the second set of metaphone strings both represent an identical name.
7. The data normalization system of claim 1 , wherein the threshold confidence score is based on a reliability level of a data set including the first string and the second string.
8. A method comprising:
converting, using a metaphone algorithm, a first string into a first metaphone string, and a second string into a second metaphone string, the first string being positioned before the second string in an initial string ordering;
searching, based on the first metaphone string and the second metaphone string, a name index including a listing of metaphone strings representing common names and probabilities that the common names are either a given name or a surname;
determining, based on searching the name index, a signal strength that the first metaphone string is a given name, a signal strength that the first metaphone string is a surname, a signal strength that the second metaphone string is a given name, and a signal strength that the second metaphone string is a surname;
determining an absolute difference between the signal strength that the first metaphone string is a surname and the signal strength that the first metaphone string is a given name, yielding an absolute signal difference for the first string;
determining an absolute difference between the signal strength that the second metaphone string is a surname and the signal strength that the second metaphone string is a given name, yielding an absolute signal difference for the second string;
determining that the absolute signal difference for the first string is greater than the absolute signal difference for the second string;
determining, based on the absolute difference for the first string, a confidence score indicating a confidence level that the first string represents the given name and that the second string represents the surname;
determining that the confidence score meets or exceeds a threshold confidence score; and
in response to determining that the confidence score meets or exceeds the threshold confidence score, ordering the first string and the second string according to an updated string, wherein the method is performed by one or more computer processors.
9. The method of claim 8 , further comprising:
determining that the second string includes a set of characters or patterns indicating that the second string is the surname.
10. The method of claim 9 , wherein determining that the second string includes the set of characters or patterns indicating that the second string is a surname comprises:
analyzing the second string based on a list of known character sets or patterns included in surnames; and
determining that the set of characters included in the second string matches a known character set or pattern included in the list of known character sets or patterns included in surnames.
11. The method of claim 9 , wherein determining the confidence score is further based on determining that the second string includes the set of characters or patterns indicating that the second string is the surname.
12. The method of claim 8 , wherein the name index includes at least a first set of metaphone strings corresponding to a first language set, and a second set of metaphone strings corresponding to a second language set.
13. The method of claim 12 , wherein a metaphone string from the first set of metaphone strings and a metaphone string from the second set of metaphone strings both represent an identical name.
14. The method of claim 8 , wherein the threshold confidence score is based on a reliability level of a data set including the first string and the second string.
15. A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors of a data normalization system, causes the data normalization system to perform operations comprising:
converting, using a metaphone algorithm, a first string into a first metaphone string, and a second string into a second metaphone string, the first string being positioned before the second string in an initial string ordering;
searching, based on the first metaphone string and the second metaphone string, a name index including a listing of metaphone strings representing common names and probabilities that the common names are either a given name or a surname;
determining, based on searching the name index, a signal strength that the first metaphone string is a given name, a signal strength that the first metaphone string is a surname, a signal strength that the second metaphone string is a given name, and a signal strength that the second metaphone string is a surname;
determining an absolute difference between the signal strength that the first metaphone string is a surname and the signal strength that the first metaphone string is a given name, yielding an absolute signal difference for the first string;
determining an absolute difference between the signal strength that the second metaphone string is a surname and the signal strength that the second metaphone string is a given name, yielding an absolute signal difference for the second string;
determining that the absolute signal difference for the first string is greater than the absolute signal difference for the second string;
determining, based on the absolute difference for the first string, a confidence score indicating a confidence level that the first string represents the given name and that the second string represents the surname;
determining that the confidence score meets or exceeds a threshold confidence score; and
in response to determining that the confidence score meets or exceeds the threshold confidence score, ordering the first string and the second string according to an updated string ordering based on the absolute signal difference of the first string.
16. The non-transitory computer-readable medium of claim 15 , the operations further comprising:
determining that the second string includes a set of characters or patterns indicating that the second string is the surname.
17. The non-transitory computer-readable medium of claim 16 , wherein determining that the second string includes the set of characters or patterns indicating that the second string is a surname comprises:
analyzing the second string based on a list of known character sets included in surnames; and
determining that the set of characters or patterns included in the second string matches a known character set included in the list of known character sets commonly included in surnames.
18. The non-transitory computer-readable medium of claim 16 , wherein determining the confidence score is further based on determining that the second string includes the set of characters or patterns indicating that the second string is the surname.
19. The non-transitory computer-readable medium of claim 15 , wherein the name index includes at least a first set of metaphone strings corresponding to a first language set, and a second set of metaphone strings corresponding to a second language set.
20. The non-transitory computer-readable medium of claim 15 , wherein the threshold confidence score is based on a reliability level of a data set including the first string and the second string.