Systems and methods for tokenizing user-annotated names
A disclosed computer-implemented method for tokenizing user-annotated names may include (1) identifying an example set of user-annotated names, (2) creating a custom dictionary that includes known keywords by (a) extracting a set of known keywords from the example set of user-annotated names and (b) assigning a frequency score to each known keyword in the set of known keywords based on the respective frequency of each known keyword within the example set, and (3) enabling the computing device to tokenize an additional user-annotated name of arbitrary structure by performing a semantic analysis including (a) assigning, using the custom dictionary, a frequency score to a substring of the additional user-annotated name based on the substring matching the known keyword and (b) splitting the additional user-annotated name into tokens according to a permutation of substrings that received a top combined frequency score. Various other methods, systems, and computer-readable media are also disclosed.
1. A computer-implemented method for tokenizing user-annotated names, at least a portion of the method being performed by a computing device comprising at least one processor, the method comprising:
identifying an example set of user-annotated names that represent strings of text used to identify digital entities within a particular organization;
creating, based on the example set of user-annotated names, a custom dictionary that comprises known keywords by:
extracting a set of known keywords from the example set of user-annotated names;
assigning, based on the respective frequency of each of the known keywords within the example set of user-annotated names, a frequency score to each known keyword in the set of known keywords that represents a frequency at which the known keyword appears within the example set of user-annotated names;
enabling the computing device to tokenize an additional user-annotated name of arbitrary structure by performing a semantic analysis comprising:
applying a tokenization algorithm that, at each iterative step of the tokenization algorithm:
accepts a text string as a string input;
generates each possible single split of the string input, generating a total number of permutations equal to the length of the string input;
uses, as an input to a current iterative step in the tokenization algorithm, a previous output of the tokenization algorithm that used one side of the single split as a previous string input to the tokenization algorithm; and
assigns, using the custom dictionary, a frequency score to the other side of the single split based on whether the substring matches a known keyword stored in the custom dictionary; and
splitting the additional user-annotated name into tokens according to an output of the tokenization algorithm that used the additional user-annotated name as an initial input and received a top combined frequency score, thereby enabling a computing system to process the additional user-annotated name according to the tokens produced as the output of the tokenization algorithm.
2. The method of claim 1 , wherein creating the custom dictionary that comprises known keywords further comprises splitting each entry in the example set of user-annotated names stored in metadata along hard delimiters into preliminary keywords.
3. The method of claim 2 , wherein creating the custom dictionary that comprises known keywords further comprises adding preliminary keywords that match a regular expression to the custom dictionary that comprises known keywords.
4. The method of claim 3 , wherein the regular expression defines at least one of the following sequences:
all alphanumeric capital letters;
all alphanumeric lowercase letters;
a single alphanumeric capital letter followed by a series of alphanumeric lowercase letters.
5. The method of claim 2 , wherein the hard delimiters consist of non-alphanumeric characters.
6. The method of claim 1 , wherein creating the custom dictionary that comprises known keywords further comprises identifying an additional dictionary that comprises known keywords.
7. The method of claim 6 , wherein assigning, using the custom dictionary, a frequency score for a known keyword to a substring of the additional user-annotated name further comprises applying a weighted interpolation function, wherein the weighted interpolation function:
assigns a dictionary weight value to the custom dictionary and an additional dictionary weight value to the additional dictionary;
merges, based on the weight value and the additional weight value, the frequency scores for the known keyword from the custom dictionary and the additional dictionary into a merged frequency score.
8. The method of claim 6 , wherein the additional dictionary that comprises known keywords comprises one of:
a natural language dictionary;
a dictionary of user-defined known keywords;
a previously generated custom dictionary of known keywords.
9. The method of claim 1 , wherein splitting the additional user-annotated name into tokens comprises applying a function to a substring that uses a previous output of the function as an input to the function.
10. The method of claim 1 , wherein the semantic analysis further comprises assigning a penalty score as the frequency score of a substring that is not found in the custom dictionary that comprises known keywords.
11. A system for tokenizing user-annotated names, the system comprising:
an identification module, stored in memory, that identifies an example set of user-annotated names that represent strings of text used to identify digital entities within a particular organization;
a dictionary creation module, stored in memory, that creates, based on the example set of user-annotated names, a custom dictionary of known keywords by:
extracting a set of known keywords from the example set of user-annotated names;
assigning, based on the respective frequency of each of the known keywords within the example set of user-annotated names, a frequency score to each known keyword in the set of known keywords that represents a frequency at which the known keyword appears within the example set of user-annotated names;
a tokenizing module, stored in memory, that enables a computing device to tokenize an additional user-annotated name of arbitrary structure by performing a semantic analysis comprising:
applying a tokenization algorithm that, at each iterative step of the tokenization algorithm:
accepts a text string as a string input;
generates each possible single split of the string input, generating a total number of permutations equal to the length of the string input;
uses, as an input to a current iterative step in the tokenization algorithm, a previous output of the tokenization algorithm that used one side of the single split as a previous string input to the tokenization algorithm; and
assigns a frequency score to the other side of the single split based on whether the substring matches a known keyword stored in the custom dictionary;
splitting the additional user-annotated name into tokens according to an output of the tokenization algorithm that used the additional user-annotated name as an initial input and received a top combined frequency score, thereby enabling a computing system to process the additional user-annotated name according to the tokens produced as the output of the tokenization algorithm;
at least one physical processor configured to execute the identification module, the dictionary creation module, and the tokenizing module.
12. The system of claim 11 , wherein the dictionary creation module further creates the custom dictionary of known keywords by splitting each entry in the example set of user-annotated names stored in metadata along hard delimiters into preliminary keywords.
13. The system of claim 12 , wherein the dictionary creation module further creates the custom dictionary of known keywords by adding preliminary keywords that match a regular expression to the custom dictionary of known keywords.
14. The system of claim 13 , wherein the regular expression defines at least one of the following sequences:
all alphanumeric capital letters;
all alphanumeric lowercase letters;
a single alphanumeric capital letter followed by a series of alphanumeric lowercase letters.
15. The system of claim 12 , wherein the hard delimiters consist of non-alphanumeric characters.
16. The system of claim 11 , wherein the dictionary creation module further creates the custom dictionary of known keywords by identifying an additional dictionary that comprises known keywords.
17. The system of claim 16 , wherein the dictionary creation module further assigns, using the custom dictionary, a frequency score for a known keyword to a substring of the additional user-annotated name by applying a weighted interpolation function, wherein the weighted interpolation function:
assigns a dictionary weight value to the custom dictionary and an additional dictionary weight value to the additional dictionary;
merges, based on the weight value and the additional weight value, the frequency scores for the known keyword from the custom dictionary and the additional dictionary into a merged frequency score.
18. The system of claim 16 , wherein the additional dictionary that comprises known keywords comprises one of:
a natural language dictionary;
a dictionary of user-defined known keywords;
a previously generated custom dictionary of known keywords.
19. The system of claim 11 , wherein the tokenizing module further splits the additional user-annotated name into tokens by applying a function to a substring that uses a previous output of the function as an input to the function.
20. A non-transitory computer-readable medium comprising one or more computer-readable instructions that, when executed by at least one processor of a computing device, cause the computing device to:
identify an example set of user-annotated names that represent strings of text used to identify digital entities within a particular organization;
create, based on the example set of user-annotated names, a custom dictionary that comprises known keywords by:
extracting a set of known keywords from the example set of user-annotated names;
assigning, based on the respective frequency of each of the known keywords within the example set of user-annotated names, a frequency score to each known keyword in the set of known keywords that represents a frequency at which the known keyword appears within the example set of user-annotated names;
enable tokenization of an additional user-annotated name of arbitrary structure by performing a semantic analysis comprising:
applying a tokenization algorithm that, at each iterative step of the tokenization algorithm:
accepts a text string as a string input;
generates each possible single split of the string input, generating a total number of permutations equal to the length of the string input;
uses, as an input to a current iterative step in the tokenization algorithm, a previous output of the tokenization algorithm that used one side of the single split as a previous string input to the tokenization algorithm; and
assigns, using the custom dictionary, a frequency score to the other side of the single split based on whether the substring matches a known keyword stored in the custom dictionary; and
splitting the additional user-annotated name into tokens according to an output of the tokenization algorithm that used the additional user-annotated name as an initial input and received a top combined frequency score, thereby enabling a computing system to process the additional user-annotated name according to the tokens produced as the output of the tokenization algorithm.