Training a tokenizer using altered text data

1 . A method, comprising:

retrieving, by a server system from a source database, a set of original text data including one or more characters;

identifying, by the server system, a plurality of strings included in the set of original text data;

altering, by the server system, one or more of the plurality of strings included in the set of original text data to generate a set of altered text data, wherein the altering includes:

selecting one or more current characters located one third of the length of one or more strings included in the plurality of strings to be altered; and

replacing the one or more current characters of the one or more strings to be altered with one or more randomly selected replacement characters;

training, by the server system using the set of altered text data, a tokenizer, wherein the training includes:

assigning a string frequency threshold to the tokenizer, including inputting the string frequency threshold into the tokenizer; and

inputting the set of altered text data into the tokenizer, wherein the tokenizer generates a set of tokens for strings in the set of altered text data, and wherein a number of tokens generated by the tokenizer and included in the set of tokens is based on the string frequency threshold; and

storing, by the server system, the trained tokenizer, including the set of tokens for strings in the set of altered text data.

2 . The method of claim 1 , further comprising:

training, by the server system, a machine learning model by:

inputting the set of original text data into the trained tokenizer, wherein the trained tokenizer:

compares strings in the set of original text data with the set of tokens for the set of altered text data; and

generates, based on the comparing, a new set of tokens for the strings in the set of original text data;

inputting the new set of tokens for strings in the set of original text data into the machine learning model; and

updating, based on comparing output of the machine learning model for tokens in the new set of tokens with known labels for tokens in the new set of tokens, one or more weights of the machine learning model.

3 . The method of claim 2 , further comprising:

performing, by the server system, natural language processing on a new set of text data, including identifying, using the trained machine learning model, one or more terms included in unidentified strings of the new set of text data.

4 . The method of claim 1 , wherein the randomly selected replacement characters are selected:

from an allowed list of characters including a subset of a set of available characters.

5 . The method of claim 1 , wherein the altering includes:

identifying that a given string in the set of original text data includes a space; and

swapping a first set of characters located before the space within the given string with a second set of characters located after the space within the given string.

6 . The method of claim 1 , wherein the set of altered text data includes one or more altered strings and one or more original strings included in the original set of text data that have not been altered.

7 . The method of claim 1 , wherein altering a given string includes:

randomly selecting two current characters included in the given string; and

switching locations of the two current characters within the given string.

8 . The method of claim 1 , wherein assigning the string frequency threshold includes:

determining, based on the identified plurality of strings, an occurrence frequency of respective strings included in the set of original text data;

determining, based on the determined occurrence frequencies of the respective strings, an average string occurrence frequency for the set of original text data; and

selecting, based on the average string occurrence frequency, the string frequency threshold that is less than the average string occurrence frequency.

9 . The method of claim 1 , wherein one or more strings in the set of original text data further include one or more of: symbols, special characters, and numbers.

10 . A non-transitory computer-readable medium having instructions stored thereon that are executable by a server system to perform operations comprising:

retrieving, from a source database, a set of original text data including one or more characters;

identifying a plurality of strings included in the set of original text data;

altering one or more of the plurality of strings included in the set of original text data to generate a set of altered text data, wherein the altering includes selecting one or more current characters included in respective ones of the plurality of strings to be altered;

training, using the set of altered text data, a tokenizer, wherein the training includes:

assigning a string frequency threshold to the tokenizer, including inputting the string frequency threshold into the tokenizer; and

inputting the set of altered text data into the tokenizer, wherein the tokenizer generates a set of tokens for strings in the set of altered text data, and wherein a number of tokens generated by the tokenizer and included in the set of tokens is based on the string frequency threshold; and

training, using the trained tokenizer, a machine learning model, including:

executing the trained tokenizer to generate a new set of tokens for strings in the set of original text data, wherein the trained tokenizer generates the new set of tokens based on comparing strings in the set of original text data with the set of tokens for the set of altered text data;

inputting the new set of tokens for strings in the set of original text data into the machine learning model; and

updating the machine learning model based on comparing output of the machine learning model for tokens in the new set of tokens with known labels for tokens in the new set of tokens.

11 . The non-transitory computer-readable medium of claim 10 , wherein the altering includes:

identifying that a given string in the set of original text data includes a space; and

swapping a first set of characters located before the space within the given string with a second set of characters located after the space within the given string.

12 . The non-transitory computer-readable medium of claim 10 , further comprising:

performing, by the server system, natural language processing on a new set of text data, including identifying, using the trained machine learning model, one or more terms included in unidentified strings of the new set of text data.

13 . The non-transitory computer-readable medium of claim 10 , wherein the altering includes:

randomly selecting, from an allowed list of characters including a subset of a set of available characters, one or more replacement characters for the selected one or more current characters; and

replacing the one or more current characters of a given string with the randomly selected one or more replacement characters.

14 . The non-transitory computer-readable medium of claim 10 , wherein altering the set of text data further includes:

augmenting one or more of the plurality of strings included in the set of original text data by adding noise to the one or more strings.

15 . A system, comprising:

a processor; and

a non-transitory computer-readable medium having instructions stored thereon that are executable by the processor to cause the system to perform operations comprising:

retrieving, from a source database, a set of original text data including one or more characters;

identifying a plurality of strings included in the set of original text data;

altering one or more of the plurality of strings included in the set of original text data to generate a set of altered text data, wherein the altering includes:

selecting one or more current characters located one third of the length of one or more strings included in the plurality of strings to be altered; and

replacing the one or more current characters of the one or more strings to be altered with one or more randomly selected replacement characters;

training, using the set of altered text data, a tokenizer, wherein the training includes:

assigning a string frequency threshold to the tokenizer, including inputting the string frequency threshold into the tokenizer; and

inputting the set of altered text data into the tokenizer, wherein the tokenizer generates a set of tokens for strings in the set of altered text data, and wherein a number of tokens generated by the tokenizer and included in the set of tokens is based on the string frequency threshold; and

storing the trained tokenizer, including the set of tokens for strings in the set of altered text data.

16 . The system of claim 15 , wherein the randomly selected replacement characters are selected

from an allowed list of characters including a subset of a set of available characters.

17 . The system of claim 15 , wherein the altering further includes:

identifying that one or more of the plurality of strings include a space; and

swapping a first set of characters located before the space within the one or more strings with a second set of characters located after the space within the one or more strings.

18 . The system of claim 15 , wherein altering a given string includes:

randomly selecting two current characters included in the given string; and

switching locations of the two current characters within the given string.

19 . The system of claim 15 , wherein assigning the string frequency threshold includes:

determining, based on the identified plurality of strings, an occurrence frequency of respective strings included in the set of original text data;

determining, based on the determined occurrence frequencies of the respective strings, an average string occurrence frequency for the set of original text data; and

selecting, based on the average string occurrence frequency, the string frequency threshold that is less than the average string occurrence frequency.