Parallel unicode tokenization in a distributed network environment
Unicode data can be protected in a distributed tokenization environment. Data to be tokenized can be accessed or received by a security server, which instantiates a number of tokenization pipelines for parallel tokenization of the data. Unicode token tables are accessed by the security server, and each tokenization pipeline uses the accessed token tables to tokenization a portion of the data. Each tokenization pipeline performs a set of encoding or tokenization operations in parallel and based at least in part on a value received from another tokenization pipeline. The outputs of the tokenization pipelines are combined, producing tokenized data, which can be provided to a remote computing system for storage or processing.
1. A method comprising:
receiving, at a local computing system, a string of characters in a character domain represented by Unicode;
accessing, by the local computing system, a set of token tables, each token table mapping, for each character in the character domain, a Unicode code value for the character to a different token value;
instantiating, by the local computing system, a set of tokenization pipelines configured to operate in parallel to tokenize the string of characters, each tokenization pipeline corresponding to a different subset of the string of characters and configured to:
access a subset of the set of token tables;
perform one or more sequential tokenization operations on the subset of the string of characters using the accessed subset of token tables, each tokenization operation comprising replacing an input value with a token value mapped to the input value by a token table of the accessed subset of token tables; and
output a tokenized set of characters based on a last tokenization operation of the sequential tokenization operations;
wherein a first input value within a first tokenization pipeline is based on a Unicode code value corresponding to at least one character of the subset of the string of characters, and wherein a second input value within the first tokenization pipeline is based on an input value or a token value from a different tokenization pipeline; and
combining the tokenized sets of characters output by each tokenization pipeline and providing, by the local computing system, the combined tokenized sets of characters to a remote computing system.
2. The method of claim 1 , wherein the character domain comprises one or more of:
Latin characters, Greek characters, Cyrillic characters, Armenian characters, Hebrew characters, Arabic characters, Syriac characters, Thaana characters, Devanagari characters, Bengali characters, Gurmukhi characters, Oriya characters, Tamil characters, Telugu characters, Kannada characters, Malayalam characters, Sinhala characters, Thai characters, Lao characters, Tibetan characters, Myanmar characters, Georgian characters, Hangul characters, Ethiopic characters, Cherokee characters, Canadian Aboriginal Syllabics, Khmer characters, Mongolian characters, Han ideographs, Hiragana characters, Katakana characters, and Yi characters.
3. The method of claim 1 , wherein the string of characters is modified by performing one or more initialization operations on the string of characters prior to being tokenized.
4. The method of claim 1 , wherein an input value of a tokenization operation of a tokenization pipeline comprises a token value from a previous tokenization operation within the sequential tokenization operations.
5. The method of claim 1 , wherein an input value of a tokenization operation of a tokenization pipeline comprises a token value from a previous tokenization operation within the sequential tokenization operations modified by an initialization vector.
6. The method of claim 1 , wherein an input value of a tokenization operation of a tokenization pipeline comprises one of the input value or the token value from the different tokenization pipeline.
7. The method of claim 1 , wherein an input value of a tokenization operation of a tokenization pipeline comprises one of the input value or the token value from the different tokenization pipeline modified by an initialization vector.
8. A non-transitory computer-readable storage medium storing executable instructions that, when executed by a hardware processor, cause the hardware processor to perform steps comprising:
receiving, at a local computing system, a string of characters in a character domain represented by Unicode;
accessing, by the local computing system, a set of token tables, each token table mapping, for each character in the character domain, a Unicode code value for the character to a different token value;
instantiating, by the local computing system, a set of tokenization pipelines configured to operate in parallel to tokenize the string of characters, each tokenization pipeline corresponding to a different subset of the string of characters and configured to:
access a subset of the set of token tables;
perform one or more sequential tokenization operations on the subset of the string of characters using the accessed subset of token tables, each tokenization operation comprising replacing an input value with a token value mapped to the input value by a token table of the accessed subset of token tables; and
output a tokenized set of characters based on a last tokenization operation of the sequential tokenization operations;
wherein a first input value within a first tokenization pipeline is based on a Unicode code value corresponding to at least one character of the subset of the string of characters, and wherein a second input value within the first tokenization pipeline is based on an input value or a token value from a different tokenization pipeline; and
combining the tokenized sets of characters output by each tokenization pipeline and providing, by the local computing system, the combined tokenized sets of characters to a remote computing system.
9. The non-transitory computer-readable storage medium of claim 8 , wherein the character domain comprises one or more of: Latin characters, Greek characters, Cyrillic characters, Armenian characters, Hebrew characters, Arabic characters, Syriac characters, Thaana characters, Devanagari characters, Bengali characters, Gurmukhi characters, Oriya characters, Tamil characters, Telugu characters, Kannada characters, Malayalam characters, Sinhala characters, Thai characters, Lao characters, Tibetan characters, Myanmar characters, Georgian characters, Hangul characters, Ethiopic characters, Cherokee characters, Canadian Aboriginal Syllabics, Khmer characters, Mongolian characters, Han ideographs, Hiragana characters, Katakana characters, and Yi characters.
10. The non-transitory computer-readable storage medium of claim 8 , wherein the string of characters is modified by performing one or more initialization operations on the string of characters prior to being tokenized.
11. The non-transitory computer-readable storage medium of claim 8 , wherein an input value of a tokenization operation of a tokenization pipeline comprises a token value from a previous tokenization operation within the sequential tokenization operations.
12. The non-transitory computer-readable storage medium of claim 8 , wherein an input value of a tokenization operation of a tokenization pipeline comprises a token value from a previous tokenization operation within the sequential tokenization operations modified by an initialization vector.
13. The non-transitory computer-readable storage medium of claim 8 , wherein an input value of a tokenization operation of a tokenization pipeline comprises one of the input value or the token value from the different tokenization pipeline.
14. The non-transitory computer-readable storage medium of claim 8 , wherein an input value of a tokenization operation of a tokenization pipeline comprises one of the input value or the token value from the different tokenization pipeline modified by an initialization vector.
15. A system comprising a hardware processor and a non-transitory computer-readable storage medium storing executable instructions that, when executed by the hardware processor, cause the hardware processor to perform steps comprising:
receiving a string of characters in a character domain represented by Unicode;
accessing a set of token tables, each token table mapping, for each character in the character domain, a Unicode code value for the character to a different token value;
instantiating a set of tokenization pipelines configured to operate in parallel to tokenize the string of characters, each tokenization pipeline corresponding to a different subset of the string of characters and configured to:
access a subset of the set of token tables;
perform one or more sequential tokenization operations on the subset of the string of characters using the accessed subset of token tables, each tokenization operation comprising replacing an input value with a token value mapped to the input value by a token table of the accessed subset of token tables; and
output a tokenized set of characters based on a last tokenization operation of the sequential tokenization operations;
wherein a first input value within a first tokenization pipeline is based on a Unicode code value corresponding to at least one character of the subset of the string of characters, and wherein a second input value within the first tokenization pipeline is based on an input value or a token value from a different tokenization pipeline; and
combining the tokenized sets of characters output by each tokenization pipeline and providing, by the local computing system, the combined tokenized sets of characters to a remote computing system.
16. The system of claim 15 , wherein the character domain comprises one or more of:
Latin characters, Greek characters, Cyrillic characters, Armenian characters, Hebrew characters, Arabic characters, Syriac characters, Thaana characters, Devanagari characters, Bengali characters, Gurmukhi characters, Oriya characters, Tamil characters, Telugu characters, Kannada characters, Malayalam characters, Sinhala characters, Thai characters, Lao characters, Tibetan characters, Myanmar characters, Georgian characters, Hangul characters, Ethiopic characters, Cherokee characters, Canadian Aboriginal Syllabics, Khmer characters, Mongolian characters, Han ideographs, Hiragana characters, Katakana characters, and Yi characters.
17. The system of claim 15 , wherein the string of characters is modified by performing one or more initialization operations on the string of characters prior to being tokenized.
18. The system of claim 15 , wherein an input value of a tokenization operation of a tokenization pipeline comprises a token value from a previous tokenization operation within the sequential tokenization operations.
19. The system of claim 15 , wherein an input value of a tokenization operation of a tokenization pipeline comprises a token value from a previous tokenization operation within the sequential tokenization operations modified by an initialization vector.
20. The system of claim 15 , wherein an input value of a tokenization operation of a tokenization pipeline comprises one of the input value or the token value from the different tokenization pipeline.
21. The system of claim 15 , wherein an input value of a tokenization operation of a tokenization pipeline comprises one of the input value or the token value from the different tokenization pipeline modified by an initialization vector.