Parallel tokenization of date and time information in a distributed network environment
Data in various formats can be protected in a distributed tokenization environment. Examples of such formats include date and time data, decimal data, and floating point data. Such data can tokenized by a security device that instantiates a number of tokenization pipelines for parallel tokenization of the data. Characteristics of such data can be used to tokenize the data. For instance, token tables specific to the data format can be used to tokenized the data. Likewise, a type, order, or configuration of the operations within each tokenization pipeline can be selected based on the data format or characteristics of the data format. Each tokenization pipeline performs a set of encoding or tokenization operations in parallel and based at least in part on a value received from another tokenization pipeline. The tokenization pipeline outputs are combined, producing tokenized data, which can be provided to a remote system for storage or processing.
1. A method comprising:
receiving, at a local computing system, a string of characters comprising a date portion of characters, a time portion of characters, and a microseconds portion of characters, the string of characters representative of a date and time at a microsecond granularity;
querying, by the local computing system, a token server using the date portion of characters to access a first set of token tables, using the time portion of characters to access a second set of token tables, and using the microseconds portion of characters to access a third set of token tables;
instantiating, by the local computing system, a first tokenization pipeline, a second tokenization pipeline, and a third tokenization pipeline configured to operate in parallel to tokenize the string of characters, wherein:
the first tokenization pipeline is configured to perform one or more sequential tokenization operations on the date portion of characters using the first set of token tables to produce a tokenized date portion of characters,
the second tokenization pipeline is configured to perform one or more sequential tokenization operations on the time portion of characters using the second set of token tables to produce a tokenized time portion of characters,
the third tokenization pipe is configured to perform one or more sequential tokenization operations on the microseconds portion of characters using the third set of token tables to produce a tokenized microseconds portion of characters, and
each tokenization pipeline is configured to receive as input a first output of a tokenization operation from a different tokenization pipeline such that the first output of the tokenization operation from the different tokenization pipeline is used by both the tokenization pipeline and the different tokenization pipeline to perform different sequential tokenization operations associated with the tokenization pipeline and the different tokenization pipeline, and to provide a second output of at least one sequential tokenization operation associated with the tokenization pipeline to at least one additional tokenization pipeline such that the second output of the tokenization pipeline is used by both the tokenization pipeline and the additional tokenization pipeline to perform different sequential tokenization operations associated with the tokenization pipeline and the additional tokenization pipeline; and
combining the tokenized date portion of characters, the tokenized time portion of characters, and the tokenized microseconds portion of characters to produce a combined tokenized output and providing, by the local computing system, the combined tokenized output to a remote computing system.
2. The method of claim 1 , wherein the first tokenization pipeline is configured to delay the performance of one or more tokenization operations until the second tokenization pipeline completes the performance of a tokenization operation.
3. The method of claim 2 , wherein the second tokenization pipeline is configured to delay the performance of one or more tokenization operations until the third tokenization pipeline completes the performance of a tokenization operation.
4. The method of claim 1 , wherein the date portion of characters comprises four year characters, two month characters, and two day characters; wherein the time portion of characters comprises two hour characters, two minute characters, and two second characters; and wherein the microseconds portion of characters comprises six microsecond characters.
5. The method of claim 1 , wherein the token server is located remotely from the local computing system and the remote computing system.
6. The method of claim 1 , wherein each tokenization pipeline is configured to perform a processing operation on a set of characters based on an output from a different tokenization pipeline and prior to performing a tokenization operation on the set of characters.
7. The method of claim 1 , wherein a portion of the tokenized date portion of characters, the tokenized time portion of characters, or the tokenized microseconds portion of characters matches the corresponding portion of the date portion of characters, the time portion of characters, or the microseconds portion of characters.
8. A non-transitory computer-readable storage medium storing executable instructions that, when executed by a hardware processor, cause the hardware processor to perform steps comprising:
receiving, at a local computing system, a string of characters comprising a date portion of characters, a time portion of characters, and a microseconds portion of characters, the string of characters representative of a date and time at a microsecond granularity;
querying, by the local computing system, a token server using the date portion of characters to access a first set of token tables, using the time portion of characters to access a second set of token tables, and using the microseconds portion of characters to access a third set of token tables;
instantiating, by the local computing system, a first tokenization pipeline, a second tokenization pipeline, and a third tokenization pipeline configured to operate in parallel to tokenize the string of characters, wherein:
the first tokenization pipeline is configured to perform one or more sequential tokenization operations on the date portion of characters using the first set of token tables to produce a tokenized date portion of characters,
the second tokenization pipeline is configured to perform one or more sequential tokenization operations on the time portion of characters using the second set of token tables to produce a tokenized time portion of characters,
the third tokenization pipe is configured to perform one or more sequential tokenization operations on the microseconds portion of characters using the third set of token tables to produce a tokenized microseconds portion of characters, and
each tokenization pipeline is configured to receive as input a first output of a tokenization operation from a different tokenization pipeline such that the first output of the tokenization operation from the different tokenization pipeline is used by both the tokenization pipeline and the different tokenization pipeline to perform different sequential tokenization operations associated with the tokenization pipeline and the different tokenization pipeline, and to provide a second output of at least one sequential tokenization operation associated with the tokenization pipeline to at least one additional tokenization pipeline such that the second output of the tokenization pipeline is used by both the tokenization pipeline and the additional tokenization pipeline to perform different sequential tokenization operations associated with the tokenization pipeline and the additional tokenization pipeline; and
combining the tokenized date portion of characters, the tokenized time portion of characters, and the tokenized microseconds portion of characters to produce a combined tokenized output and providing, by the local computing system, the combined tokenized output to a remote computing system.
9. The non-transitory computer-readable storage medium of claim 8 , wherein the first tokenization pipeline is configured to delay the performance of one or more tokenization operations until the second tokenization pipeline completes the performance of a tokenization operation.
10. The non-transitory computer-readable storage medium of claim 9 , wherein the second tokenization pipeline is configured to delay the performance of one or more tokenization operations until the third tokenization pipeline completes the performance of a tokenization operation.
11. The non-transitory computer-readable storage medium of claim 8 , wherein the date portion of characters comprises four year characters, two month characters, and two day characters; wherein the time portion of characters comprises two hour characters, two minute characters, and two second characters; and wherein the microseconds portion of characters comprises six microsecond characters.
12. The non-transitory computer-readable storage medium of claim 8 , wherein the token server is located remotely from the local computing system and the remote computing system.
13. The non-transitory computer-readable storage medium of claim 8 , wherein each tokenization pipeline is configured to perform a processing operation on a set of characters based on an output from a different tokenization pipeline and prior to performing a tokenization operation on the set of characters.
14. The non-transitory computer-readable storage medium of claim 8 , wherein a portion of the tokenized date portion of characters, the tokenized time portion of characters, or the tokenized microseconds portion of characters matches the corresponding portion of the date portion of characters, the time portion of characters, or the microseconds portion of characters.
15. A system comprising:
a hardware processor; and
a non-transitory computer-readable storage medium storing executable instructions that, when executed by the hardware processor, cause the hardware processor to perform steps comprising:
receiving, at a local computing system, a string of characters comprising a date portion of characters, a time portion of characters, and a microseconds portion of characters, the string of characters representative of a date and time at a microsecond granularity;
querying, by the local computing system, a token server using the date portion of characters to access a first set of token tables, using the time portion of characters to access a second set of token tables, and using the microseconds portion of characters to access a third set of token tables;
instantiating, by the local computing system, a first tokenization pipeline, a second tokenization pipeline, and a third tokenization pipeline configured to operate in parallel to tokenize the string of characters, wherein:
the first tokenization pipeline is configured to perform one or more sequential tokenization operations on the date portion of characters using the first set of token tables to produce a tokenized date portion of characters,
the second tokenization pipeline is configured to perform one or more sequential tokenization operations on the time portion of characters using the second set of token tables to produce a tokenized time portion of characters,
the third tokenization pipe is configured to perform one or more sequential tokenization operations on the microseconds portion of characters using the third set of token tables to produce a tokenized microseconds portion of characters, and
each tokenization pipeline is configured to receive as input a first output of a tokenization operation from a different tokenization pipeline such that the first output of the tokenization operation from the different tokenization pipeline is used by both the tokenization pipeline and the different tokenization pipeline to perform different sequential tokenization operations associated with the tokenization pipeline and the different tokenization pipeline, and to provide a second output of at least one sequential tokenization operation associated with the tokenization pipeline to at least one additional tokenization pipeline such that the second output of the tokenization pipeline is used by both the tokenization pipeline and the additional tokenization pipeline to perform different sequential tokenization operations associated with the tokenization pipeline and the additional tokenization pipeline; and
combining the tokenized date portion of characters, the tokenized time portion of characters, and the tokenized microseconds portion of characters to produce a combined tokenized output and providing, by the local computing system, the combined tokenized output to a remote computing system.
16. The system of claim 15 , wherein the first tokenization pipeline is configured to delay the performance of one or more tokenization operations until the second tokenization pipeline completes the performance of a tokenization operation.
17. The system of claim 16 , wherein the second tokenization pipeline is configured to delay the performance of one or more tokenization operations until the third tokenization pipeline completes the performance of a tokenization operation.
18. The system of claim 15 , wherein the date portion of characters comprises four year characters, two month characters, and two day characters; wherein the time portion of characters comprises two hour characters, two minute characters, and two second characters; and wherein the microseconds portion of characters comprises six microsecond characters.
19. The system of claim 15 , wherein the token server is located remotely from the local computing system and the remote computing system.
20. The system of claim 15 , wherein each tokenization pipeline is configured to perform a processing operation on a set of characters based on an output from a different tokenization pipeline and prior to performing a tokenization operation on the set of characters.