Computing systems and methods for automatic file tokenization

1 . A system for processing input data, the system comprising:

a memory, a communication interface, and a processor operatively coupled to the memory and the communication interface;

the processor configured to at least:

obtain a data record comprising a plurality of groupings of data elements, each one of the plurality of groupings of data elements associated with a data type;

select a first template for tokenization based on at least a first data type associated with a first given grouping of data elements from amongst the plurality of groupings of data elements, the first template comprising a first tokenization rule and a first detokenization rule;

tokenize at least a portion of each of the data elements in the first grouping of data elements according to the first tokenization rule, to generate a tokenized data record;

generate a tokenization file that is associated with the tokenized data record, the tokenization file comprising the first detokenization rule and the data record; and

transmit the tokenized data record.

2 . The system of claim 1 , wherein the data record comprises a plurality of rows and a plurality of columns, and the plurality of groupings of data elements comprises a plurality of columns.

3 . The system of claim 1 , wherein the processor is further configured to:

select a second template for tokenization based on at least a second data type associated with a second given grouping of data elements from amongst the plurality of groupings of data elements, the second template comprising a second tokenization rule and a second detokenization rule;

tokenize a portion of each of the data elements in the second grouping of data elements according to the second tokenization rule, to further generate the tokenized data record;

add to the tokenization file the second detokenization rule; and

wherein the second tokenization rule is different from the first tokenization rule, and the second detokenization rule is different from the first detokenization rule.

4 . The system of claim 1 further configured to receive a plurality of data records, including the data record, from a plurality of different data sources, and wherein the processor is further configured to generate a plurality of tokenized data records corresponding respectively to the plurality of data records, and output the plurality of tokenized data records to a plurality of different client devices.

5 . The system of claim 4 , wherein the plurality of different data sources operate on at least two or more different operating systems.

6 . The system of claim 1 , wherein the processor is configured to: mask an ancillary portion of each of the data elements in the first grouping of data elements according to a first masking rule in the first template, and store a first demasking rule in the tokenization file.

7 . The system of claim 1 , wherein the data record comprises a plurality of rows and a plurality of columns, a given column comprises the first given grouping of data elements and the first given grouping of data elements are respectively assigned a plurality of original row IDs, and

wherein the processor is further configured to:

generate a plurality of unique IDs in a new unique ID column assigned to each data element in the first given grouping of data elements and to each of the plurality of original row IDs;

sort the plurality of rows according to the plurality of unique IDs to generate a new ordering of the plurality of rows, and assign a plurality of new row IDs in a new row ID column according to the new ordering of the plurality of rows;

replace the first given grouping of data elements with a plurality of incremental token values based on the new ordering of the plurality of rows;

wherein the plurality of incremental token values and the plurality of new row IDs form at least part of the tokenized record.

8 . The system of claim 1 , wherein the processor is further configured to:

identify that a second given grouping of data elements, from amongst the plurality of groupings of data elements, satisfies one or more criteria for erasure; and

replace each of the data elements in the second given grouping of data elements with an empty string.

9 . The system of claim 8 , wherein the tokenization file is unchanged by replacing each of the data elements in the second given grouping of data elements with the empty string.

10 . The system of claim 1 , wherein the tokenized data record is transmitted to a machine learning system.

11 . A method for processing input data, the method executed in a computing environment comprising one or more processors and memory, and the method comprising:

obtaining a data record comprising a plurality of groupings of data elements, each one of the plurality of groupings of data elements associated with a data type;

selecting a first template for tokenization based on at least a first data type associated with a first given grouping of data elements from amongst the plurality of groupings of data elements, the first template comprising a first tokenization rule and a first detokenization rule;

tokenizing at least a portion of each of the data elements in the first grouping of data elements according to the first tokenization rule, to generate a tokenized data record;

generating a tokenization file that is associated with the tokenized data record, the tokenization file comprising the first detokenization rule and the data record; and

transmitting the tokenized data record.

12 . The method of claim 11 , wherein the data record comprises a plurality of rows and a plurality of columns, and the plurality of groupings of data elements comprises a plurality of columns.

13 . The method of claim 11 , further comprising:

selecting a second template for tokenization based on at least a second data type associated with a second given grouping of data elements from amongst the plurality of groupings of data elements, the second template comprising a second tokenization rule and a second detokenization rule;

tokenizing a portion of each of the data elements in the second grouping of data elements according to the second tokenization rule, to further generate the tokenized data record;

adding to the tokenization file the second detokenization rule; and

wherein the second tokenization rule is different from the first tokenization rule, and the second detokenization rule is different from the first detokenization rule.

14 . The method of claim 11 , further comprising: receiving a plurality of data records, including the data record, from a plurality of different data sources; generating a plurality of tokenized data records corresponding respectively to the plurality of data records; and outputting the plurality of tokenized data records to a plurality of different client devices.

15 . The method of claim 14 , wherein the plurality of different data sources operate on at least two or more different operating systems.

16 . The method of claim 11 , further comprising: masking an ancillary portion of each of the data elements in the first grouping of data elements according to a first masking rule in the first template, and store a first demasking rule in the tokenization file.

17 . The method of claim 11 , wherein the data record comprises a plurality of rows and a plurality of columns, a given column comprises to the first given grouping of data elements and the first given grouping of data elements are respectively assigned a plurality of original row IDs, and

wherein the method further comprises:

generating a plurality of unique IDs in a new unique ID column assigned to each data element in the first given grouping of data elements and to each of the plurality of original row IDs;

sorting the plurality of rows according to the plurality of unique IDs to generate a new ordering of the plurality of rows, and assign a plurality of new row IDs in a new row ID column according to the new ordering of the plurality of rows;

replacing the first given grouping of data elements with a plurality of incremental token values based on the new ordering of the plurality of rows; and

wherein the plurality of incremental token values and the plurality of new row IDs form at least part of the tokenized record.

18 . The method of claim 11 , further comprising:

identifying that a second given grouping of data elements, from amongst the plurality of groupings of data elements, satisfies one or more criteria for erasure; and

replacing each of the data elements in the second given grouping of data elements with an empty string.

19 . The method of claim 18 , wherein the tokenization file is unchanged by replacing each of the data elements in the second given grouping of data elements with the empty string.

20 . A non-transitory computer readable medium storing computer executable instructions which, when executed by at least one computer processor, cause the at least one computer processor to carry out a method for processing input data, and the method comprising:

obtaining a data record comprising a plurality of groupings of data elements, each one of the plurality of groupings of data elements associated with a data type;

selecting a first template for tokenization based on at least a first data type associated with a first given grouping of data elements from amongst the plurality of groupings of data elements, the first template comprising a first tokenization rule and a first detokenization rule;

tokenizing at least a portion of each of the data elements in the first grouping of data elements according to the first tokenization rule, to generate a tokenized data record;

generating a tokenization file that is associated with the tokenized data record, the tokenization file comprising the first detokenization rule and the data record; and

transmitting the tokenized data record.