System and method for protecting specified data combinations
View Patent ↗A method in one example implementation includes extracting a plurality of data elements from a record of a data file, tokenizing the data elements into tokens, and storing the tokens in a first tuple of a registration list. The method further includes selecting one of the tokens as a token key for the first tuple, where the token is selected because it occurs less frequently in the registration list than each of the other tokens in the first tuple. In specific embodiments, at least one data element is an expression element having a character pattern matching a predefined expression pattern that represents at least two words and a separator between the words. In other embodiments, at least one data element is a word defined by a character pattern of one or more consecutive essential characters. Other specific embodiments include determining an end of the record by recognizing a predefined delimiter.
1. A system for protecting data in a network, the system comprising:
one or more hardware processors;
a registration subsystem that when running on at least one of the one or more hardware processors is to:
create a plurality of tuples, wherein each tuple includes a respective set of data file tokens, each set of data file tokens corresponding to a respective set of data elements in a data file;
select a data file token of a first tuple as a token key to index the first tuple, wherein the data file token is selected as the token key based on determining the data file token occurs with less frequency across the plurality of tuples than frequencies at which other data file tokens of the first tuple occur across the plurality of tuples; and
create an index table with a particular index including the token key; and
a detection subsystem that when running on at least one of the one or more hardware processors is to:
tokenize data elements in an object into corresponding object tokens, the object captured via network traffic traversing the network;
identify the first tuple based on determining the token key of the particular index in the index table corresponds to a particular object token of the object tokens; and
validate a correspondence between the first tuple and the object based on determining that the data file tokens of the first tuple correspond to the object tokens according to a predetermined threshold.
2. The system of claim 1 , wherein the network traffic is being sent out of the network.
3. The system of claim 1 , wherein the index table is to include a plurality of indexes each including a unique token key.
4. The system of claim 3 , wherein the detection subsystem, when running on at least one of the one or more hardware processors, is to:
store the particular object token as a pending key in a pending key list, the pending key list to include any other object tokens corresponding to respective other token keys of other indexes in the index table.
5. The system of claim 4 , wherein to identify the first tuple is to:
search the index table based on the pending key; and
determine the pending key corresponds to the token key in the particular index, wherein the particular index is to include an offset linked to the token key, the offset indicating a location of the first tuple.
6. The system of claim 1 , wherein, if two or more tuples are indexed by the token key, the particular index is to include two or more unique offsets indicating respective locations of the two or more tuples, wherein two or more sets of data file tokens corresponding to the two or more tuples, respectively, are to each include the token key.
7. The system of claim 1 , wherein a first set of data elements corresponding to a first set of data file tokens is stored in a record of the data file, and wherein a second tuple of the plurality of tuples includes a second set of data file tokens corresponding to a second set of data elements stored in a record of a different data file.
8. At least one non-transitory machine readable medium for protecting data in a network, the at least one non-transitory machine readable medium comprising instructions that, when executed, cause one or more processors to:
create a plurality of tuples, wherein each tuple includes a respective set of data file tokens, each set of data file tokens corresponding to a respective set of data elements in a data file;
select a data file token of a first tuple as a token key to index the first tuple, wherein the data file token is selected as the token key based on determining the data file token occurs with less frequency across the plurality of tuples than frequencies at which other data file tokens of the first tuple occur across the plurality of tuples;
create an index table with a particular index including the token key;
tokenize data elements in an object into corresponding object tokens, the object captured via network traffic traversing the network;
identify the first tuple based on determining the token key of the particular index in the index table corresponds to a particular object token of the object tokens; and
validate a correspondence between the first tuple and the object based on determining that the data file tokens of the first tuple correspond to the object tokens according to a predetermined threshold.
9. The at least one non-transitory machine readable medium of claim 8 , wherein the instructions, when executed, cause the one or more processors to:
generate a prime count table for a plurality of data file tokens in the plurality of tuples, wherein the prime count table is to be generated by forcing each unique data file token into a boundary of memory with modulus, the boundary defined by a prime number, and wherein the prime count table is to include a count for each unique data file token representing a total number of occurrences of the unique data file token in the plurality of tuples.
10. The at least one non-transitory machine readable medium of claim 8 , wherein the network traffic is being sent out of the network.
11. The at least one non-transitory machine readable medium of claim 8 , wherein each object token is to be represented by a bit set in a bit hash table.
12. The at least one non-transitory machine readable medium of claim 11 , wherein the instructions, when executed, cause the one or more processors to:
evaluate the bit hash table to determine if each one of the file tokens in the first tuple corresponds to at least one of the object tokens.
13. The at least one non-transitory machine readable medium of claim 8 , wherein the index table is to include a plurality of indexes each including a unique token key.
14. The at least one non-transitory machine readable medium of claim 13 , wherein the instructions, when executed, cause the one or more processors to:
store the particular object token as a pending key in a pending key list, the pending key list to include any other object tokens corresponding to respective other token keys of other indexes in the index table.
15. The at least one non-transitory machine readable medium of claim 8 , wherein to identify the first tuple is to:
search the index table based on the pending key; and
determine the pending key corresponds to the token key in the particular index, wherein the particular index is to include an offset linked to the token key, the offset indicating a location of the first tuple.
16. The at least one non-transitory machine readable medium of claim 8 , wherein, if two or more tuples are indexed by the token key, the particular index is to include two or more unique offsets indicating respective locations of the two or more tuples, wherein two or more sets of data file tokens corresponding to the two or more tuples, respectively, are to include the token key.
17. The at least one non-transitory machine readable medium of claim 8 , wherein a first set of data elements corresponding to a first set of data file tokens is stored in a record of the data file, the record having a predefined delimiter indicating an end of the first set of data elements to include in the first tuple.
18. The at least one non-transitory machine readable medium of claim 8 , wherein a first set of data elements corresponding to a first set of data file tokens is stored in a record of the data file, and wherein a second tuple of the plurality of tuples includes a second set of data file tokens corresponding to a second set of data elements stored in a record of a different data file.
19. A method for protecting data in a network, the method comprising:
creating, by at least one hardware processor of a registration system, a plurality of tuples, wherein each tuple includes a respective set of data file tokens, each set of data file tokens corresponding to a respective set of data elements in a data file;
selecting a data file token of a first tuple as a token key to index the first tuple, wherein the data file token is selected as the token key based on determining the data file token occurs with less frequency across the plurality of tuples than frequencies at which other data file tokens of the first tuple occur across the plurality of tuples;
creating an index table with a particular index including the token key;
tokenizing, by at least one hardware processor of a detection system, data elements in an object into corresponding object tokens, the object captured via network traffic traversing the network;
identifying the first tuple based on determining the token key of the particular index in the index table corresponds to a particular object token of the object tokens; and
validating a correspondence between the first tuple and the object based on determining that the data file tokens of the first tuple correspond to the object tokens according to a predetermined threshold.
20. The method of claim 19 , wherein the index table includes a plurality of indexes each including a unique token key, wherein the creating the index table includes forcing the unique token keys into a boundary of memory with modulus, wherein the boundary is defined by a prime number.