System and method for protecting specified data combinations
A method in one example implementation includes extracting a plurality of data elements from a record of a data file, tokenizing the data elements into tokens, and storing the tokens in a first tuple of a registration list. The method further includes selecting one of the tokens as a token key for the first tuple, where the token is selected because it occurs less frequently in the registration list than each of the other tokens in the first tuple. In specific embodiments, at least one data element is an expression element having a character pattern matching a predefined expression pattern that represents at least two words and a separator between the words. In other embodiments, at least one data element is a word defined by a character pattern of one or more consecutive essential characters. Other specific embodiments include determining an end of the record by recognizing a predefined delimiter.
1. At least one non-transitory, computer readable medium comprising instructions that, when executed, cause one or more processors to:
tokenize a plurality of data elements in an object into a plurality of object tokens;
identify, in an index table of token keys, a token key that corresponds to an object token of the plurality of object tokens;
identify a first tuple of a plurality of tuples in a registration list based, at least in part, on the token key, wherein the first tuple includes a set of registered tokens that represents a set of data elements, and the token key is a registered token in the set of registered tokens;
determine a number of the registered tokens that are found in at least one object token of the plurality of object tokens; and
take an action based on determining that the number of the registered tokens satisfies a predetermined threshold associated with the set of data elements, wherein the action is preventing transmission of the object or locking down a storage repository.
2. The at least one computer readable medium of claim 1 , wherein the instructions, when executed by the at least one processor:
create a pending key list based on each token key in the index table that corresponds to one or more object tokens of the plurality of object tokens.
3. The at least one computer readable medium of claim 2 , wherein the identifying the first tuple includes:
selecting a pending key from the pending key list;
identifying the token key in the index table based on the selected pending key; and
using an offset associated with the token key in the index table to identify the first tuple, wherein the offset indicates a location of the first tuple in the registration list.
4. The at least one computer readable medium of claim 1 , wherein the registered token occurs with less frequency across the plurality of tuples in the registration list than frequencies at which other registered tokens of the first tuple occur across the plurality of tuples in the registration list.
5. The at least one computer readable medium of claim 1 , wherein the predetermined threshold is satisfied based on each registered token in the set of registered tokens being found in at least one of the plurality of object tokens.
6. The at least one computer readable medium of claim 1 , wherein the instructions, when executed by the at least one processor:
represent the plurality of object tokens in a bit hash table by setting a respective bit in the bit hash table for unique object tokens of the plurality of object tokens.
7. The at least one computer readable medium of claim 6 , wherein determining the number of the registered tokens includes determining, for each registered token of the set of registered tokens, whether a bit is set in a bit position of the bit hash table that corresponds to that registered token.
8. The at least one computer readable medium of claim 1 , wherein the data elements of the plurality of data elements are tokenized by converting each data element to a respective hash value.
9. The at least one computer readable medium of claim 1 , wherein the object is intercepted in network traffic sent out of a network.
10. The at least one computer readable medium of claim 1 , wherein the index table includes a plurality of indexes, each index including a unique token key and an offset to a location in the registration list.
11. The at least one computer readable medium of claim 1 , wherein the instructions, when executed by the at least one processor:
prior to tokenizing a data element of the data elements in the object, identify the data element in the object and extract the data element from the object.
12. The at least one computer readable medium of claim 1 , wherein, if two or more tuples are indexed by the token key, an index is to include two or more unique offsets indicating respective locations of the two or more tuples, each of the two or more tuples is to include a respective set of data file tokens, and each of the respective sets of data file tokens is to include the token key.
13. An apparatus, comprising:
a memory device including a set of instructions; and
a processor, coupled to the memory device, that, when executing the set of instructions, is to
tokenize a plurality of data elements in an object into a plurality of object tokens;
identify, in an index table of token keys, a token key that corresponds to an object token of the plurality of object tokens;
identify a tuple of a plurality of tuples in a registration list based, at least in part, on the token key, wherein the tuple includes a set of registered tokens that represents a set of data elements, and the token key is a registered token in the set of registered tokens;
determine a number of the registered tokens that are found in at least one object token of the plurality of object tokens; and
take an action based on determining that the number of the registered tokens satisfies a predetermined threshold associated with the set of data elements, wherein the action is preventing transmission of the object or locking down a storage repository.
14. The apparatus of claim 13 , wherein the processor, when executing the set of instructions, is to:
create a pending key list based on each token key in the index table that corresponds to one or more object tokens of the plurality of object tokens.
15. The apparatus of claim 14 , wherein identifying the first tuple includes:
selecting a pending key from the pending key list;
identifying the token key in the index table based on the selected pending key; and
using an offset associated with the token key in the index table to identify the tuple, wherein the offset indicates a location of the tuple in the registration list.
16. The apparatus of claim 13 , wherein the registered token occurs with less frequency across the plurality of tuples in the registration list than frequencies at which other registered tokens of the tuple occur across the plurality of tuples in the registration list.
17. The apparatus of claim 13 , wherein the data elements of the plurality of data elements are tokenized by converting each data element to a respective hash value.
18. A method, the method comprising:
tokenizing a plurality of data elements in an object into a plurality of object tokens;
identifying, in an index table of token keys, a token key that corresponds to an object token of the plurality of object tokens;
identifying a tuple of a plurality of tuples in a registration list based, at least in part, on the token key, wherein the tuple includes a set of registered tokens that represents a set of data elements, and the token key is a registered token in the set of registered tokens;
determining a number of the registered tokens that are found in at least one object token of the plurality of object tokens; and
taking an action based on determining that the number of the registered tokens satisfies a predetermined threshold associated with the set of data elements, wherein the action is preventing transmission of the object or locking down a storage repository.
19. The method of claim 18 , wherein the registered token occurs with less frequency across the plurality of tuples in the registration list than frequencies at which other registered tokens of the tuple occur across the plurality of tuples in the registration list.
20. The method of claim 18 , wherein the index table includes a plurality of indexes, each index including a unique token key and an offset to a location in the registration list.