System and method for data mining and security policy management
View Patent ↗A system and method to generate and maintain controlled growth DAG are described. The controlled growth DAG conveys information about objects captured by a capture system.
1. A method, comprising:
capturing, by an electronic computing device, one or more registered documents;
classifying the one or more registered documents with one or more classifications using one or more of the six mechanisms: (1) content signature, (2) grammar analysis, (3) statistical analysis, (4) file classification, (5) document biometrics, and (6) concept maps;
capturing, by electronic computing device, an object traversing a network environment;
classifying the object with one or more classifications using the one or more of the six mechanisms;
generating a plurality of signatures of the object;
comparing the plurality of signatures with previously stored signatures in a signatures database, wherein parts of the signatures database is excluded from said comparing step based on the one or more classifications of the object and/or the one or more classifications of the registered documents; and
identifying that at least a portion of the plurality of signatures match a portion of the previously stored signatures;
prohibiting delivery of the object;
identifying one of the registered documents associated with the portion of the previously stored signatures; and
sending an alert to respective owner of the registered documents about the object.
2. The method of claim 1 , wherein the portion of the plurality of signatures corresponds to a first content in the object, wherein the portion of the previously stored signatures corresponds to a second content in the registered documents, and wherein if the first content is the same as the second content, then a match is declared between the portion of the plurality of signatures and the portion of the previously stored signatures.
3. The method of claim 1 , wherein each of the previously stored signatures includes hashes over at least one selection from a group consisting of embedded content, indices, headers, footers, formatting information, and font utilization of the associated registered document.
4. The method of claim 1 , wherein the alert includes a request for permission from the respective owner to deliver the object to its intended destination.
5. The method of claim 1 , wherein the generating the plurality of signatures, comprises:
extracting and decoding a content in the object;
normalizing a text included in the content, wherein the normalizing comprises:
comparing the text to a plurality of patterns according to a pattern priority that is generated from a hierarchy of restrictiveness, with a more restrictive pattern having a higher priority than a less restrictive pattern; and
removing delimiters in the text according to matching patterns in the plurality of patterns to create a plurality of delimited items;
tokenizing the plurality of delimited items into a list of tokens; and
calculating hashes of the tokens.
6. The method of claim 5 , wherein the tokenizing includes converting the delimited items into corresponding tokens of fixed size according to a hash function.
7. The method of claim 5 , wherein the calculating hashes of the tokens comprises:
selecting M tokens from the list of tokens, wherein M is a positive integer;
selecting N tokens from the M tokens, wherein N is a positive integer that is less than or equal to M;
calculating a hash of the N tokens;
skipping P tokens of the list of tokens, wherein P is a non-zero integer that can be less than M, equal to M, or greater than M;
determining whether a number of remaining tokens to be hashed in the list of tokens is less than M;
if the number of remaining tokens is less than M, stopping the calculating; and
if the number of remaining tokens is greater than or equal to M, continuing the calculating.
8. The method of claim 1 , further comprising:
storing a plurality of objects in a storage medium;
creating a keyword index and a metadata index having respective entries corresponding to a content in the plurality of objects; and
querying at least one of the keyword index and the metadata index to find a specific object.
9. A non-transitory machine-readable storage medium including program code which, when executed by a processor, causes the processor to perform a method, the method comprising:
capturing, by an electronic computing device, one or more registered documents;
classifying the one or more registered documents with one or more classifications using one or more of the six mechanisms: (1) content signature, (2) grammar analysis, (3) statistical analysis, (4) file classification, (5) document biometrics, and (6) concept maps;
capturing, by the electronic computing device, an object traversing a network environment;
classifying the object with one or more classifications using the one or more of the six mechanisms;
generating a plurality of signatures of the object;
comparing the plurality of signatures with previously stored signatures in a signatures database, wherein parts of the signatures database is excluded from said comparing step based on the one or more classifications of the object and/or the one or more classifications of the registered documents; and
identifying that at least a portion of the plurality of signatures match a portion of the previously stored signatures;
prohibiting delivery of the object;
identifying one of the registered documents associated with the portion of the previously stored signatures; and
sending an alert to respective owner of the registered documents about the object.
10. The non-transitory machine-readable storage medium of claim 9 , wherein the portion of the plurality of signatures corresponds to a first content in the object, wherein the portion of the previously stored signatures corresponds to a second content in the registered documents, and wherein if the first content is the same as the second content, then a match is declared between the portion of the plurality of signatures and the portion of the previously stored signatures.
11. The non-transitory machine-readable storage medium of claim 9 , wherein the alert includes a request for permission from the respective owner to deliver the object to its intended destination.
12. The non-transitory machine-readable storage medium of claim 9 , wherein the generating the plurality of signatures comprises:
extracting and decoding a content in the object;
normalizing a text included in the content, wherein the normalizing comprises:
comparing the text to a plurality of patterns according to a pattern priority that is generated from a hierarchy of restrictiveness, with a more restrictive pattern having a higher priority than a less restrictive pattern; and
removing delimiters in the text according to matching patterns in the plurality of patterns to create a plurality of delimited items;
tokenizing the plurality of delimited items into a list of tokens; and
calculating hashes of the tokens.
13. The non-transitory machine-readable storage medium of claim 12 , wherein the calculating hashes of the tokens comprises:
selecting M tokens from the list of tokens, wherein M is a positive integer;
selecting N tokens from the M tokens, wherein N is a positive integer that is less than or equal to M;
calculating a hash of the N tokens;
skipping P tokens of the list of tokens, wherein P is a non-zero integer that can be less than M, equal to M, or greater than M;
determining whether a number of remaining tokens to be hashed in the list of tokens is less than M;
if the number of remaining tokens is less than M, stopping the calculating; and
if the number of remaining tokens is greater than or equal to M, continuing the calculating.
14. The non-transitory machine-readable storage medium of claim 9 , wherein the method further comprises:
storing a plurality of objects in a storage medium;
creating a keyword index and a metadata index having respective entries corresponding to a content in the plurality of objects; and
querying at least one of the keyword index and the metadata index to find a specific object.
15. An apparatus, comprising:
a memory for storing data; and
a processor operable to execute instructions associated with the data, wherein the apparatus is configured for:
capturing, by an electronic computing device, one or more registered documents;
classifying the one or more registered documents with one or more classifications using one or more of the six mechanisms: (1) content signature, (2) grammar analysis, (3) statistical analysis, (4) file classification, (5) document biometrics, and (6) concept maps;
capturing, by the electronic computing device, an object traversing a network environment;
classifying the object with one or more classifications using the one or more of the six mechanisms;
generating a plurality of signatures of the object;
comparing the plurality of signatures with previously stored signatures in a signatures database, wherein parts of the signatures database is excluded from said comparing step based on the one or more classifications of the object and/or the one or more classifications of the registered documents; and
identifying that at least a portion of the plurality of signatures match a portion of the previously stored signatures;
prohibiting delivery of the object;
identifying one of the registered documents associated with the portion of the previously stored signatures; and
sending an alert to respective owner of the registered documents about the object.
16. The apparatus of claim 15 , wherein the portion of the plurality of signatures corresponds to a first content in the object, wherein the portion of the previously stored signatures corresponds to a second content in the registered documents, and wherein if the first content is the same as the second content, then a match is declared between the portion of the plurality of signatures and the portion of the previously stored signatures.
17. The apparatus of claim 15 , wherein the alert includes a request for permission from the respective owner to deliver the object to its intended destination.
18. The apparatus of claim 15 , wherein the generating the plurality of signatures comprises:
extracting and decoding a content in the object;
normalizing a text included in the content, wherein the normalizing comprises:
comparing the text to a plurality of patterns according to a pattern priority that is generated from a hierarchy of restrictiveness, with a more restrictive pattern having a higher priority than a less restrictive pattern; and
removing delimiters in the text according to matching patterns in the plurality of patterns to create a plurality of delimited items;
tokenizing the plurality of delimited items into a list of tokens; and
calculating hashes of the tokens.
19. The apparatus of claim 18 , wherein the calculating hashes of the tokens comprises:
selecting M tokens from the list of tokens, wherein M is a positive integer;
selecting N tokens from the M tokens, wherein N is a positive integer that is less than or equal to M;
calculating a hash of the N tokens;
skipping P tokens of the list of tokens, wherein P is a non-zero integer that can be less than M, equal to M, or greater than M;
determining whether a number of remaining tokens to be hashed in the list of tokens is less than M;
if the number of remaining tokens is less than M, stopping the calculating; and
if the number of remaining tokens is greater than or equal to M, continuing the calculating.
20. The apparatus of claim 15 , further configured for:
storing a plurality of objects in a storage medium;
creating a keyword index and a metadata index having respective entries corresponding to a content in the plurality of objects; and
querying at least one of the keyword index and the metadata index to find a specific object.