Generating signatures over a document
View Patent ↗A document accessible over a network can be registered. A registered document, and the content contained therein, cannot be transmitted undetected over and off of the network. In one embodiment, the invention includes maintaining a plurality of stored signatures over a registered document. In one embodiment, the plurality of stored signatures are generated by extracting content from the document, normalizing the extracted content, and generating the plurality of signatures using the normalized content.
1. A method for generating a plurality of signatures over a document, the method comprising:
extracting content from the document;
normalizing the extracted content; generating the plurality of signatures using the normalized content; and tokenizing the normalized content prior to generating the plurality of signatures;
wherein tokenizing the normalized content comprises creating an ordered list of tokens by converting each delimited item in the normalized content into a token;
wherein generating the plurality of signatures comprises:
i) selecting M consecutive tokens from the ordered list of tokens,
M being a positive integer;
ii) selecting N special tokens from the M consecutive tokens, N being a positive integer not greater than M;
iii) generating a signature by calculating a hash over the N special tokens;
iv) skipping ahead P tokens from the first of the M consecutive tokens, P being a positive integer; and
v) repeating i), ii), iii), and iv) until the plurality of signatures have been generated.
2. The method of claim 1 , wherein extracting the content from the document comprises identifying a content type of the document, and applying an appropriate text extractor/decoder to the document based on the content type.
3. The method of claim 1 , wherein extracting the content from the document further comprises decoding the content contained in the document.
4. The method of claim 1 , wherein normalizing the extracted content comprises eliminating irrelevant text from the extracted content.
5. The method of claim 1 , wherein normalizing the extracted content comprises delimiting the extracted content into items by comparing strings of the extracted content to a plurality of normalization patterns.
6. The method of claim 1 , wherein each token comprises a fixed-length hash of a delimited item.
7. The method of claim 1 , wherein generating the plurality of signatures comprises selecting a plurality of tokens from the ordered list of tokens and calculating a plurality of hashes using the selected tokens.
8. The method of claim 1 , wherein selecting N special tokens comprises selecting N special tokens based, at least in part, on token priority.
9. The method of claim 1 , wherein token priority is based on a type of delimited item used to create the token.
10. The method of claim 1 , wherein token priority is based on the frequency of each token.
11. The method of claim 1 , further comprising comparing the plurality of signatures with signatures stored in a signature database, and observing registered content contained in the document based on the comparison.
12. An apparatus comprising:
an extractor/decoder to extract content from a document;
a normalizer to normalize the extracted content;
a signature generator to generate a plurality of signatures using the normalized content; and
a tokenizer configured to create an ordered list of tokens by converting each delimited item in the normalized content into a token;
wherein the signature generator is configured to generate the plurality of signature by:
i) selecting M consecutive tokens from the ordered list of tokens, M being a positive integer;
ii) selecting N special tokens from the M consecutive tokens, N being a positive integer not greater than M;
iii) generating a signature by calculating a hash over the N special tokens;
iv) skipping ahead P tokens from the first of the M consecutive tokens, P being a positive integer; and
v) repeating i), ii), iii), and iv) until the plurality of signatures have been generated.
13. The apparatus of claim 12 , further comprising a content classifier to determine a type of content contained in the document, wherein the extractor/decoder used to extract the content from the document is selected from a plurality of extractor/decoders based on the type of content.
14. The apparatus of claim 12 , wherein the normalizer eliminates irrelevant text from the extracted content.
15. The apparatus of claim 12 , wherein each token comprises a fixed-length hash of a delimited item.
16. The apparatus of claim 12 , wherein the signature generator selects a plurality of tokens from the ordered list of tokens, and calculates a plurality of hashes using the selected tokens.
17. The apparatus of claim 12 , wherein the signature generator selects the N special tokens based, at least in part, on token priority.
18. The apparatus of claim 12 , wherein token priority is based on a type of delimited item used to create the token.
19. The apparatus of claim 12 , wherein token priority is based on the frequency of each token.
20. The apparatus of claim 12 , further comprising search engine to compare the plurality of signatures with signatures stored in a signature database, and a notification module to alert a user that registered content is contained in the document based on the comparison.
21. A tangible computer readable medium having stored thereon data representing instructions that, when executed by a processor, cause the processor to perform operations comprising:
extracting content from the document;
normalizing the extracted content; and
generating a plurality of signatures using the normalized content;
wherein the instructions further cause the processor to tokenize the normalized content prior to generating the plurality of signatures;
wherein tokenizing the normalized content comprises creating an ordered list of tokens by converting each delimited item in the normalized content into a token; and
wherein generating the plurality of signatures comprises:
i) selecting M consecutive tokens from the ordered list of tokens, M being a positive integer;
ii) selecting N special tokens from the M consecutive tokens, N being a positive integer not greater than M;
iii) generating a signature by calculating a hash over the N special tokens;
iv) skipping ahead P tokens from the first of the M consecutive tokens, P being a positive integer; and
v) repeating i), ii), iii), and iv) until the plurality of signatures have been generated.
22. The tangible computer readable medium of claim 21 , wherein extracting the content from the document comprises identifying a content type of the document, and applying an appropriate text extractor/decoder to the document based on the content type.
23. The tangible computer readable medium of 21 , wherein extracting the content from the document further comprises decoding the content contained in the document.
24. The tangible computer readable medium of 21 , wherein normalizing the extracted content comprises eliminating irrelevant text from the extracted content.
25. The tangible computer readable medium of claim 21 , wherein each token comprises a fixed-length hash of a delimited item.
26. The tangible computer readable medium of claim 21 , wherein generating the plurality of signatures comprises selecting a plurality of tokens from the ordered list of tokens and calculating a plurality of hashes using the selected tokens.
27. The tangible computer readable medium of 21 , wherein the instructions further cause the processor to compare the plurality of signatures with signatures stored in a signature database, and to observe registered content contained in the document based on the comparison.