System and method for word indexing in a capture system and querying thereof
View Patent ↗Searching of objects captured by a capture system can be improved by eliminating irrelevant objects from a query. In one embodiment, the present invention includes receiving such a query for objects captured by a capture system, the query including at least one search term. This search term is then hashed to a term bit position using a hash function. Then objects can be eliminated if, in a word index associated with the object, the term bit position is not set.
1. A method to be performed by a processor in an electronic environment, comprising:
receiving a query for an object captured by a capture system, the query including at least one search term, wherein a content type of the object is determined and the content type is positioned in a content field of a tag associated with the object;
hashing the at least one search term to a term bit position using a hash function; and
eliminating at least one other object from the query using a word index associated with the at least one other object, the word index comprising a plurality of bits, if a bit in the term bit position is not set, wherein eliminating at least one other object from the query comprises performing a search over a tag database containing tags associated with the captured objects.
2. The method of claim 1 , further comprising retrieving at least one object, wherein a second word index associated with the retrieved object included a set bit in the term bit position.
3. The method of claim 1 , wherein each tag in the tag database comprises an index field, the index field including a plurality of bits, wherein at least one of the bits is set.
4. The method of claim 3 , wherein the at least one set bit has a value of “1”.
5. A method to be performed by a processor in an electronic environment, the method comprising:
capturing an object being transmitted over a network, wherein a content type of the object is determined and the content type is positioned in a content field of a tag associated with the object;
extracting text contained in the captured object;
generating a plurality of tokens from the extracted text;
hashing the plurality of tokens to a plurality of bit position values; and
creating a word index to be associated with the captured object by setting bits of a bit vector corresponding to the plurality of bit position values, wherein generating the plurality of tokens comprises tokenizing the extracted text using a context-aware parser, and wherein the context-aware parser uses a list of patterns associated with a content type of the captured object to sub-tokenize at least one of the tokens generated from the extracted text.
6. The method of claim 5 , wherein the context-aware parser uses a list of separators associated with a content type of the captured object.
7. The method of claim 5 , further comprising inserting the word index into a tag associated with the captured object.
8. The method of claim 7 , further comprising storing the captured object in a canonical object store and storing the tag associated with the captured object in a tag database.
9. The method of claim 5 , further comprising:
receiving a query including a search term;
hashing the search term to a term bit position of the bit vector; and
eliminating the captured object from the query by observing that a bit in the term bit position is not set in the word index contained in the tag associated with the captured object.
10. A capture system comprising:
one or more capture modules including a network interface card to capture an object being transmitted over a network, wherein a content type of the object is determined and the content type is positioned in a content field of a tag associated with the object;
a text extractor to extract text contained in the captured object;
a tokenizer to generate a plurality of tokens from the extracted text;
a sub-tokenizer to sub-tokenize the plurality of tokens comprises using a context-aware parser; and
an index generator to create a word index by hashing the plurality of tokens to a plurality of bit position values and setting bits of a bit vector corresponding to the plurality of bit position values, wherein the context-aware parser uses a list of patterns associated with a content type of the captured object to sub-tokenize at least one of the tokens generated from the extracted text.
11. The capture system of claim 10 , wherein the context-aware parser uses a list of separators associated with a content type of the captured object.
12. The capture system of claim 10 , wherein the index generator also inserts the word index into a tag associated with the captured object.
13. The capture system of claim 12 , further comprising an object store to store the captured object and a tag database to store the tag associated with the captured object.
14. A machine-readable medium having stored thereon data representing instructions that, when executed by a processor, cause the processor to perform operations comprising:
receiving a query for an object captured by a capture system, the query including at least one search term, wherein a content type of the object is determined and the content type is positioned in a content field of a tag associated with the object;
hashing the at least one search term to a term bit position using a hash function; and
eliminating at least one other object from the query using a word index associated with the at least one other object, the word index comprising a plurality of bits, if a bit in the term bit position is not set, wherein eliminating the at least one other object from the query comprises performing a search over a tag database containing tags associated with the capture objects.
15. The machine-readable medium of claim 14 , wherein the instructions further cause the processor to retrieve at least one object, wherein a second word index associated with the retrieved object included a set bit in the term bit position.
16. The machine-readable medium of claim 14 , wherein each tag in the tag database comprises an index field, the index field including a plurality of bits, wherein at least one of the bits is set.
17. The machine-readable medium of claim 16 , wherein the at least one set bit has a value of “1”.
18. A machine-readable medium having stored thereon data representing instructions that, when executed by a processor of a capture system, cause the processor to perform operations comprising:
extracting text contained in a captured object, wherein a content type of the object is determined and the content type is positioned in a content field of a tag associated with the object;
generating a plurality of tokens from the extracted text;
hashing the plurality of tokens to a plurality of bit position values; and
creating a word index to be associated with the captured object by setting bits of a bit vector corresponding to the plurality of bit position values, wherein generating the plurality of tokens comprises tokenizing the extracted text using a context-aware parser, and wherein the context-aware parser uses a list of patterns associated with a content type of the captured object to sub-tokenize at least one of the tokens generated from the extracted text.
19. The machine-readable medium of claim 18 , wherein the context-aware parser uses a list of separators associated with a content type of the captured object.
20. The machine-readable medium of claim 18 , wherein the instructions further cause the processor to insert the word index into a tag associated with the captured object.
21. The machine-readable medium of claim 20 , wherein the instructions further cause the processor to store the captured object in a canonical object store and storing the tag associated with the captured object in a tag database.
22. The machine-readable medium of claim 18 , wherein the instructions further cause the processor perform operations comprising:
receiving a query including a search term;
hashing the search term to a term bit position of the bit vector; and
eliminating the captured object from the query by observing that a bit in the term bit position is not set in the word index contained in the tag associated with the captured object.