Processor for fast contextual matching
Words having selected characteristics in a corpus of documents are found using a data processor arranged to execute queries. Memory stores an index structure in which entries in the index structure map words and marks for words having the selected characteristics to locations within documents in the corpus. Entries in the index structure represent words and other entries represent marks with the location information of a marked word. The entries for the marks can be tokens coalesced with prefixes of respective marked words or adjacent. A query processor forms a modified query by adding a mark for a word to the query. The processor executes the modified query.
1. An article of manufacture for use with a data processor for finding information in a corpus of documents, comprising:
a non-transitory machine readable data storage medium including executable instructions to:
identify selected words in the corpus of documents;
for a designated selected word, form a contextual derivation of the selected word including a contextual term associated with the selected word and a character from the selected word;
define a dictionary including stopwords, unique words and a contextual derivation associated with the corpus of documents;
form a contextual derivation query from a character of a marked word in a query and a contextual term associated with the marked word; and
apply the contextual derivation query to the dictionary to identify a match.
2. The article of manufacture of claim 1 further comprising executable instructions to locate a pointer associated with the match.
3. The article of manufacture of claim 2 further comprising executable instructions to follow the pointer to an inverted index.
4. The article of manufacture of claim 3 further comprising executable instructions to process document identification information and word location information in the inverted index.
5. The article of manufacture of claim 4 further comprising executable instructions to supply documents responsive to the query.
6. The article of manufacture of claim 5 further comprising executable instructions to supply word location information in documents responsive to the query.
7. The article of manufacture of claim 1 wherein the contextual derivation includes N leading characters of the selected word.
8. The article of manufacture of claim 7 wherein N is 3 or less.
9. The article of manufacture of claim 7 wherein N is 1.
10. The article of manufacture of claim 1 wherein a stopword is a special character.
11. The article of manufacture of claim 1 wherein the stopwords include: “the”, “of”, “and”, “a”, “is”, and “in”.
12. The article of manufacture of claim 1 wherein the dictionary includes an abnormal word including a stopword and a character of a word adjacent to the stopword.
13. The article of manufacture of claim 1 further comprising executable instructions to form a plurality of contextual derivations for the selected word.
14. A method for finding information in a corpus of documents, the method being implemented by one or more data processors and comprising:
identifying, by at least one data processor, selected words in the corpus of documents;
for a designated selected word, forming, by at least one data processor, a contextual derivation of the selected word including a contextual term associated with the selected word and a character from the selected word;
defining, by at least one data processor, a dictionary including stopwords, unique words and a contextual derivation associated with the corpus of documents;
forming, by at least one data processor, a contextual derivation query from a character of a marked word in a query and a contextual term associated with the marked word; and
applying, by at least one data processor, the contextual derivation query to the dictionary to identify a match.
15. The method of claim 14 further comprising:
locating, by at least one data processor, a pointer associated with the match,
following, by at least one data processor, the pointer to an inverted index;
processing, by at least one data processor, document identification information and word location information in the inverted index.
16. The method of claim 15 further comprising: supplying, by at least one data processor, documents responsive to the query.
17. The method of claim 16 further comprising: supplying, by at least one data processor, word location information in documents responsive to the query.
18. The method of manufacture of claim 14 , wherein the contextual derivation includes N leading characters of the selected word.
19. The method of manufacture of claim 14 wherein a stopword is a special character.
20. The method of manufacture of claim 14 further comprising forming, by at least one data processor, a plurality of contextual derivations for the selected word.
21. An apparatus for finding information in a corpus of documents comprising:
at least one data processor; and
memory coupled to the at least one data processor, the memory storing instructions, which when executed, cause the at least one data processor to:
identify selected words in the corpus of documents;
for a designated selected word, form a contextual derivation of the selected word including a contextual term associated with the selected word and a character from the selected word;
define a dictionary including stopwords, unique words and a contextual derivation associated with the corpus of documents;
form a contextual derivation query from a character of a marked word in a query and a contextual term associated with the marked word; and
apply the contextual derivation query to the dictionary to identify a match.