Processor for fast phrase searching
Phrases in a corpus of documents including stopwords are found using a data processor arranged to execute phrase queries. Memory stores an index structure which maps entries in the index structure to documents in the corpus. Entries in the index structure represent words and other entries represent stopwords found in the corpus coalesced with prefixes of respective adjacent words adjacent to the stopwords. The prefixes comprise one or more leading characters of the respective adjacent words. A query processor forms a modified query by substituting a stopword with a search token representing the stopword coalesced with a prefix of the next word in the query. The processor executes the modified query. Also, index structures including coalesced stopwords are created and maintained.
1. An article of manufacture for use with a data processor for finding information in a corpus of documents, wherein words in the corpus of documents include a set of stopwords, comprising:
a machine readable data storage medium including executable instructions to:
identify stopwords in the corpus of documents;
form abnormal words, each abnormal word including a stop word and a character of a word adjacent to the stopword, such that each abnormal word includes a complete word and an incomplete word;
define a dictionary including stopwords, unique words and abnormal words associated with the corpus of documents;
form an abnormal query word with a stopword and a character of a word adjacent to the stopword in a query;
apply the abnormal query word to the dictionary to identify a match.
2. The article of manufacture of claim 1 further comprising executable instructions to locate a pointer associated with the match.
3. The article of manufacture of claim 2 further comprising executable instructions to follow the pointer to an inverted index.
4. The article of manufacture of claim 3 further comprising executable instructions to process document identification information and word location information in the inverted index.
5. The article of manufacture of claim 4 further comprising executable instructions to supply documents responsive to the query.
6. The article of manufacture of claim 5 further comprising executable instructions to supply word location information in documents responsive to the query.
7. The article of manufacture of claim 1 wherein an abnormal word includes a stop word and N leading characters of a subsequent word.
8. The article of manufacture of claim 7 wherein N is 3 or less.
9. The article of manufacture of claim 7 wherein N is 1.
10. The article of manufacture of claim 1 wherein a stopword is a special character.
11. The article of manufacture of claim 1 wherein the stopwords include: “the”, “of”, “and”, “a”, “is”, and “in”.