Processor for fast contextual searching
Words having selected characteristics in a corpus of documents are found using a data processor arranged to execute queries. Memory stores an index structure in which entries in the index structure map words and marks for words having the selected characteristics to locations within documents in the corpus. Entries in the index structure represent words and other entries represent marks with the location information of a marked word. The entries for the marks can be tokens coalesced with prefixes of respective marked words or adjacent. A query processor forms a modified query by adding a mark for a word to the query. The processor executes the modified query.
1. An apparatus for contextual match in a corpus of documents, comprising:
a data processor arranged to execute queries to match words in the corpus of documents;
memory storing an index structure readable by the data processor, the index structure mapping entries in the index structure to locations of words in the documents in the corpus, the index structure including entries representing words found in the corpus of documents, and entries representing marks which identify a characteristic of corresponding marked words, and wherein one or more entries representing marks include fewer, if any, than all of the characters of the corresponding marked words;
wherein the data processor includes a query processor which modifies a subject query to form a modified query adapted to use the entries representing marks, and executes the modified query using said index structure;
wherein at least one entry representing a mark in the index structure comprises a token representing a type of mark coalesced with a prefix of a corresponding marked word, the prefix comprising one or more leading characters of the corresponding marked word.
2. The apparatus of claim 1 , wherein the prefix comprises N leading characters of the marked word, and N is 3 or less.
3. The apparatus of claim 1 , wherein the prefix comprises N leading characters of the marked word and N is 1 .
4. The apparatus of claim 1 , including an index processor which processes documents in the corpus to generate said index structure.
5. The apparatus of claim 1 , wherein the index structure comprises a dictionary and a reverse index including said entries.
6. The apparatus of claim 1 , wherein the characteristic identified by at least one mark includes a context of the corresponding marked word.
7. The apparatus of claim 1 , wherein the corpus includes stopwords, and index structure includes entries representing marks that identify the corresponding marked words as stopwords, and wherein the entries representing marks that identify the corresponding marked words as stopwords comprise tokens coalesced with prefixes of adjacent words adjacent to the corresponding marked words, the prefixes comprising one or more leading characters of the respective adjacent words.
8. A method for finding phrases in a corpus of documents using a data processor, wherein the words in the corpus of documents include a set of stopwords, comprising:
storing an index structure on a medium readable by the data processor, the index structure mapping entries in the index structure to documents in the corpus, the index structure including entries representing words found in the corpus of documents associated with locations of the words in the documents, and entries representing marks which identify a characteristic of corresponding marked words associated with locations of the marked words in the documents, and wherein one or more entries representing marks include fewer, if any, than all of the characters of the corresponding marked words;
modifying an input phrase query provided to the data processor to form a modified query by adding a mark corresponding to a word in a subject phrase; and
executing the modified query using said index structure and the data processor;
wherein at least one entry representing a mark in the index structure comprises a token representing a type of mark coalesced with a prefix of a corresponding marked word, the prefix comprising one or more leading characters of the corresponding marked word.
9. The method of claim 8 , wherein the prefix comprises N leading characters of the marked word, and N is 3 or less.
10. The method of claim 8 , wherein the prefix comprises N leading characters of the marked word, and N is 1 .
11. The method of claim 8 , including processing documents in the corpus to generate said index structure.
12. The method of claim 8 , wherein the index structure comprises a dictionary and an inverted index including said entries.
13. The method of claim 8 , wherein the characteristic identified by at least one mark includes a context of the corresponding marked word.
14. The method of claim 8 , wherein the index structure includes entries representing stopwords in the corpus including tokens coalesced with prefixes of respective adjacent words adjacent to the stopwords, the prefixes comprising one or more leading characters of the respective adjacent words.
15. An apparatus for indexing a corpus of documents, wherein the words in the corpus of documents include a set of words having a characteristic to be subject of queries, comprising:
a data processor arranged to parse documents in the corpus of documents to identify words found in the documents and locations of the words in the documents, and to create an index structure including entries representing words found in the corpus of documents mapping entries in the index structure to locations of the words in documents in the corpus
memory storing the index structure writable and readable by the data processor; wherein the data processor includes an indexing processor which indentifies words in a set of words having a characteristic represent d by a mark in a set of marks, and add entries in the index structure representing marks for the identified the set mapping the marks to the locations of the identified words, and wherein one or more entries representing marks include fewer, if any, than all of the characters of the corresponding identified words;
wherein entries in the index structure representing the marks comprise tokens coalesced with prefixes of respective marked words, the prefixes comprising one or more leading characters of the respective marked words.
16. The apparatus of claim 15 , wherein the prefix comprises N leading characters of the marked word, and N is 3 or less.
17. The apparatus of claim 15 , wherein the prefix comprises N leading characters of the marked word, and N is 1 .
18. The apparatus of claim 15 , wherein the index structure comprises a dictionary and a reverse index including said entries.
19. The apparatus of claim 15 , wherein the characteristic identified by at least one mark includes a context of the marked word.
20. The apparatus of claim 15 , wherein the indexing processor identifies stopwords in the set of words found in documents in the corpus, and adds entries in the index structure representing marks for the stopwords, the entries representing marks for the stopwords comprising tokens coalesced with prefixes of respective adjacent words adjacent to the stopwords, the prefixes comprising one or more leading characters of the respective adjacent words.
21. A method for finding phrases in a corpus of documents using a data processor, wherein the words in the corpus of document: include a set of stopwords. comprising:
parsing documents in the corpus of documents using the data processor to identify words found in the documents and the locations of the words in the documents, and adding entries representing words found in the corpus of documents to an index structure mapping entries in the index structure to documents in the corpus;
storing the index structure in memory writable and readable by the data processor; and
identifying identifies words in a set of words having a characteristic represented by a mark in a set of marks found in documents in the corpus, and adds entries in the index structure representing marks for the identified words in the set mapping the marks to the locations of the identified words, and wherein one or more entries representing marks include fewer, if any, than all of the characters of the corresponding identified words;
wherein the entries in the index structure representing marks comprise tokens coalesced with prefixes of respective marked words, the prefixes comprising one or more leading characters of the respective marked words.
22. The method of claim 21 , wherein the prefix comprises N leading characters of the marked word, and N is 3 or less.
23. The method of claim 21 , wherein the prefix comprises N leading characters of the marked word, and N is 1 .
24. The method of claim 21 , wherein the index structure comprises a dictionary and an inverted index including said entries.
25. The method of claim 21 , wherein the characteristic identified by at least one mark includes a context of the marked word.
26. The method of claim 21 , including identifying stopwords in the set of words found in documents in the corpus, and adding entries representing marks in the index structure for the stopwords the entries representing marks for the stopwords comprising tokens coalesced with prefixes of respective adjacent words adjacent to the stopwords, the prefixes comprising one or more leading characters of the respective adjacent words.
27. An article of manufacture for use with a data processor for finding phrases in a corpus of documents, wherein the words in the corpus of documents include a set of stopwords, comprising:
a machine readable data storage medium, instructions stored on the medium executable by the data processor to perform the steps of:
parsing documents in the corpus of documents using the data processor to identify words found in the documents and the locations of the words in the documents, and adding entries representing words found in the corpus of documents to an index structure mapping entries in the index structure to documents in the corpus;
storing the index structure in memory writable and readable by the data processor;
identifying words in a set of words having a characteristic represented by a mark in a set of marks found in documents in the corpus, and adding entries in the index structure representing marks for the identified words in the set mapping the marks to the locations of the identified words, and wherein one or more entries representing marks include fewer, if any, than all of the characters of the corresponding identified words;
modifying an input phrase query provided to the data processor to form a modified query by adding a mark corresponding to a word found in a subject phrase; and
executing the modified query using said index structure and the data processor;
wherein at least one entry in the index structure representing a mark in the index structure comprises a token representing a type of mark coalesced with a prefix of a corresponding marked word, the prefix comprising one or more leading characters of the corresponding marked word.