Query phrasification
An information retrieval system uses phrases to index, retrieve, organize and describe documents. Phrases are extracted from the document collection. Documents are the indexed according to their included phrases, using phrase posting lists. The phrase posting lists are stored in an cluster of index servers. The phrase posting lists can be tiered into groups, and sharded into partitions. Phrases in a query are identified based on possible phrasifications. A query schedule based on the phrases is created from the phrases, and then optimized to reduce query processing and communication costs. The execution of the query schedule is managed to further reduce or eliminate query processing operations at various ones of the index servers.
1. A method comprising:
decomposing a search query that includes three or more words into a plurality of candidate phrasifications, each candidate phrasification having a different grouping of component phrases, a component phrase including a non-zero quantity of the words and each candidate phrasification including all of the words of the search query;
scoring one or more candidate phrasifications, a candidate phrasification being scored by applying a scoring model, the scoring model based on a number of component phrases in the phrasification, a probability of occurrence of each of the component phrases in the candidate phrasification from a valid phrase table, and parameters for adjusting precision and recall of the candidate phrasification, the parameters including a first parameter to adjust precision of candidate phrasifications and a second parameter to adjust bias against obtaining too many phrases;
selecting at least one highest scoring candidate phrasification; and
executing the at least one highest scoring candidate phrasification against an index that includes posting lists for phrases, the executing identifying documents associated with each of the component phrases of the highest scoring candidate phrasification.
2. The method of claim 1 , further comprising generating, prior to scoring candidate phrasifications, at least one additional candidate phrasification by replacing at least one of the component phrases in a grouping with a related phrase.
3. The method of claim 2 , further comprising assigning a confidence level to component phrases of a candidate phrasification, wherein component phrases not present in the search query are assigned a lesser confidence level than component phrases present in the search query.
4. The method of claim 2 , wherein the related phrase includes a phrase synonymous with one of the component phrases.
5. The method of claim 2 , wherein the related phrase includes a phrase chosen for semantic similarity to at least one of the component phrases.
6. The method of claim 2 , wherein the related phrase includes a spell-corrected version of at least one of the component phrases.
7. The method of claim 1 , wherein the probability of occurrence of the component phrases of a candidate phrasification is based on an estimated probability figure.
8. The method of claim 1 , wherein the probability of occurrence of the component phrases of a candidate phrasification is based on a proportion of documents to be searched that contain the component phrase.
9. The method of claim 1 , wherein smaller values for the second parameter reduces recall.
10. The method of claim 1 , wherein the probability of occurrence of each component phrase in a candidate phrasification is based on the incidence of the component phrase in an indexed document collection.
11. The method of claim 1 , wherein the probability of occurrence of each component phrase in a candidate phrasification is based on natural language.
12. The method of claim 1 , wherein a higher value for the first parameter increases precision and reduces recall.