Locating meaningful stopwords or stop-phrases in keyword-based retrieval systems
A stopword detection component detects stopwords (also stop-phrases) in search queries input to keyword-based information retrieval systems. Potential stopwords are initially identified by comparing the terms in the search query to a list of known stopwords. Context data is then retrieved based on the search query and the identified stopwords. In one implementation, the context data includes documents retrieved from a document index. In another implementation, the context data includes categories relevant to the search query. Sets of retrieved context data are compared to one another to determine if they are substantially similar. If the sets of context data are substantially similar, this fact may be used to infer that the removal of the potential stopword(s) is not material to the search. If the sets of context data are not substantially similar, the potential stopword can be considered material to the search and should not be removed from the query.
1. A method implemented by one or more processors, comprising:
receiving a query that includes a plurality of terms;
identifying a potential stopword in the query;
determining that a category is relevant to the query based on comparing the query with the potential stopword to category descriptive information of the category, the category being included in a stored taxonomy of categories that are each associated with corresponding category descriptive information;
determining, based on the category determined to be relevant to the query, whether the potential stopword is an actual stopword;
in response to determining that the potential stopword is not an actual stopword, classifying the potential stopword as not an actual stopword;
in response to receiving the query and in response to classifying the potential stopword as not an actual stopword, issuing a search that is based on the query and that includes the potential stopword; and
presenting at least one search result that is identified in response to issuing the search that includes the potential stopword.
2. The method of claim 1 , further comprising:
determining that the category is not relevant to a refined version of the query that omits the potential stopword;
wherein classifying the potential stopword as not an actual stopword is further based on determining that the category is not relevant to the refined version of the query that omits the potential stopword.
3. The method of claim 1 , further comprising:
receiving a further query that includes a plurality of further query terms, including the potential stopword;
identifying the potential stopword in the query;
determining that a further category is relevant to the query based on comparing the further query with the potential stopword to further category descriptive information of the further category, the further category being included in the stored taxonomy of categories;
classifying the potential stopword as an actual stopword based on determining that the further category is relevant to the query;
in response to receiving the further query and in response to classifying the potential stopword as an actual stopword, issuing a further search that is based on the further query and that omits the potential stopword; and
presenting at least one further search result that is identified in response to issuing the further search that omits the potential stopword.
4. The method of claim 1 , wherein issuing the search comprises submitting the search to an Internet search engine.
5. The method of claim 1 , further comprising:
determining a degree of relevance of the category to the query;
wherein classifying the potential stopword as not an actual stopword is further based on the degree of relevance of the category to the query.
6. A system comprising:
a database storing a taxonomy of categories that are each associated with corresponding category descriptive information; and
one or more devices to:
receive a query that includes a plurality of terms;
identify a potential stopword in the query;
determine that a category, of the taxonomy of categories of the database, is relevant to the query based on comparing the query with the potential stopword to the category descriptive information associated with the category in the database;
determine, based on the category determined to be relevant to the query, whether the potential stopword is an actual stopword;
in response to determining that the potential stopword is not an actual stopword, classify the potential stopword as not an actual stopword;
in response to receiving the query and in response to classifying the potential stopword as not an actual stopword, issue a search that is based on the query and that includes the potential stopword; and
present at least one search result that is identified in response to issuing the search that includes the potential stopword.
7. The system of claim 6 , wherein the one or more devices are further to:
determine that the category is not relevant to a refined version of the query that omits the potential stopword;
wherein in classifying the potential stopword as not an actual stopword, the one or more devices are to classify the potential stopword as not an actual stopword further based on determining that the category is not relevant to the refined version of the query that omits the potential stopword.
8. The system of claim 6 , wherein the one or more devices are further to:
receive a further query that includes a plurality of further query terms, including the potential stopword;
identify the potential stopword in the query;
determine that a further category, of the taxonomy of categories of the database, is relevant to the query based on comparing the further query with the potential stopword to the category descriptive information associated with the further category in the database;
classify the potential stopword as an actual stopword based on determining that the further category is relevant to the query;
in response to receiving the further query and in response to classifying the potential stopword as an actual stopword, issue a further search that is based on the further query and that omits the potential stopword; and
present at least one further search result that is identified in response to issuing the further search that omits the potential stopword.
9. The system of claim 6 , wherein in issuing the search the one or more devices are to submit the search to an Internet search engine.
10. The system of claim 6 , wherein the one or more devices are further to:
determine a degree of relevance of the category to the query;
wherein in classifying the potential stopword as not an actual stopword the one or more devices are to classify the potential stopword as not an actual stopword further based on the degree of relevance of the category to the query.
11. At least one non-transitory computer readable medium storing instructions which, when executed by at least one processor, cause the at least one processor to perform a method comprising:
receiving a query that includes a plurality of terms;
identifying a potential stopword in the query;
determining that a category is relevant to the query based on comparing the query with the potential stopword to category descriptive information of the category, the category being included in a stored taxonomy of categories that are each associated with corresponding category descriptive information;
determining, based on the category determined to be relevant to the query, whether the potential stopword is an actual stopword;
in response to determining that the potential stopword is not an actual stopword, classifying the potential stopword as not an actual stopword;
in response to receiving the query and in response to classifying the potential stopword as not an actual stopword, issuing a search that is based on the query and that includes the potential stopword; and
presenting at least one search result that is identified in response to issuing the search that includes the potential stopword.
12. The non-transitory computer readable medium of claim 11 , wherein the method further comprises:
determining that the category is not relevant to a refined version of the query that omits the potential stopword;
wherein classifying the potential stopword as not an actual stopword is further based on determining that the category is not relevant to the refined version of the query that omits the potential stopword.
13. The non-transitory computer readable medium of claim 11 , wherein the method further comprises:
receiving a further query that includes a plurality of further query terms, including the potential stopword;
identifying the potential stopword in the query;
determining that a further category is relevant to the query based on comparing the further query with the potential stopword to further category descriptive information of the further category, the further category being included in the stored taxonomy of categories;
classifying the potential stopword as an actual stopword based on determining that the further category is relevant to the query;
in response to receiving the further query and in response to classifying the potential stopword as an actual stopword, issuing a further search that is based on the further query and that omits the potential stopword; and
presenting at least one further search result that is identified in response to issuing the further search that omits the potential stopword.
14. The non-transitory computer readable medium of claim 11 , wherein issuing the search comprises submitting the search to an Internet search engine.
15. The non-transitory computer readable medium of claim 11 , wherein the method further comprises:
determining a degree of relevance of the category to the query;
wherein classifying the potential stopword as not an actual stopword is further based on the degree of relevance of the category to the query.