Click or skip evaluation of reordering rules
View Patent ↗Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for evaluating a query term reordering rule. In one aspect, a method includes accessing query log data that identifies, for an initial search query, two or more query terms that are included in a particular order in the initial search query, search results that were generated using the initial search query, and a particular search result selected by the user. A determination is made, using the query log data, that the particular search result includes the two or more query terms in a different order than the particular order in which the two or more query terms are ordered in the initial search query. In response to this determination, a click count is incremented for a query term reordering rule associated with the two or more query terms.
1. A computer-implemented method comprising:
accessing query log data that identifies, for an initial search query that includes a sequence of query terms:
two or more particular query terms that are included in a particular order in the sequence of query terms of the initial search query,
search results that were generated using the initial search query, and
a particular search result selected by the user;
determining, using the query log data, that the particular search result includes the two or more particular query terms in a different order than the particular order in which the two or more particular query terms are ordered in the sequence of query terms of the initial search query;
in response to determining that the particular search result includes the two or more particular query terms in the different order, adjusting a click count for a query term reordering rule associated with the two or more particular query terms; and
determining, based at least on the adjusted click count, whether to revise a search query using the query term reordering rule.
2. The method of claim 1 , wherein adjusting the click count for the query term reordering rule comprises incrementing the click count for the query term reordering rule, and wherein the method comprises assigning a score to the query term reordering rule based on the click count and a skip count.
3. The method of claim 2 , wherein the score assigned to the query term reordering rule is based on a ratio of (i) the click count to (ii) the click count and the skip count.
4. The method of claim 2 , comprising:
determining that the score assigned to the query term reordering rule does not meet a threshold; and
removing the query term reordering rule from a set of query term reordering rules used to generate revised search queries, based on determining that the score assigned to the query term reordering rule does not meet the threshold.
5. The method of claim 2 , comprising associating a first weight with the click count and a second weight with the skip count.
6. The method of claim 5 , wherein the score assigned to the query term reordering rule satisfies:
W
1
(
click
count
)
W
1
(
click
count
)
+
W
2
(
skip
count
)
wherein W1 represents the first weight associated with the click count and W2 represents the second weight associated with the skip count.
7. The method of claim 1 , wherein the query term reordering rule comprises a query term revision rule for revising search queries by reordering the two or more particular query terms from the particular order in which the two or more particular query terms are ordered in the sequence of query terms of the initial search query to the different order.
8. The method of claim 1 , wherein the query term reordering rule comprises a query term scoring rule for scoring occurrences of the two or more particular query terms that occur in the different order in search results that were generated using search queries.
9. The method of claim 1 , wherein the different order is a reverse of the particular order in which the two or more particular query terms are ordered in the sequence of query terms of the initial search query.
10. A computer-implemented method comprising:
accessing query log data that identifies, for an initial search query that includes a sequence of query terms:
two or more particular query terms that are included in a particular order in the sequence of query terms of the initial search query,
search results that were generated using the initial search query,
a ranking of the search results, and
a particular search result selected by the user;
determining, using the query log data, that a search result that is ranked above the particular search result includes the two or more particular query terms in a different order than the particular order in which the two or more particular query terms are ordered in the sequence of query terms of the initial search query;
in response to determining that the search result that is ranked above the particular search result includes the two or more particular query terms in the different order, adjusting a skip count for a query term reordering rule associated with the two or more particular query terms; and
determining, based at least on the adjusted click count, whether to revise a search query using the query term reordering rule.
11. The method of claim 10 , wherein adjusting the skip count for the query term reordering rule comprises incrementing the skip count for the query term reordering rule, and wherein the method comprises assigning a score to the query term reordering rule based on a click count and the skip count.
12. The method of claim 11 , wherein the score assigned to the query term reordering rule is based on a ratio of (i) the click count to (ii) the click count and the skip count.
13. The method of claim 11 , comprising:
determining that the score assigned to the query term reordering rule does not meet a threshold; and
removing the query term reordering rule from a set of query term reordering rules used to generate revised search queries, based on determining that the score assigned to the query term reordering rule does not meet the threshold.
14. The method of claim 11 , comprising associating a first weight with the click count and a second weight with the skip count.
15. The method of claim 14 , wherein the score assigned to the query term reordering rule satisfies:
W
1
(
click
count
)
W
1
(
click
count
)
+
W
2
(
skip
count
)
wherein W1 represents the first weight associated with the click count and W2 represents the second weight associated with the skip count.
16. The method of claim 10 , wherein the query term reordering rule comprises a query term revision rule for revising search queries by reordering the two or more particular query terms from the particular order in which the two or more particular query terms are ordered in the sequence of query terms of the initial search query to the different order.
17. The method of claim 10 , wherein the query term reordering rule comprises a query term scoring rule for scoring occurrences of the two or more particular query terms that occur in the different order in search results that were generated using search queries.
18. The method of claim 10 , wherein the different order is a reverse of the particular order in which the two or more particular query terms are ordered in the sequence of query terms of the initial search query.
19. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
accessing query log data that identifies, for an initial search query that includes a sequence of query terms:
two or more particular query terms that are included in a particular order in the sequence of query terms of the initial search query,
search results that were generated using the initial search query, and
a particular search result selected by the user;
determining, using the query log data, that the particular search result includes the two or more particular query terms in a different order than the particular order in which the two or more particular query terms are ordered in the sequence of query terms of the initial search query;
in response to determining that the particular search result includes the two or more particular query terms in the different order, adjusting a click count for a query term reordering rule associated with the two or more particular query terms; and
determining, based at least on the adjusted click count, whether to revise a search query using the query term reordering rule.
20. The system of claim 19 , wherein adjusting the click count for the query term reordering rule comprises incrementing the click count for the query term reordering rule, and wherein the operations comprise assigning a score to the query term reordering rule based on the click count and a skip count.
21. The system of claim 20 , wherein the score assigned to the query term reordering rule is based on a ratio of (i) the click count to (ii) the click count and the skip count.
22. The system of claim 20 , the operations comprising:
determining that the score assigned to the query term reordering rule does not meet a threshold; and
removing the query term reordering rule from a set of query term reordering rules used to generate revised search queries, based on determining that the score assigned to the query term reordering rule does not meet the threshold.
23. The system of claim 20 , comprising associating a first weight with the click count and a second weight with the skip count.
24. The system of claim 19 , wherein the query term reordering rule comprises a query term revision rule for revising search queries by reordering the two or more particular query terms from the particular order in which the two or more particular query terms are ordered in the sequence of query terms of the initial search query to the different order.
25. The system of claim 19 , wherein the query term reordering rule comprises a query term scoring rule for scoring occurrences of the two or more particular query terms that occur in the different order in search results that were generated using search queries.