Modifying search result ranking based on implicit user feedback
The present disclosure includes systems and techniques relating to ranking search results of a search query. In general, the subject matter described in this specification can be embodied in a computer-implemented method that includes determining a measure of relevance for a document result within a context of a search query for which the document result is returned, the determining being based on a first number in relation to a second number, the first number corresponding to longer views of the document result, and the second number corresponding to at least shorter views of the document result; and outputting the measure of relevance to a ranking engine for ranking of search results, including the document result, for a new search corresponding to the search query. The subject matter described in this specification can also be embodied in various corresponding computer program products, apparatus and systems.
1. A system comprising:
one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
maintaining, in a result selection log, data regarding users' interactions with search results from an Internet search engine for multiple users, each log entry in the result selection log for an actual interaction being specific to one interaction and including data identifying a respective user, a query submitted by the user, one or more search results presented by the search engine in response to the query, a document selected by the user from among the search results, an ordinal position in a presentation order of the search results of the search result selected by the user, a time the user spent on the document, a language employed by the user, and a country where the user is likely located,
wherein the log entries include entries identifying multiple users, multiple documents, multiple languages, and multiple countries;
determining from the log entries in the result selection log (i) weighted-click fractions for each of multiple query-document pairs, (ii) weighted-click fractions for each of multiple query-document-language pairs, and (iii) weighted-click fractions for each of multiple query-document-language-country pairs, wherein each weighted-click fraction is based on a sum of a weighted number of documents selected by the user, and each weight is based on how much time the user spent on the document; and
modifying an information retrieval score in the Internet search engine for a specific document by applying one of the weighted-click fractions or a transform of the one of the weighted-click fractions to the information retrieval score for the specific document.
2. The system of claim 1 , wherein
the time the user spent on the document is measured as how much time passed between an initial click through to the document result until the user comes back to the search results presented by the search engine and selects another document result.
3. The system of claim 1 , wherein the log data also includes for each of multiple presentations of search results by the search engine:
whether a document result was presented to the respective user but was not selected, respective positions of one or more selections in a search results presentation user interface, information retrieval scores of selected documents, information retrieval scores of all documents shown before a selected document, and titles and snippets shown to a user before the user selected a document.
4. The system of claim 1 , wherein the operations further comprise:
assigning lower weights to click fractions based on users who almost always select the highest ranked result lower weights than click fractions based on users who more often select results lower in the ranking first in the weighted-click fractions.
5. The system of claim 1 , wherein the operations further comprise:
classifying individual selections of document results into two or more viewing time categories, and assigning weights to the individual selections based on the classifying, the viewing time categories including a category for a short click and a category for a long click.
6. The system of claim 5 , wherein the viewing time categories further comprise a category for a medium click and a category for a last click.
7. The system of claim 6 , wherein the operations further comprise:
determining respective time frames to classify selections as short, medium and long clicks, based on a viewing length differentiator, the viewing length differentiator including a factor governed by a determined category of the search query, a factor governed by a determined type of a user generating the individual selections, or a combination of them.
8. A method comprising:
maintaining, in a result selection log, data regarding users' interactions with search results from an Internet search engine for multiple users, each log entry in the result selection log for an actual interaction being specific to one interaction and including data identifying a respective user, a query submitted by the user, one or more search results presented by the search engine in response to the query, a document selected by the user from among the search results, an ordinal position in a presentation order of the search results of the search result selected by the user, a time the user spent on the document, a language employed by the user, and a country where the user is likely located,
wherein the log entries include entries identifying multiple users, multiple documents, multiple languages, and multiple countries;
determining from the log entries in the result selection log (i) weighted-click fractions for each of multiple query-document pairs, (ii) weighted-click fractions for each of multiple query-document-language pairs, and (iii) weighted-click fractions for each of multiple query-document-language-country pairs, wherein each weighted-click fraction is based on a sum of a weighted number of documents selected by the user, and each weight is based on how much time the user spent on the document; and
modifying an information retrieval score in the Internet search engine for a specific document by applying one of the weighted-click fractions or a transform of the one of the weighted-click fractions to the information retrieval score for the specific document.
9. The method of claim 8 , wherein the log data also includes for each of multiple presentations of search results by the search engine:
whether a document result was presented to the respective user but was not selected, respective positions of one or more selections in a search results presentation user interface, information retrieval scores of selected documents, information retrieval scores of all documents shown before a selected document, and titles and snippets shown to a user before the user selected a document.
10. The method of claim 8 , further comprising:
assigning lower weights to click fractions based on users who almost always select the highest ranked result lower weights than click fractions based on users who more often select results lower in the ranking first in the weighted-click fractions.
11. The method of claim 8 , further comprising:
classifying individual selections of document results into two or more viewing time categories, and assigning weights to the individual selections based on the classifying, the viewing time categories including a category for a short click, a category for a long click, and a category for a medium click.
12. The method of claim 11 , further comprising:
determining respective time frames to classify selections as short, medium and long clicks, based on a viewing length differentiator, the viewing length differentiator including a factor governed by a determined category of the search query, a factor governed by a determined type of a user generating the individual selections, or a combination of them.
13. One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
maintaining, in a result selection log, data regarding users' interactions with search results from an Internet search engine for multiple users, each log entry in the result selection log for an actual interaction being specific to one interaction and including data identifying a respective user, a query submitted by the user, one or more search results presented by the search engine in response to the query, a document selected by the user from among the search results, an ordinal position in a presentation order of the search results of the search result selected by the user, a time the user spent on the document, a language employed by the user, and a country where the user is likely located,
wherein the log entries include entries identifying multiple users, multiple documents, multiple languages, and multiple countries;
determining from the log entries in the result selection log (i) weighted-click fractions for each of multiple query-document pairs, (ii) weighted-click fractions for each of multiple query-document-language pairs, and (iii) weighted-click fractions for each of multiple query-document-language-country pairs, wherein each weighted-click fraction is based on a sum of a weighted number of documents selected by the user, and each weight is based on how much time the user spent on the document; and
modifying an information retrieval score in the Internet search engine for a specific document by applying one of the weighted-click fractions or a transform of the one of the weighted-click fractions to the information retrieval score for the specific document.
14. The one or more non-transitory computer-readable storage media of claim 13 , wherein
the time the user spent on the document is measured as how much time passed between an initial click through to the document result until the user comes back to the search results presented by the search engine and selects another document result.
15. The one or more non-transitory computer-readable storage media of claim 13 , wherein the log data also includes for each of multiple presentations of search results by the search engine:
whether a document result was presented to the respective user but was not selected, respective positions of one or more selections in a search results presentation user interface, information retrieval scores of selected documents, information retrieval scores of all documents shown before a selected document, and titles and snippets shown to a user before the user selected a document.
16. The one or more non-transitory computer-readable storage media of claim 13 , wherein the operations further comprise:
assigning lower weights to click fractions based on users who almost always select the highest ranked result lower weights than click fractions based on users who more often select results lower in the ranking first in the weighted-click fractions.
17. The one or more non-transitory computer-readable storage media of claim 13 , wherein the operations further comprise:
classifying individual selections of document results into two or more viewing time categories, and assigning weights to the individual selections based on the classifying, the viewing time categories including a category for a short click and a category for a long click.
18. The one or more non-transitory computer-readable storage media of claim 17 , wherein the viewing time categories further comprise a category for a medium click and a category for a last click.
19. The one or more non-transitory computer-readable storage media of claim 18 , wherein the operations further comprise:
determining respective time frames to classify selections as short, medium and long clicks, based on a viewing length differentiator, the viewing length differentiator including a factor governed by a determined category of the search query, a factor governed by a determined type of a user generating the individual selections, or a combination of them.