Offline evaluation of ranking functions
The claimed subject matter includes techniques for offline evaluation of ranking functions. An example system includes a first module configured to receive production log data, the first module to pre-process the production log data to generate an exploration data set. The example system also includes a second module configured to perform offline estimation of online metrics for ranking functions using the exploration data set. The example system also includes a third module to evaluate a proposed ranking function by comparing the estimated online metrics to a set of baseline metrics of a baseline ranking function and detecting that the estimated online metrics of the proposed ranking function exceed, are lower than, or are within a predetermined range of the baseline metrics.
1. A system for offline evaluation of ranking functions, comprising a processor to:
receive production log data and pre-process the production log data to generate an exploration data set, wherein the production log data comprises continuous updates of a ranking function associated with different actions for a same query issued by a same user in different impressions;
perform offline estimation of online metrics for ranking functions using the exploration data set, wherein a proposed ranking function is to be received and tested offline prior to being tested online on users;
evaluate the proposed ranking function by comparing the estimated online metrics to a set of baseline metrics of a baseline ranking function and detecting that the estimated online metrics of the proposed ranking function exceed, are lower than, or are within a predetermined range of the baseline metrics;
detect that the proposed ranking function is a preferred ranking function having a higher detected metric score than the baseline ranking function based on a comparison of the estimated online metrics with the baseline metrics of the baseline ranking function;
execute an action on an information retrieval system based on the preferred ranking function in response to detecting that a quality of the preferred ranking function exceeds a threshold; and
display generated results from the information retrieval system.
2. The system of claim 1 , wherein the processor is to pre-process the production log data by aggregating the production log data by query, by action, by probability of action and by reward value.
3. The system of claim 1 , wherein the processor is to use approximate action matching of rankings to estimate online metrics.
4. The system of claim 3 , wherein approximate action matching comprises comparing a predetermined number of higher-ranked results for each action generated by a respective ranking function.
5. The system of claim 1 , the online metrics comprising at least one of a click-through rate (CTR), a time to click on a search engine results page (SERP), and a mean reciprocal of click positions.
6. The system of claim 1 , the preferred ranking function to be used to execute an action on an information retrieval system in response to detecting that a quality of the preferred ranking function exceeds a threshold during the test.
7. The system of claim 6 , the action comprising displaying a search engine results page (SERP) in response to a query.
8. The system of claim 1 , wherein the processor is to generate query results with the proposed ranking function as a test of the proposed ranking function; and
display the generated query results.
9. A method for offline evaluation of ranking function performance, the method comprising:
receiving production log data;
pre-processing the production log data to generate an exploration data set, wherein the production log data comprises continuous updates of a ranking function associated with different actions for a same query issued by a same user in different impressions;
performing an offline estimation of online metrics using the exploration data set for a plurality of ranking functions, wherein a proposed ranking function is received and tested offline prior to being tested online on users;
comparing the plurality of ranking functions based on the estimated online metrics to generate comparison results;
identifying one or more preferred ranking functions having a higher detected metric score than the baseline ranking function based on the comparison results;
generating query results with the preferred ranking function during a testing process; and
displaying the generated query results.
10. The method of claim 9 , further comprising approximately matching actions in the exploration data set.
11. The method of claim 10 , further comprising approximately matching actions in the exploration data set by matching a predetermined number of higher-ranked results for each action.
12. The method of claim 9 , further comprising detecting whether a first ranking function from the plurality of ranking functions has a relevance score that is within a predetermined range of the relevance score of a second ranking function, higher than the predetermined range, or lower than the predetermined range.
13. The method of claim 9 , further comprising calculating a confidence score that indicates a level of certainty of the comparison results and displaying the confidence score with an associated comparison result.
14. The method of claim 9 , preprocessing the production log data further comprising aggregating the production log data by query, by action, by probability of action and by reward value.
15. The method of claim 9 , further comprising sending the preferred ranking function to a server during the testing process.
16. The method of claim 15 , further comprising testing the preferred ranking function on users via the server during the testing process.
17. One or more computer-readable memory storage devices for storing computer readable instructions that, when executed by one or more processing devices, instruct the offline evaluation of ranking function performance, the computer-readable instructions comprising code to:
receive production log data;
preprocess the production log data to generate an exploration data set, wherein the production log data comprises continuous updates of a ranking function associated with different actions for a same query issued by a same user in different impressions;
perform offline estimates of online metrics for a ranking function based at least in part on the exploration data set and an approximate action matching process, wherein a proposed ranking function is received and tested offline prior to being tested online on users;
detect that the ranking function is a preferred ranking function having a higher detected metric score than the baseline ranking function based on a comparison of the estimated online metrics with baseline ranking function metrics;
execute an action on an information retrieval system based on the preferred ranking function in response to detecting that a quality of the preferred ranking function exceeds a threshold during a testing process; and
display generated results from the information retrieval system.
18. The one or more computer-readable memory storage devices of claim 17 , the code for the comparison of online metrics further comprising code to:
calculate a delta metric score between the preferred ranking function and the baseline ranking function; and
detect that the delta metric score indicates that the preferred ranking function has a higher estimated online metric than the online metric of the baseline ranking function.
19. The system of claim 1 , wherein a contextual bandit model is used to generate the exploration data set.
20. The system of claim 1 , wherein the ranking function behavior comprises a changing feature of a query-document pair or an update of an engine index.