Offline evaluation of ranking functions

1. A system for offline evaluation of ranking functions, comprising a processor to:

receive production log data and pre-process the production log data to generate an exploration data set, wherein the production log data comprises continuous updates of a ranking function associated with different actions for a same query issued by a same user in different impressions;

perform offline estimation of online metrics for ranking functions using the exploration data set, wherein a proposed ranking function is to be received and tested offline prior to being tested online on users;

evaluate the proposed ranking function by comparing the estimated online metrics to a set of baseline metrics of a baseline ranking function and detecting that the estimated online metrics of the proposed ranking function exceed, are lower than, or are within a predetermined range of the baseline metrics;

detect that the proposed ranking function is a preferred ranking function having a higher detected metric score than the baseline ranking function based on a comparison of the estimated online metrics with the baseline metrics of the baseline ranking function;

execute an action on an information retrieval system based on the preferred ranking function in response to detecting that a quality of the preferred ranking function exceeds a threshold; and

display generated results from the information retrieval system.

2. The system of claim 1 , wherein the processor is to pre-process the production log data by aggregating the production log data by query, by action, by probability of action and by reward value.

3. The system of claim 1 , wherein the processor is to use approximate action matching of rankings to estimate online metrics.

4. The system of claim 3 , wherein approximate action matching comprises comparing a predetermined number of higher-ranked results for each action generated by a respective ranking function.

5. The system of claim 1 , the online metrics comprising at least one of a click-through rate (CTR), a time to click on a search engine results page (SERP), and a mean reciprocal of click positions.

6. The system of claim 1 , the preferred ranking function to be used to execute an action on an information retrieval system in response to detecting that a quality of the preferred ranking function exceeds a threshold during the test.

7. The system of claim 6 , the action comprising displaying a search engine results page (SERP) in response to a query.

8. The system of claim 1 , wherein the processor is to generate query results with the proposed ranking function as a test of the proposed ranking function; and

display the generated query results.

9. A method for offline evaluation of ranking function performance, the method comprising:

receiving production log data;

pre-processing the production log data to generate an exploration data set, wherein the production log data comprises continuous updates of a ranking function associated with different actions for a same query issued by a same user in different impressions;

performing an offline estimation of online metrics using the exploration data set for a plurality of ranking functions, wherein a proposed ranking function is received and tested offline prior to being tested online on users;

comparing the plurality of ranking functions based on the estimated online metrics to generate comparison results;

identifying one or more preferred ranking functions having a higher detected metric score than the baseline ranking function based on the comparison results;

generating query results with the preferred ranking function during a testing process; and

displaying the generated query results.

10. The method of claim 9 , further comprising approximately matching actions in the exploration data set.

11. The method of claim 10 , further comprising approximately matching actions in the exploration data set by matching a predetermined number of higher-ranked results for each action.

12. The method of claim 9 , further comprising detecting whether a first ranking function from the plurality of ranking functions has a relevance score that is within a predetermined range of the relevance score of a second ranking function, higher than the predetermined range, or lower than the predetermined range.

13. The method of claim 9 , further comprising calculating a confidence score that indicates a level of certainty of the comparison results and displaying the confidence score with an associated comparison result.

14. The method of claim 9 , preprocessing the production log data further comprising aggregating the production log data by query, by action, by probability of action and by reward value.

15. The method of claim 9 , further comprising sending the preferred ranking function to a server during the testing process.

16. The method of claim 15 , further comprising testing the preferred ranking function on users via the server during the testing process.

17. One or more computer-readable memory storage devices for storing computer readable instructions that, when executed by one or more processing devices, instruct the offline evaluation of ranking function performance, the computer-readable instructions comprising code to:

receive production log data;

preprocess the production log data to generate an exploration data set, wherein the production log data comprises continuous updates of a ranking function associated with different actions for a same query issued by a same user in different impressions;

perform offline estimates of online metrics for a ranking function based at least in part on the exploration data set and an approximate action matching process, wherein a proposed ranking function is received and tested offline prior to being tested online on users;

detect that the ranking function is a preferred ranking function having a higher detected metric score than the baseline ranking function based on a comparison of the estimated online metrics with baseline ranking function metrics;

execute an action on an information retrieval system based on the preferred ranking function in response to detecting that a quality of the preferred ranking function exceeds a threshold during a testing process; and

display generated results from the information retrieval system.

18. The one or more computer-readable memory storage devices of claim 17 , the code for the comparison of online metrics further comprising code to:

calculate a delta metric score between the preferred ranking function and the baseline ranking function; and

detect that the delta metric score indicates that the preferred ranking function has a higher estimated online metric than the online metric of the baseline ranking function.

19. The system of claim 1 , wherein a contextual bandit model is used to generate the exploration data set.

20. The system of claim 1 , wherein the ranking function behavior comprises a changing feature of a query-document pair or an update of an engine index.