Position-bias correction for predictive and ranking systems
Methods, systems, and computer programs are presented for eliminating bias while training an ML model using training data that includes past experimental data. One method includes accessing experiment results, for A/B testing of a first model, that comprise information regarding engagement with a first set of items presented to users, each item being presented within an ordered list of results. A position bias is calculated for positions within the ordered list of results where the items were presented. A machine-learning program is trained to obtain a second model using a training set comprising values for features that include the calculated position bias. The method includes detecting a second set of items to be ranked for presentation to a first user, and calculates, using the second model, a relevance score for the second set of items, which are ranked based on the respective relevance score and presented on a display.
1. A computer-implemented method comprising:
collecting a set of items from a database;
causing a first presentation of a first plurality of the set of items on a display feed of a website, the first plurality of items including a first item;
applying a transformation to one or more of the items including the first item, wherein applying the transformation to an item includes repositioning the respective item on the display feed;
creating a first training set comprising accessed experiment results for A/B testing of an update to a first model, the experiment results comprising information regarding user engagement with the first plurality of the set of items when the first plurality of the set of items is presented to users, each item being presented within an ordered list of results on the display feed of the website, wherein the first item has a first position in the display feed that changes during A/B testing;
using the first training set, calculating, based on the experiment results, a position bias for positions within the ordered list of results;
creating a second training set comprising the calculated position biases;
training a machine-learning program using the second training set to obtain a second model having positionally debiased outputs;
detecting a second plurality of items to be ranked for presentation to a first user, the second plurality of items comprising the first item;
calculating, using the second model, a relevance score for each of the second plurality of items;
ranking the second plurality of items based on the respective relevance score; and
causing a second presentation of the ranked second plurality of items on the display feed of the website, wherein a second position of the first item in the second presentation changes from the first position in the first presentation based in part on the positionally debiased outputs of the second model.
2. The method as recited in claim 1 , wherein calculating the position bias further includes:
utilizing an instrumental variables (IV) method to estimate an effect of feed position on responses of users utilizing an instrumental variable for a past experiment that affected item position.
3. The method as recited in claim 2 , wherein the A/B testing affected positions of the first plurality of items, wherein requests in which an item was shown in different positions during the experiment are considered for calculating the position bias.
4. The method as recited in claim 1 , wherein the position bias is a difference in expected performance of an item based on a location where the item is placed within the ordered list of results.
5. The method as recited in claim 1 , wherein the first plurality of items includes candidate recommendations of people you may know (PYMK) for a viewing user, wherein the first model is for predicting a probability that one candidate will accept an invite from the viewing user.
6. The method as recited in claim 1 , wherein the first plurality of items includes ads for placement on a user feed of a viewing user, wherein the first model is for A/B testing affected ad positions based on an ad-bidding procedure.
7. The method as recited in claim 6 , wherein calculating the position bias further includes:
calculating a coefficient on experiment showing an impact of a bidding experiment for a plurality of ad campaigns, the coefficient-on-experiment reflecting an effect of the experiment on average positions of ads in a news feed.
8. The method as recited in claim 6 , wherein the position bias is a difference in click-through rate for one ad when the ad is moved from a first position to a second position.
9. The method as recited in claim 1 , wherein A/B training includes testing a first version of the first model with a first group of users and testing a second version of the first model with a second group of users.
10. The method as recited in claim 1 , wherein results obtained by the second model during operation are used to improve the training set and using the improved training set to generate a newer version of the second model.
11. A system comprising:
a memory comprising instructions; and
one or more computer processors, wherein the instructions, when executed by the one or more computer processors, cause the system to perform operations comprising:
collecting a set of items from a database;
causing a first presentation of a first plurality of the set of items on a display feed of a website, the first plurality of items including a first item;
applying a transformation to one or more of the items including the first item, wherein applying the transformation to an item includes repositioning the respective item on the display feed;
creating a first training set comprising accessed experiment results for A/B testing of an update to a first model, the experiment results comprising information regarding user engagement with the first plurality of the set of items when the first plurality of the set of items is presented to users, each item being presented within an ordered list of results on the display feed of the website, wherein the first item has a first position in the display feed that changes during A/B testing;
using the first training set, calculating, based on the experiment results, a position bias for positions within the ordered list of results;
creating a second training set comprising the calculated position biases;
training a machine-learning program using the second training set to obtain a second model having positionally debiased outputs;
detecting a second plurality of items to be ranked for presentation to a first user, the second plurality of items comprising the first item;
calculating, using the second model, a relevance score for each of the second plurality of items;
ranking the second plurality of items based on the respective relevance score; and
causing a second presentation of the ranked second plurality of items on the display feed of the website, wherein a second position of the first item in the second presentation changes from the first position in the first presentation based in part on the positionally debiased outputs of the second model.
12. The system as recited in claim 11 , wherein calculating the position bias further includes:
utilizing an instrumental variables (IV) method to estimate an effect of feed position on responses of users utilizing an instrumental variable for a past experiment that affected item position.
13. The system as recited in claim 12 , wherein the A/B testing affected positions of the first plurality of items, wherein requests in which an item was shown in different positions during the experiment are considered for calculating the position bias.
14. The system as recited in claim 11 , wherein the position bias is a difference in expected performance of an item based on a location where the item is placed within the ordered list of results.
15. The system as recited in claim 11 , wherein the first plurality of items includes candidate recommendations of people you may know (PYMK) for a viewing user, wherein the first model is for predicting a probability that one candidate will accept an invite from the viewing user.
16. A tangible non-transitory machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:
collecting a set of items from a database;
causing a first presentation of a first plurality of the set of items on a display feed of a website, the first plurality of items including a first item;
applying a transformation to one or more of the items including the first item, wherein applying the transformation to an item includes repositioning the respective item on the display feed;
creating a first training set comprising accessed experiment results for A/B testing of an update to a first model, the experiment results comprising information regarding user engagement with the first plurality of the set of items when the first plurality of the set of items is presented to users, each item being presented within an ordered list of results on the display feed of the website, wherein the first item has a first position in the display feed that changes during A/B testing;
using the first training set, calculating, based on the experiment results, a position bias for positions within the ordered list of results;
creating a second training set comprising the calculated position biases;
training a machine-learning program using the second training set to obtain a second model having positionally debiased outputs;
detecting a second plurality of items to be ranked for presentation to a first user, the second plurality of items comprising the first item;
calculating, using the second model, a relevance score for each of the second plurality of items;
ranking the second plurality of items based on the respective relevance score; and
causing a second presentation of the ranked second plurality of items on the display feed of the website, wherein a second position of the first item in the second presentation changes from the first position in the first presentation based in part on the positionally debiased outputs of the second model.
17. The tangible non-transitory machine-readable storage medium as recited in claim 16 , wherein calculating the position bias further includes:
utilizing an instrumental variables (IV) method to estimate an effect of feed position on responses of users utilizing an instrumental variable for a past experiment that affected item position.
18. The tangible non-transitory machine-readable storage medium as recited in claim 17 , wherein the A/B testing affected positions of the first plurality of items, wherein requests in which an item was shown in different positions during the experiment are considered for calculating the position bias.
19. The tangible non-transitory machine-readable storage medium as recited in claim 16 , wherein the position bias is a difference in expected performance of an item based on a location where the item is placed within the ordered list of results.
20. The tangible machine-readable storage medium as recited in claim 16 , wherein the first plurality of items includes candidate recommendations of people you may know (PYMK) for a viewing user, wherein the first model is for predicting a probability that one candidate will accept an invite from the viewing user.