First pass ranker calibration for news feed ranking
An on-line social network system is configured to generate a news feed for a member by processing updates originating from different sources using different first pass ranker models. The first pass ranker models generate respective sets of raw scores, which are calibrated based on a consistent scale of feed engagement metrics of interest, such as a click through rate. The calibrated scores are then used as training data to train a second pass ranker and/or as input into the second pass ranker at the time when the second pass ranker is to generate respective ranks for items in an inventory of updates identified as potentially of interest to a focus member and to select a subset of items from the inventory based on the generated respective ranks.
1. A computer-implemented method comprising:
detecting a request to generate a news feed web page for a focus member represented by a profile in an on-line social network system, the news feed web page to display selected updates that represent content items associated with respective categories, a first category from the respective categories indicating a first source from which updates displayed on the news feed web page generated for a focus member originated and a second category from the respective categories indicating a second source from which updates displayed on the news feed web page generated for a focus member originated;
processing updates that originated from the first source by a first preliminary ranker model to generate a first set of scores;
processing updates that originated from the second source by a second preliminary ranker model to generate a second set of scores;
accessing a set of raw scores comprising the first set of scores and the second set of scores, each score from the set of raw scores reflecting a member's interaction with an update;
using at least one processor, producing a set of calibrated preliminary ranker scores by executing a statistical model using the set of raw scores as input;
using the set of calibrated preliminary ranker scores as input into a second pass ranker model, the second pass ranker model to generate respective ranks for items in an inventory of updates identified as potentially of interest to the focus member and to select a subset of items from the inventory of updates based on the generated respective ranks.
2. The method of claim 1 , wherein the statistical model is logistic regression or Nadaraya-Watson kernel regression.
3. The method of claim 2 , wherein the statistical model takes, as input, a transformation of the set of raw scores, the transformation is a logarithmic or an exponential function.
4. The method of claim 1 , comprising:
constructing a news feed web page that includes the subset of items from the inventory; and
causing presentation of the news feed web page on a display device of the focus member.
5. The method of claim 1 , comprising including scores from the set of calibrated preliminary ranker scores, as training data for training the second pass ranker model.
6. The method of claim 1 , wherein a score from the set of raw scores represents estimated relevance of an update to the focus member and the respective feed engagement value reflects a click through rate.
7. The method of claim 1 , comprising:
monitoring engagement events with respect to updates of the first category in news feeds generated for members in the on-line social network system;
generating engagement features based on the monitored events; and
including the engagement features as training data for training the first preliminary ranker model.
8. The method of claim 7 , wherein the monitored events are views or viral actions with respect to items in a news feed generated in the on-line social network system.
9. The method of claim 1 , wherein updates of the first category are updates posted by members who are connections of the focus member in the on-line social network system.
10. The method of claim 1 , wherein updates of the second category are job postings.
11. A computer-implemented system comprising:
one or more processors; and
a non-transitory computer readable storage medium comprising instructions that when executed by the one or processors cause the one or more processors to perform operations comprising:
detecting a request to generate a news feed web page for a focus member represented by a profile in an on-line social network system, the news feed web page to display selected updates that represent content items associated with respective categories, a first category from the respective categories indicating a first source from which updates displayed on the news feed web page generated for a focus member originated and a second category from the respective categories indicating a second source from which updates displayed on the news feed web page generated for a focus member originated;
processing updates that originated from the first source by a first preliminary ranker model to generate a first set of scores;
processing updates that originated from the second source by a second preliminary ranker model to generate a second set of scores;
accessing a set of raw scores comprising the first set of scores and the second set of scores, each score from the set of raw scores reflecting a member's interaction with an update;
producing a set of calibrated preliminary ranker scores by executing a statistical model using the set of raw scores as input;
using the set of calibrated preliminary ranker scores as input into a second pass ranker model, the second pass ranker model to generate respective ranks for items in an inventory of updates identified as potentially of interest to the focus member and to select a subset of items from the inventory of updates based on the generated respective ranks.
12. The system of claim 11 , wherein the statistical model is logistic regression or Nadaraya-Watson kernel regression.
13. The system of claim 12 , wherein the statistical model takes, as input, a transformation of the set of raw scores, the transformation is a logarithmic or an exponential function.
14. The system of claim 11 , wherein the one or more processors to perform operations comprising:
a news feed web page generator, implemented using at least one processor, to construct constructing a news feed web page that includes the subset of items from the inventory; and
a presentation module, implemented using at least one processor, to cause causing presentation of the news feed web page on a display device of the focus member.
15. The system of claim 11 , wherein scores from the set of calibrated preliminary ranker scores are used as training data for training the second pass ranker model.
16. The system of claim 11 , wherein a score from the set of raw scores represents estimated relevance of an update to the focus member and the respective feed engagement value reflects a click through rate.
17. The system of claim 11 , wherein the one or more processors to perform operations comprising:
monitor monitoring engagement events with respect to updates of the first category in news feeds generated for members in the on-line social network system;
generate engagement features based on the monitored events; and
include the engagement features as training data for training the first preliminary ranker model.
18. The system of claim 17 , wherein the monitored events are views or viral actions with respect to items in a news feed generated in the on-line social network system.
19. The system of claim 11 , wherein updates of the first category are updates posted by members who are connections of the focus member in the on-line social network system.
20. A machine-readable non-transitory storage medium having instruction data executable by a machine to cause the machine to perform operations comprising:
detecting a request to generate a news feed web page for a focus member represented by a profile in an on-line social network system, the news feed web page to display selected updates that represent content items associated with respective categories, a first category from the respective categories indicating a first source from which updates displayed on the news feed web page generated for a focus member originated and a second category from the respective categories indicating a second source from which updates displayed on the news feed web page generated for a focus member originated;
processing updates that originated from the first source by a first preliminary ranker model to generate a first set of scores;
processing updates that originated from the second source by a second preliminary ranker model to generate a second set of scores;
accessing a set of raw scores comprising the first set of scores and the second set of scores, each score from the set of raw scores reflecting a member's interaction with an update;
using at least one processor, producing a set of calibrated preliminary ranker scores by executing a statistical model using the set of raw scores as input;
using the set of calibrated preliminary ranker scores as input into a second pass ranker model, the second pass ranker model to generate respective ranks for items in an inventory of updates identified as potentially of interest to the focus member and to select a subset of items from the inventory of updates based on the generated respective ranks.