Linear programming-based dynamic blending model
In an example embodiment, a blending model is presented based on a linear programming approach. The blending model produces a slate of sponsored and non-sponsored pieces of content for display in a graphical user interface, with the ordering and placement of the sponsored and non-sponsored pieces of content selected in order to maximize an objective function. Such an approach can fine tune each piece of content using content-level parameters and holistically examine global constraints and opportunities. It establishes a robust optimization framework that can adapt to content and domain changes without requiring tuning through online experiments.
1 . A system comprising:
a computer-readable medium having instructions stored thereon, which, when executed by a processor, cause the system to perform operations comprising:
in an offline mode:
generate one or more parameters by executing a linear programming blending model on a plurality of sponsored pieces of content and a plurality of non-sponsored pieces of content, the linear programming blending model calculating a first slate of sponsored and non-sponsored pieces of content, the first slate maximizing an objective function subject to a guardrail, wherein a guardrail is a metric whose value cannot drop below a preset value in the first slate;
in an online mode:
generate a first ordered list of sponsored pieces of content using a sponsored relevance model and a second ordered list of non-sponsored pieces of content using a non-sponsored relevance model, wherein the sponsored relevance model and the non-sponsored relevance model are machine learning (ML) models using a set of features comprising information about a user, a display channel, or a piece of content;
execute the linear programming blending model on pieces of content from the first ordered list of sponsored pieces of content and the second ordered list of non-sponsored pieces of content, using the one or more parameters generated during the offline mode and a parameter generated during the online mode, the linear programming blending model calculating a second slate of sponsored pieces of content and non-sponsored pieces of content, the second slate maximizing the objective function;
create a pattern for display of pieces of content by indicating, at each slot in the second slate, whether a corresponding piece of content in the second slate is sponsored or non-sponsored;
assign one or more sponsored pieces of content from the first ordered list to slots in the pattern that have been identified as sponsored and one or more non-sponsored pieces of content from the second ordered list to slots in the pattern that have been identified as non-sponsored; and
cause display, in a graphical user interface, of the assigned one or more sponsored pieces of content and the assigned one or more non-sponsored pieces of content in the slots in the pattern in which they have been assigned.
2 . The system of claim 1 , wherein the operations further comprise:
determine position bias scores for a plurality of possible slots based on a position bias function; and
wherein the executing is additionally performed using the determined position bias scores.
3 . The system of claim 1 , wherein the linear programming model implements a Lagrange dual function.
4 . The system of claim 1 , wherein the guardrail is total revenue.
5 . The system of claim 1 , wherein the guardrail is overall views.
6 . The system of claim 1 , wherein the pieces of content are job listings and the one or more guardrails include a number of applies to jobs corresponding to the job listings.
7 . The system of claim 1 , wherein the linear programming blending model takes as input a bid for each piece of content.
8 . The system of claim 7 , wherein for sponsored pieces of content the bid is an amount a content poster has agreed to pay for an impression of the corresponding sponsored piece of content.
9 . The system of claim 7 , wherein for non-sponsored pieces of content the bid is a shadow bid indicative of a hypothetical value of an impression of the corresponding non-sponsored piece of content.
10 . The system of claim 1 , wherein the second slate includes a pattern in accordance with one or more constraints on placement locations for sponsored and/or non-sponsored pieces of content.
11 . The system of claim 10 , wherein the one or more constraints include a maximum number of consecutive sponsored pieces of content beginning at a highest slot of a display area.
12 . The system of claim 10 , wherein the one or more constraints include a maximum number of consecutive sponsored pieces of content.
13 . The system of claim 10 , wherein the one or more constraints include a minimum gap between two blocks of a set block size number of sponsored pieces of content.
14 . The system of claim 10 , wherein the one or more constraints include a total maximum number of sponsored pieces of content in a displayed page.
15 . The system of claim 10 , wherein the one or more constraints include a minimum number of consecutive non-sponsored pieces of content beginning at a highest slot of a display area.
16 . The system of claim 10 , wherein the one or more constraints include a maximum number of sponsored pieces of content from any one job poster or company per page.
17 . The system of claim 1 , wherein the pattern is dependent on a device on which the graphical user interface is rendered.
18 . A computerized method comprising:
in an offline mode:
generating one or more parameters by executing a linear programming blending model on a plurality of sponsored pieces of content and a plurality of non-sponsored pieces of content, the linear programming blending model calculating a first slate of sponsored and non-sponsored pieces of content, the first slate maximizing an objective function subject to a guardrail, wherein a guardrail is a metric whose value cannot drop below a preset value in the first slate;
in an online mode:
generating a first ordered list of sponsored pieces of content using a sponsored relevance model and a second ordered list of non-sponsored pieces of content using a non-sponsored relevance model, wherein the sponsored relevance model and the non-sponsored relevance model are machine learning (ML) models using a set of features comprising information about a user, a display channel, or a piece of content;
executing the linear programming blending model on pieces of content from the first ordered list of sponsored pieces of content and the second ordered list of non-sponsored pieces of content, using the one or more parameters generated during the offline mode and a parameter generated during the online mode, the linear programming blending model calculating a second slate of sponsored pieces of content and non-sponsored pieces of content, the second slate maximizing the objective function;
creating a pattern for display of pieces of content by indicating, at each slot in the second slate, whether a corresponding piece of content in the second slate is sponsored or non-sponsored;
assigning one or more sponsored pieces of content from the first ordered list to slots in the pattern that have been identified as sponsored and one or more non-sponsored pieces of content from the second ordered list to slots in the pattern that have been identified as non-sponsored; and
causing display, in a graphical user interface, of the assigned one or more sponsored pieces of content and the assigned one or more non-sponsored pieces of content in the slots in the pattern in which they have been assigned.
19 . A system comprising:
means for, in an offline mode:
generating one or more parameters by executing a linear programming blending model on a plurality of sponsored pieces of content and a plurality of non-sponsored pieces of content, the linear programming blending model calculating a first slate of sponsored and non-sponsored pieces of content, the first slate maximizing an objective function subject to a guardrail, wherein a guardrail is a metric whose value cannot drop below a preset value in the first slate;
means for, in an online mode:
generating a first ordered list of sponsored pieces of content using a sponsored relevance model and a second ordered list of non-sponsored pieces of content using a non-sponsored relevance model, wherein the sponsored relevance model and the non-sponsored relevance model are machine learning (ML) models using a set of features comprising information about a user, a display channel, or a piece of content;
executing the linear programming blending model on pieces of content from a first ordered list of sponsored pieces of content and a second ordered list of non-sponsored pieces of content, using the one or more parameters generated during the offline mode and a parameter generated during the online mode, the linear programming blending model calculating a second slate of sponsored pieces of content and non-sponsored pieces of content, the second slate maximizing the objective function;
creating a pattern for display of pieces of content by indicating, at each slot in the second slate, whether a corresponding piece of content in the second slate is sponsored or non-sponsored;
assigning one or more sponsored pieces of content from the first ordered list to slots in the pattern that have been identified as sponsored and one or more non-sponsored pieces of content from the second ordered list to slots in the pattern that have been identified as non-sponsored; and
causing display, in a graphical user interface, of the assigned one or more sponsored pieces of content and the assigned one or more non-sponsored pieces of content in the slots in the pattern in which they have been assigned.