Estimating ad quality from observed user behavior
A system obtains ratings associated with a first set of advertisements hosted by one or more servers, where the ratings indicate a quality of the first set of advertisements. The system observes multiple different first user actions associated with user selection of advertisements of the first set of advertisements and derives a statistical model using the observed first user actions and the obtained ratings. The system further observes second user actions associated with user selection of a second advertisement hosted by the one or more servers and uses the statistical model and the second user actions to estimate a quality of the second advertisement.
1. A method implemented by one or more processors of a computer system, comprising:
obtaining, using one or more processors of the computer system, ratings associated with a set of advertisements, where the ratings, which include a measure of quality of the set of advertisements, are received by manual input from human raters;
logging, using one or more processors of the computer system, first user actions associated with user selection of advertisements of the set of advertisements, the first user actions representing a measure of user satisfaction with regard to the selected advertisements;
logging, using one or more processors of the computer system, second user actions associated with user selection of an unrated advertisement of a set of unrated advertisements;
calculating, by one or more processors of the computer system, a probability that the unrated advertisement is of a certain measure of quality based on the logged second user actions and on a probability generated by a probability model that operates based on the logged first user actions and the obtained ratings.
2. The method of claim 1 , further comprising;
testing a fit of the probability model to determine which of the logged first user actions are correlated with advertisements of a certain measure of quality.
3. The method of claim 1 , further comprising:
calculating a quality score associated with the unrated advertisement using the calculated probability; and
estimating a measure of quality of the unrated advertisement based on the calculated quality score.
4. The method of claim 3 , where calculating the quality score associated with the unrated advertisement comprises:
applying a function to the calculated probability to calculate the quality score.
5. The method of claim 1 , where the first user actions or the second user actions each comprise at least one of the following:
a duration of the selection of the selected rated and unrated advertisements;
a number of selections of other advertisements before or after the selection of the selected rated and unrated advertisements;
a number of selections of search results before or after the selection of the selected rated and unrated advertisements;
a number of selections of other types of results before or after the selection of the selected rated and unrated advertisements;
a number of document accesses before or after the selection of the selected rated and unrated advertisements;
a number of search queries before or after the selection of the selected rated and unrated advertisements;
a number of search queries associated with a user session that shows advertisements;
a number of repeat selections on a same given advertisement of the selected rated and unrated advertisements; or
an indication of whether a given advertisement, of the selected rated and unrated advertisements, was a last advertisement selection for a given query or a last selection in a user session.
6. The method of claim 1 , where the logged first user actions include a period of time from the selection of the selected advertisements until a next user action; or
the logged second user actions include a period of time from the selection of the unrated advertisement to a next user action.
7. The method of claim 6 , where the period of time from the selection of the selected advertisements until a next user action or the period of time from the selection of the unrated advertisement to a next user action is a measure of user satisfaction with regard to the selected advertisements or the selected unrated advertisement.
8. A method implemented by one or more processors of a computer system, comprising:
obtaining, using one or more processors of the computer system, first session features associated with user selections of first advertisements hosted by one or more servers, the first session features representing a measure of user satisfaction with the selected first advertisements;
correlating, using one or more processors of the computer system, known quality ratings, associated with the first advertisements, with the first session features;
obtaining, using one or more processors of the computer system, second session features associated with user selection of a second unrated advertisement; and
estimating, using one or more processors of the computer system, a quality rating of the second unrated advertisement based on the obtained second session features and a probability generated by a probability model that operates based on the obtained first session features and the correlated quality ratings.
9. The method of claim 8 , where the first session features comprise at least one of the following:
a duration of the selection of the selected first advertisements;
a number of selections on other advertisements before or after the selection of the selected first advertisements;
a number of selections of search results before or after the selection of the selected first advertisements;
a number of selections of other types of results before or after the selection of the selected first advertisements;
a number of document accesses before or after the selection of the selected first advertisements;
a number of search queries before or after the selection of the selected first advertisements;
a number of search queries associated with a user session that shows advertisements;
a number of repeat selections on a same given advertisement of the selected first advertisements; or
an indication of whether a given advertisement, of the selected first advertisements, was a last advertisement selection for a given query or a last selection in a user session.
10. The method of claim 8 , further comprising:
testing a fit of the probability model to determine which of the obtained first session features are correlated with advertisements of at least a certain measure of quality.
11. The method of claim 8 , further comprising:
predicting a quality rating for another unrated advertisement based on the calculated quality rating of the second unrated advertisement.
12. A system, comprising:
one or more processors to:
obtain a set of rated: advertisements in response to a user query;
obtain first session features associated with user selection of advertisements of the set of rated advertisements, the first session features representing a measure of user satisfaction with the selected rated advertisements;
obtain second session features associated with user selection of an unrated advertisement of a set of unrated advertisements; and
determine a probability that the unrated advertisement is of a certain measure of quality based on the second session features and a probability generated by a probability model that is based on the obtained first session features and ratings associated with the selected rated advertisements;
calculate a quality score for the unrated advertisement based on the calculated probability; and
promote, rank, or filter the unrated advertisement based on the calculated score.
13. The system of claim 12 , where the first session features or the second session features each comprise at least one of the following:
a duration of the selection of the selected rated and unrated advertisements;
a number of selections of other advertisements before or after the selection of the selected rated and unrated advertisements;
a number of selections of search results before or after the selection of the selected rated and unrated advertisements a given advertisement selection;
a number of selections of other types of results before or after the selection of the selected rated and unrated advertisements;
a number of document accesses before or after the selection of the selected rated and unrated advertisements;
a number of search queries before or after the selection of the selected rated and unrated advertisements;
a number of search queries associated with a user session that shows advertisements;
a number of repeat selections on a same given advertisement of the selected rated and unrated advertisements; or
an indication of whether a given advertisement, of the selected rated and unrated advertisements, was a last advertisement selection for a given query or a last selection in a user session.
14. The system of claim 12 , where the one or more processors further are to:
predict a quality score for another unrated advertisement based on the calculated quality score of the unrated advertisement.