Training a classifier to identify unknown users of an online system
An online system develops a model to predict the identity of unknown users accessing the online system. The online system interacts with users who are known by the online system (e.g., because they are logged in), termed known users, and users who are unknown by the online system. The model attempts to predict the identity of unknown users. To train the model, a set of training data with training weights is generated. The training data includes a set of access events from known users. The set can include access events from unknown users who accessed the system and subsequently became identified (referred to as hindsight events). To account for a distribution in training data, the training data is applied to a scoring model to identify training data that resembles known events. A scaling model then scales the scores to generate training weights. The weights may be higher for access events with characteristics that resembles hindsight events.
1. A method comprising:
receiving a set of events each representing an interaction of a device with content and the set of events including events associated with known users and events associated with unknown users, wherein a known user represents an event associated with a user of a plurality of users of an online system and an unknown user represents an event that is not associated with any user of the online system;
identifying a hindsight event, the hindsight event being an access event that is not associated with any user of the online system at a first time when the access event occurred but is associated with a known user of the online system at a second time that is after the first time;
scoring, by a scoring model, a subset of events including at least the identified hindsight event, the subset of events associated with known users by applying a classifier that generates classification scores, each classification score reflecting a likelihood of an event being associated with a known user or an unknown user;
scaling, by a scaling model, the scores associated with the subset of events to create training weights for the subset of events, the scaling model trained to scale the scores based on a ratio of unsampled events having unknown users and known users; and
training a classifier to predict the identity of unknown users using the subset of events associated with known users as training data, wherein the subset of events are weighed in the training data according to the training weights associated with the events.
2. The method of claim 1 , wherein a known user is associated with an event because the user was synchronized with the online system at a time the event occurred.
3. The method of claim 1 , wherein a known user is associated with an event at a time later than a time the event occurred.
4. The method of claim 1 , wherein a known user is not associated with an event at a time the event occurred.
5. The method of claim 1 , wherein the scoring model is trained from a sampled subset of events including a substantially equal proportion of events associated with known users and events associated with unknown users.
6. The method of claim 1 , wherein training the scaling model further includes:
identifying a ratio of events associated with known users and events associated with unknown users in the set of events;
selecting a second subset of events having the identified ratio of events;
scoring the second subset of events by the scoring model; and
training the scaling model to scale a score for an event based on the second subset of events and the scores of the second subset of events.
7. The method of claim 1 , wherein the scaling model applies Platt scaling or logistic regression.
8. The method of claim 1 , wherein training the classifier further comprises
training the classifier to:
for a given event associated with an unknown user, select users of the online system as candidates for the unknown user and rank the candidates according to a likelihood of the candidates being the unknown user.
9. A non-transitory computer-readable storage medium comprising stored instructions for:
receiving a set of events each representing an interaction of a device with content and the set of events including events associated with known users and events associated with unknown users, wherein a known user represents an event associated with a user of a plurality of users of an online system and an unknown user represents an event that is not associated with any user of the online system;
identifying a hindsight event, the hindsight event being an access event that is not associated with any user of the online system at a first time when the access event occurred but is associated with a known user of the online system at a second time that is after the first time;
scoring, by a scoring model, a subset of events including at least the identified hindsight event, the subset of events associated with known and unknown users by applying a classifier that generates classification scores, each classification score reflecting a likelihood of an event being associated with a known user or an unknown user;
scaling, by a scaling model, the scores associated with the subset of events to create training weights for the subset of events, the scaling model trained to scale the scores based on a ratio of unsampled events having unknown users and known users; and
training a classifier to predict the identity of unknown users with the subset of events associated with known users as training data, wherein the subset of events are weighed in the training data according to the training weights associated with the events.
10. The non-transitory computer-readable storage medium of claim 9 , wherein a known user is associated with an event because the user was synchronized with the online system at a time the event occurred.
11. The non-transitory computer-readable storage medium of claim 9 , wherein a known user is associated with an event at a time later than a time the event occurred.
12. The non-transitory computer-readable storage medium of claim 9 , wherein a known user is not associated with an event at a time the event occurred.
13. The non-transitory computer-readable storage medium of claim 9 , wherein the scoring model is trained from a sampled subset of events including a substantially equal proportion of events associated with known users and events associated with unknown users.
14. The non-transitory computer-readable storage medium of claim 9 , wherein training the scaling model further includes:
identifying a ratio of events associated with known users and events associated with unknown users in the set of events;
selecting a second subset of events having the identified ratio of events;
scoring the second subset of events by the scoring model; and
training the scaling model to scale a score for an event based on the second subset of events and the scores of the second subset of events.
15. The non-transitory computer-readable storage medium of claim 9 , wherein the scaling model applies Platt scaling or logistic regression.
16. The non-transitory computer-readable storage medium of claim 9 , wherein training the classifier further comprises training the classifier to:
for a given event associated with an unknown user, select users of the online system as candidates for the unknown user and rank the candidates according to a likelihood of the candidates being the unknown user.
17. A computer system comprising:
A computer processor; and
a non-transitory computer-readable storage medium comprising stored instructions executable by the computer processor for:
receiving a set of events each representing an interaction of a device with content and the set of events including events associated with known users and events associated with unknown users, wherein a known user represents an event associated with a user of a plurality of users of an online system and an unknown user represents an event that is not associated with any user of the online system;
identifying a hindsight event, the hindsight event being an access event that is not associated with any user of the online system at a first time when the access event occurred but is associated with a known user of the online system at a second time that is after the first time;
scoring, by a scoring model, a subset of events including at least the identified hindsight event, the subset of events associated with known and unknown users by applying a classifier that generates classification scores, each classification score reflecting a likelihood of an event being associated with a known user or an unknown user;
scaling, by a scaling model, the scores associated with the subset of events to create training weights for the subset of events, the scaling model trained to scale the scores based on a ratio of unsampled events having unknown users and known users; and
training a classifier to predict the identity of unknown users with the subset of events associated with known users as training data, wherein the subset of events are weighed in the training data according to the training weights associated with the events.
18. The computer system of claim 17 , wherein a known user is associated with an event because the user was synchronized with the online system at a time the event occurred.
19. The computer system of claim 17 , wherein a known user is associated with an event at a time later than a time the event occurred.
20. The computer system of claim 17 , wherein a known user is not associated with an event at a time the event occurred.