Session embeddings for summarizing activity

1. A method, comprising:

obtaining a first sentence representing a first session with a user and a second sentence representing a second session with the user, wherein each of the first and second sessions comprises a sequence of actions between the user and a set of jobs;

applying, by one or more computer systems, a language model to token embeddings of a first set of tokens in the first sentence and position embeddings of token positions in the first sentence to produce a first session embedding for the first session;

applying, by the one or more computer systems, the language model to the token embeddings of a second set of tokens in the second sentence and the position embeddings of the token positions in the second sentence to produce a second session embedding for the second session; and

outputting the first and second session embeddings for use in characterizing job-seeking activity of the user.

2. The method of claim 1 , further comprising:

combining the first and second session embeddings into a session history embedding representing the first and second sessions.

3. The method of claim 2 , wherein combining the first and second session embeddings into the session history embedding comprises:

generating the session history embedding as an element-wise maximum of the first and second session embeddings.

4. The method of claim 2 , wherein combining the first and second session embeddings into the session history embedding comprises:

inputting the first and second session embeddings into a sequence model; and

obtaining the session history embedding as output from the sequence model.

5. The method of claim 2 , wherein combining the first and second session embeddings into the session history embedding comprises:

identifying a subset of tokens in the first and second sentences that represent a subset of actions in the first and second sessions; and

aggregating output embeddings generated by the language model from the subset of tokens into the session history embedding.

6. The method of claim 1 , further comprising:

obtaining a third sentence representing a third session with the user and a fourth sentence representing a fourth session with the user; and

applying the language model to the token embeddings of a third set of tokens in the third sentence and a fourth set of tokens in the fourth sentence, a separator embedding of a separator token between the third and fourth sets of tokens, a first segment embedding for the third sentence, a second segment embedding for the fourth sentence, and the position embeddings of the token positions in the third and fourth sentences to produce a session history embedding.

7. The method of claim 1 , further comprising:

training the language model to predict at least one of:

masked tokens in a set of sample sentences; and

a next sentence in the set of sample sentences from a current sentence in the set of sample sentences.

8. The method of claim 1 , wherein outputting the first and second session embeddings for use in characterizing the job-seeking activity of the user comprises:

inputting the first and second session embeddings and additional features related to a job into a machine learning model;

receiving, as output from the machine learning model, a score representing a likelihood of a positive outcome between the user and the job; and

generating a recommendation related to the job based on the score.

9. The method of claim 8 , wherein the recommendation comprises at least one of:

a query suggestion for a job search by the user; and

a recommendation of the job to the user.

10. The method of claim 1 , wherein the first session embedding comprises an output embedding generated by the language model from a classification token inputted into the language model.

11. The method of claim 1 , wherein the language model comprises a bidirectional transformer encoder.

12. The method of claim 1 , wherein the first and second sentences comprise at least one of:

an action type;

a job identifier;

a title;

a company;

a location; and

a search term; and

a keyword.

13. A system, comprising:

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the system to:

obtain a first sentence representing a first sequence of actions between a user and a set of jobs;

apply a language model to token embeddings of a first set of tokens in the first sentence and position embeddings of token positions in the first sentence to produce a first set of output embeddings;

combine the first set of output embeddings into a first session embedding that encodes the first sequence of actions; and

output the first session embedding for use in characterizing job-seeking activity of the user.

14. The system of claim 13 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the system to:

apply the language model to a second sentence representing a second sequence of actions between the user and an additional set of jobs to produce a second set of output embeddings;

combine the second set of output embeddings into a second session embedding that encodes the second sequence of actions; and

combine the first and second session embeddings into a session history embedding representing the first and second sequences of actions.

15. The system of claim 14 , wherein combining the first and second session embeddings into the session history embedding comprises:

inputting the first and second session embeddings into a sequence model; and

obtaining the session history embedding as output from the sequence model.

16. The system of claim 13 , wherein outputting the first session embedding for use in characterizing the job-seeking activity of the user comprises:

inputting the first session embedding and additional features related to a job into a machine learning model;

receiving, as output from the machine learning model, a score representing a likelihood of a positive outcome between the user and the job; and

generating a recommendation related to the job based on the score.

17. The system of claim 16 , wherein the recommendation comprises at least one of:

a query suggestion for a job search by the user; and

a recommendation of the job to the user.

18. The system of claim 13 , wherein the language model comprises a bidirectional transformer encoder.

19. The system of claim 13 , wherein the first session embedding comprises an output embedding generated by the language model from a classification token inputted into the language model.

20. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method, the method comprising:

obtaining a first sentence representing a first session with a user and a second sentence representing a second session with the user, wherein each of the first and second sessions comprises a sequence of actions performed by the user with an online system;

applying a language model to token embeddings of a first set of tokens in the first sentence and position embeddings of token positions in the first sentence to produce a first session embedding for the first session, wherein the first session embedding comprises an output embedding generated by the language model from a classification token added to the first sentence;

applying the language model to the token embeddings of a second set of tokens in the second sentence and the position embeddings of the token positions in the second sentence to produce a second session embedding for the second session; and

outputting the first and second session embeddings for use in characterizing activity of the user.