Systems and methods for slate optimization with recurrent neural networks

1. A computer system comprising:

one or more processors; and

one or more non-transitory computer readable media that collectively store:

a machine-learned pointer network for generating an output sequence from a list of candidate items, wherein the output sequence comprises a ranking of the candidate items, wherein the ranking of the candidate items orders the candidate items included in the output sequence based on a respective probability of user engagement with each candidate item included in the output sequence, the machine-learned pointer network comprising:

an encoder network configured to receive the list of candidate items and provide an output that includes a sequence of latent memory states;

a decoder network configured to receive a previously-selected candidate item for the output sequence and provide an output vector based at least in part on the previously-selected candidate item; and

an attention network configured to receive the sequence of latent memory states and a query including the output vector from the decoder network, the attention network configured to produce a probability distribution associated with a next candidate item to include in the output sequence, wherein the attention network produces the probability distribution based at least in part on candidate items that already appear in the output sequence;

wherein the machine-learned pointer network has been trained on a training dataset comprising a plurality of training examples, wherein each training example comprises a sequence of training items and a sequence of labels respectively associated with the sequence of training items, wherein the respective label for each training item indicates a level of user engagement with the training item; and

wherein the one or more non-transitory computer readable media also collectively store instructions that, when executed by the one or more processors, cause the computer system to perform operations, the operations comprising:

providing an input associated with the list of candidate items to the machine-learned pointer network;

implementing the machine-learned pointer network to process the list of candidate items;

receiving an output generated by the machine-learned pointer network as a result of processing the list of candidate items; and

selecting the next candidate item to include in the output sequence based at least in part on the probability distribution.

2. The computer system of claim 1 , wherein:

the attention network assigns scores to the list of candidate items as part of producing the probability distribution; and

the scores assigned to candidate items that already appear in the output sequence are lower than the scores assigned to candidate items that do not already appear in the output sequence.

3. The computer system of claim 2 , wherein:

the scores assigned to candidate items that already appear in the output sequence enforces the attention network to output permutations.

4. The computer system of claim 1 , wherein the list of candidate items comprises a sequence of features associated with the list of candidate items.

5. The computer system of claim 1 , wherein, for at least one of the plurality of training examples, the respective label for each training item comprises a binary label that indicates whether a user clicked or did not click on the training item.

6. A computer-implemented method to train a machine-learned pointer network, the method comprising:

obtaining, by one or more computing devices, data descriptive of the machine-learned pointer network configured to generate an output sequence from a list of candidate items, wherein the output sequence comprises a ranking of the candidate items, wherein the ranking of the candidate items orders the candidate items included in the output sequence based on a respective probability of user engagement with each candidate item included in the output sequence, wherein the machine-learned pointer network comprises an encoder network configured to receive the list of candidate items and provide an output that includes a sequence of latent memory states, a decoder network that operates over a plurality of decoding steps and is configured to receive a previously-selected candidate item for the output sequence and provide an output vector based at least in part on the previously-selected candidate item, and an attention network configured to receive the sequence of latent memory states and a query including the output vector from the decoder network, wherein the attention network is configured to produce a probability distribution associated with a next candidate item to include in the output sequence, and wherein, at each of the plurality of decoding steps, the attention network assigns scores to the list of candidate items as part of producing the probability distribution;

training, by the one or more computing devices, the machine-learned pointer network based on a set of training data comprising a plurality of training examples, wherein each training example comprises a sequence of training items and a sequence of labels respectively associated with the sequence of training items, wherein the respective label for each training item indicates a level of user engagement with the training item,

wherein training, by the one or more computing devices, the machine-learned pointer network comprises:

determining, by the one or more computing devices, a per-step loss for two or more of the plurality of decoding steps, the per-step loss at each of the two or more of the plurality of decoding steps representing a performance evaluation of the machine-learned pointer network based on the set of training data at such decoding step, and wherein the per-step loss at each of the two or more of the plurality of decoding steps considers only training labels assigned to training items that have not been previously selected for inclusion in the output sequence; and

modifying, by the one or more computing devices, one or more parameters of the machine-learned pointer network based at least in part on the per-step loss.

7. The computer-implemented method of claim 6 , wherein training, by the one or more computing devices, the machine-learned pointer network comprises:

enforcing a permutation constraint on an output of the machine-learned pointer network.

8. The computer-implemented method of claim 6 , wherein modifying, by the one or more computing devices, one or more parameters of the machine-learned pointer network comprises backpropagating the per-step loss to train the machine-learned pointer network end-to-end.

9. The computer-implemented method of claim 6 , wherein feedback based on the per-step loss is provided to the machine-learned pointer network at each of a plurality of decoding steps of the decoder network.

10. The computer-implemented method of claim 6 , wherein, for at least one of the plurality of training examples, the respective label for each training item comprises a binary label that indicates whether a user clicked or did not click on the training item.

11. The computer-implemented method of claim 6 , wherein the per-step loss at each of the two or more of the plurality of decoding steps comprises a respective per-step weight.

12. The computer-implemented method of claim 11 , wherein the per-step weights associated with the two or more decoding steps decrease in magnitude over the two or more of the plurality of decoding steps.

13. The computer-implemented method of claim 6 , wherein determining, by the one or more computing devices, the per-step loss for the two or more of the plurality of decoding steps comprises, for each training example and for each of the two or more of the plurality of decoding steps, adjusting to equal zero, by the one or more computing devices, each label in the sequence of labels that corresponds to training items that have been previously selected for inclusion in the output sequence.

14. A computer-implemented method to generate an output sequence from a list of candidate items, the method comprising:

ranking, by one or more computing devices, the list of candidate items to generate an initial sequence of candidate items;

inputting, by the one or more computing devices, the initial sequence of candidate items into an encoder network configured to receive the initial sequence of candidate items and provide an output that includes a sequence of latent memory states;

inputting, by the one or more computing devices, a previously-selected candidate item for the output sequence into a decoder network configured to receive the previously-selected candidate item and provide an output vector based at least in part on the previously-selected candidate item; and

inputting the sequence of latent memory states and a query including the output vector from the decoder network into an attention network configured to receive the sequence of latent memory states and the query including the output vector from the decoder network, the attention network configured to produce a probability distribution associated with a next candidate item to include in the output sequence, wherein the output sequence comprises a ranking of the candidate items, wherein the ranking of the candidate items orders the candidate items included in the output sequence based on a respective probability of user engagement with each candidate item included in the output sequence;

wherein the encoder network, the decoder network, and the attention network have been jointly trained on a training dataset comprising a plurality of training examples, wherein each training example comprises a sequence of training items and a sequence of labels respectively associated with the sequence of training items, wherein the respective label for each training item indicates a level of user engagement with the training item.

15. The computer-implemented method of claim 14 , wherein:

the decoder network operates over a plurality of decoding steps including a first decoding step; and

a learned vector is provided as an input to the first decoding step, and in response the decoder network provides an output vector based at least in part on the learned vector.

16. The computer-implemented method of claim 15 , wherein:

the plurality of decoding steps includes a second decoding step; and

the method further comprises inputting an embedding that corresponds to the previously-selected candidate item into the second decoding step of the decoder network.

17. The computer-implemented method of claim 14 , wherein the list of candidate items comprises a sequence of features associated with the list of candidate items.

18. The computer-implemented method of claim 14 , wherein the encoder network, the decoder network, and the attention network are part of a machine-learned pointer network that is trained to point to candidate items in the list of candidate items based on a plurality of learned parameters.

19. The computer-implemented method of claim 14 , wherein the attention network is configured to produce the probability distribution based at least in part on candidate items that already appear in the output sequence.

20. The computer-implemented method of claim 14 , wherein a score assigned to each candidate item that already appears in the output sequence enforces the attention network to output permutations.