Training sequence generation neural networks using quality scores

1. A method of training a neural network having a plurality of network parameters and being configured to map a system input to an output sequence comprising a plurality of system outputs, wherein the method comprises:

obtaining a batch of training examples, each training example comprising a training network input and, for each training network input, a ground truth output sequence;

for each of the training examples:

processing the training network input in the training example using the neural network and in accordance with current values of the network parameters to map the training network input to an output sequence comprising a respective system output from a vocabulary of possible system outputs at each of a plurality of output positions;

for each particular output position of the plurality of output positions in the output sequence:

identifying a prefix that includes the system outputs at positions before the particular output position in the output sequence,

for each possible system output in the vocabulary, determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by the possible system output and followed by any suffix of one or more system outputs, wherein the quality score measures a quality of a candidate output sequence relative to the ground truth output sequence, and

determining, using the highest quality scores for the possible system outputs, an update to the current values of the network parameters that increases a likelihood that the neural network generates a system output at the position that has a high quality score; and

determining updated values of the network parameters from the updates for the particular output positions in the output sequences generated by the neural network for the batch of training examples.

2. The method of claim 1 , further comprising:

outputting the trained neural network for use in mapping new network inputs to new output sequences.

3. The method of claim 1 , wherein the neural network is configured to, for each of the output positions, generate a likelihood distribution over the possible system outputs in the vocabulary and select the system output at the output position using the likelihood distribution.

4. The method of claim 3 , wherein determining, using the highest quality scores for the possible system outputs, an update to the current values of the network parameters that increases a likelihood that the neural network generates a system output at the position that has a high quality score comprises:

generating a target likelihood distribution for the output position from the highest quality sores for the possible system outputs,

determining a gradient with respect to the network parameters of an objective function that depends on a divergence between the target likelihood distribution for the output position and the likelihood distribution generated by the neural network for the output position, and

determining the update to the current values using the gradient.

5. The method of claim 4 , wherein generating the target likelihood distribution comprises applying a softmax to the highest quality scores for the possible system outputs to generate a respective likelihood for each of the possible system outputs.

6. The method of claim 5 , wherein the softmax is applied with reduced temperature.

7. The method of claim 1 , wherein the possible system outputs in the vocabulary comprise tokens in a natural language.

8. The method of claim 1 , wherein the quality score is based on the edit distance between the candidate output sequence and the ground truth output sequence, and wherein the highest quality score that can be assigned is the quality score for the candidate output sequence that has a smallest edit distance to the ground truth output sequence.

9. The method of claim 1 , wherein determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by the possible system output and followed by any suffix of one or more system outputs comprises:

determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by any ground truth suffix that is part of the ground truth output sequence;

identifying one or more ground truth prefixes of the ground truth output sequence relative to which the prefix has a highest quality score,

identifying, for each of the identified ground truth prefixes, a respective ground truth suffix that follows the identified ground truth prefix in the ground truth sequence, and

when the possible system output is a first system output in any of the identified ground truth suffixes, assigning as the highest quality score for the possible system output the highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by any ground truth suffix that is part of the ground truth output sequence.

10. The method of claim 9 , wherein determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by the possible system output and followed by any suffix of one or more system outputs comprises:

determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by a possible system output that is not a first system output in any of the identified suffixes and followed by any ground truth suffix that is part of the ground truth output sequence;

when the possible system output is not the first system output in any of the identified ground truth suffixes, assigning as the highest quality score for the possible system output the highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by a possible system output that is not a first system output in any of the identified ground truth suffixes and followed by any ground truth suffix that is part of the ground truth output sequence.

11. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a neural network having a plurality of network parameters and being configured to map a system input to an output sequence comprising a plurality of system outputs, the operations comprising:

obtaining a batch of training examples, each training example comprising a training network input and, for each training network input, a ground truth output sequence;

for each of the training examples:

processing the training network input in the training example using the neural network and in accordance with current values of the network parameters to map the training network input to an output sequence comprising a respective system output from a vocabulary of possible system outputs at each of a plurality of output positions;

for each particular output position of the plurality of output positions in the output sequence:

identifying a prefix that includes the system outputs at positions before the particular output position in the output sequence,

for each possible system output in the vocabulary, determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by the possible system output and followed by any suffix of one or more system outputs, wherein the quality score measures a quality of a candidate output sequence relative to the ground truth output sequence, and

determining, using the highest quality scores for the possible system outputs, an update to the current values of the network parameters that increases a likelihood that the neural network generates a system output at the position that has a high quality score; and

determining updated values of the network parameters from the updates for the particular output positions in the output sequences generated by the neural network for the batch of training examples.

12. The system of claim 11 , wherein the neural network is configured to, for each of the output positions, generate a likelihood distribution over the possible system outputs in the vocabulary and select the system output at the output position using the likelihood distribution.

13. The system of claim 12 , wherein determining, using the highest quality scores for the possible system outputs, an update to the current values of the network parameters that increases a likelihood that the neural network generates a system output at the position that has a high quality score comprises:

generating a target likelihood distribution for the output position from the highest quality sores for the possible system outputs,

determining a gradient with respect to the network parameters of an objective function that depends on a divergence between the target likelihood distribution for the output position and the likelihood distribution generated by the neural network for the output position, and

determining the update to the current values using the gradient.

14. The system of claim 13 , wherein generating the target likelihood distribution comprises applying a softmax to the highest quality scores for the possible system outputs to generate a respective likelihood for each of the possible system outputs.

15. The system of claim 14 , wherein the softmax is applied with reduced temperature.

16. The system of claim 11 , wherein the possible system outputs in the vocabulary comprise tokens in a natural language.

17. The system of claim 11 , wherein the quality score is based on the edit distance between the candidate output sequence and the ground truth output sequence, and wherein the highest quality score that can be assigned is the quality score for the candidate output sequence that has a smallest edit distance to the ground truth output sequence.

18. The system of claim 11 , wherein determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by the possible system output and followed by any suffix of one or more system outputs comprises:

determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by any ground truth suffix that is part of the ground truth output sequence;

identifying one or more ground truth prefixes of the ground truth output sequence relative to which the prefix has a highest quality score,

identifying, for each of the identified ground truth prefixes, a respective ground truth suffix that follows the identified ground truth prefix in the ground truth sequence, and

when the possible system output is a first system output in any of the identified ground truth suffixes, assigning as the highest quality score for the possible system output the highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by any ground truth suffix that is part of the ground truth output sequence.

19. The system of claim 11 , wherein determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by the possible system output and followed by any suffix of one or more system outputs comprises:

determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by a possible system output that is not a first system output in any of the identified suffixes and followed by any ground truth suffix that is part of the ground truth output sequence;

when the possible system output is not the first system output in any of the identified ground truth suffixes, assigning as the highest quality score for the possible system output the highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by a possible system output that is not a first system output in any of the identified ground truth suffixes and followed by any ground truth suffix that is part of the ground truth output sequence.

20. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a neural network having a plurality of network parameters and being configured to map a system input to an output sequence comprising a plurality of system outputs, the operations comprising:

obtaining a batch of training examples, each training example comprising a training network input and, for each training network input, a ground truth output sequence;

for each of the training examples:

processing the training network input in the training example using the neural network and in accordance with current values of the network parameters to map the training network input to an output sequence comprising a respective system output from a vocabulary of possible system outputs at each of a plurality of output positions;

for each particular output position of the plurality of output positions in the output sequence:

identifying a prefix that includes the system outputs at positions before the particular output position in the output sequence,

for each possible system output in the vocabulary, determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by the possible system output and followed by any suffix of one or more system outputs, wherein the quality score measures a quality of a candidate output sequence relative to the ground truth output sequence, and

determining, using the highest quality scores for the possible system outputs, an update to the current values of the network parameters that increases a likelihood that the neural network generates a system output at the position that has a high quality score; and

determining updated values of the network parameters from the updates for the particular output positions in the output sequences generated by the neural network for the batch of training examples.