Recurrent neural networks for online sequence generation
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a target sequence from a source sequence. In one aspect, the system includes a recurrent neural network configured to, at each time step, receive am input for the time step and process the input to generate a progress score and a set of output scores; and a subsystem configured to, at each time step, generate the recurrent neural network input and provide the input to the recurrent neural network; determine, from the progress score, whether or not to emit a new output at the time step; and, in response to determining to emit a new output, select an output using the output scores and emit the selected output as the output at a next position in the output order.
1. A system for generating a target sequence comprising a plurality of outputs arranged according to an output order from a source sequence comprising a plurality of inputs arranged according to an input order, the system comprising:
a recurrent neural network, wherein the recurrent neural network is configured to, at each of a plurality of time steps:
receive a recurrent neural network input for the time step comprising a current input for the time step and a most recently emitted output from the target sequence; and
process the recurrent neural network input to generate (i) a progress score and (ii) a set of output scores; and
a subsystem, wherein the subsystem is configured to, at each of the plurality of time steps:
generate the recurrent neural network input for the time step and provide the recurrent neural network input as input to the recurrent neural network;
determine, from the progress score, whether or not to emit a new output at the time step; and
in response to determining to emit a new output:
select an output using the output scores; and
emit the selected output as the output at a next position in the output order.
2. The system of claim 1 , wherein the recurrent neural network is a Grid Long Short-Term Memory (LSTM) neural network.
3. The system of claim 1 , wherein the recurrent neural network is a stacked deep LSTM neural network.
4. The system of claim 1 , wherein the recurrent neural network input further comprises a progress input indicating whether or not an output was emitted at a time step immediately preceding the time step.
5. The system of claim 1 , wherein the source sequence comprises k inputs, wherein k is an integer equal to or greater than one, wherein the plurality of time steps are ordered according to a time step order, and wherein, when i is less than or equal to k, the current input for an i-th time step in the time step order is the i-th input in the input order.
6. The system of claim 5 , wherein when i is greater than k, the current input for an i-th time step in the time step order is a last input in the input order.
7. The system of claim 5 , wherein when i is greater than k, the current input for an i-th time step in the time step order is a placeholder input.
8. The system of claim 1 , wherein determining, from the progress score, whether or not to emit a new output at the time step comprises:
sampling from a two-point distribution in accordance with the progress score to select either an emit value or a do not emit value for a binary variable; and
determining to emit a new output when the emit value is selected and determining not to emit a new output when the do not emit value is selected.
9. The system of claim 1 , wherein the set of output scores includes a respective score for each output in a vocabulary of possible outputs and a score for an end of sequence output, and wherein selecting an output comprises selecting an output having a highest output score.
10. The system of claim 9 , wherein the subsystem is further configured to terminate the output sequence if the selected output is the end of sequence output.
11. A method for generating a target sequence comprising a plurality of outputs arranged according to an output order from a source sequence comprising a plurality of inputs arranged according to an input order, the method comprising, at each of a plurality of time steps:
generating a recurrent neural network input for the time step and providing the recurrent neural network input as input to a recurrent neural network, wherein the recurrent neural network input for the time step comprises a current input for the time step and a most recently emitted output from the target sequence, and wherein the recurrent neural network is configured to, at each of the plurality of time steps:
receive the recurrent neural network input for the time step; and
process the recurrent neural network input to generate (i) a progress score for the time step and (ii) a set of output scores for the time step;
determining, from the progress score for the time step, whether or not to emit a new output at the time step; and
in response to determining to emit a new output at the time step:
selecting an output using the output scores; and
emitting the selected output as the output at a next position in the output order.
12. The method of claim 11 , wherein the recurrent neural network is a Grid Long Short-Term Memory (LSTM) neural network.
13. The method of claim 11 , wherein the recurrent neural network is a stacked deep LSTM neural network.
14. The method of claim 11 , wherein the recurrent neural network input further comprises a progress input indicating whether or not an output was emitted at a time step immediately preceding the time step.
15. The method of claim 11 , wherein the source sequence comprises k inputs, wherein k is an integer equal to or greater than one, wherein the plurality of time steps are ordered according to a time step order, and wherein, when i is less than or equal to k, the current input for an i-th time step in the time step order is the i-th input in the input order.
16. The method of claim 15 , wherein when i is greater than k, the current input for an i-th time step in the time step order is a last input in the input order.
17. The method of claim 15 , wherein when i is greater than k, the current input for an i-th time step in the time step order is a placeholder input.
18. The method of claim 11 , wherein determining, from the progress score, whether or not to emit a new output at the time step comprises:
sampling from a two-point distribution in accordance with the progress score to select either an emit value or a do not emit value for a binary variable; and
determining to emit a new output when the emit value is selected and determining not to emit a new output when the do not emit value is selected.
19. The method of claim 11 , wherein the set of output scores includes a respective score for each output in a vocabulary of possible outputs and a score for an end of sequence output, and wherein selecting an output comprises selecting an output having a highest output score.
20. One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for generating a target sequence comprising a plurality of outputs arranged according to an output order from a source sequence comprising a plurality of inputs arranged according to an input order, the operations comprising, at each of a plurality of time steps:
generating a recurrent neural network input for the time step and providing the recurrent neural network input as input to a recurrent neural network, wherein the recurrent neural network input for the time step comprises a current input for the time step and a most recently emitted output from the target sequence, and wherein the recurrent neural network is configured to, at each of the plurality of time steps:
receive the recurrent neural network input for the time step; and
process the recurrent neural network input to generate (i) a progress score for the time step and (ii) a set of output scores for the time step;
determining, from the progress score for the time step, whether or not to emit a new output at the time step; and
in response to determining to emit a new output at the time step:
selecting an output using the output scores; and
emitting the selected output as the output at a next position in the output order.
21. The computer-readable storage media of claim 20 , wherein the recurrent neural network input further comprises a progress input indicating whether or not an output was emitted at a time step immediately preceding the time step.
22. The computer-readable storage media of claim 20 , wherein the source sequence comprises k inputs, wherein k is an integer equal to or greater than one, wherein the plurality of time steps are ordered according to a time step order, and wherein, when i is less than or equal to k, the current input for an i-th time step in the time step order is the i-th input in the input order.
23. The computer-readable storage media of claim 22 , wherein when i is greater than k, the current input for an i-th time step in the time step order is a last input in the input order.
24. The computer-readable storage media of claim 22 , wherein when i is greater than k, the current input for an i-th time step in the time step order is a placeholder input.
25. The computer-readable storage media of claim 20 , wherein determining, from the progress score, whether or not to emit a new output at the time step comprises:
sampling from a two-point distribution in accordance with the progress score to select either an emit value or a do not emit value for a binary variable; and
determining to emit a new output when the emit value is selected and determining not to emit a new output when the do not emit value is selected.
26. The computer-readable storage media of claim 20 , wherein the set of output scores includes a respective score for each output in a vocabulary of possible outputs and a score for an end of sequence output, and wherein selecting an output comprises selecting an output having a highest output score.
27. The computer-readable storage media of claim 20 , the operations further comprising terminating the output sequence if the selected output is the end of sequence output.
28. The method of claim 11 , further comprising terminating the output sequence if the selected output is the end of sequence output.