Long short-term memory cells with saturating gating functions

1. A system comprising:

a recurrent neural network implemented by one or more computers, wherein the recurrent neural network is configured to receive a respective neural network input at each of a plurality of time steps and to generate a respective neural network output at each of the plurality of time steps, and wherein the recurrent neural network comprises:

a first Long Short-Term Memory (LSTM) cell, wherein the first LSTM cell is configured to, for each of the plurality of time steps, generate a new cell state and a new cell output by applying a plurality of gates to a current cell input, a current cell state, and a current cell output, each of the plurality of gates being configured to, for each of the plurality of time steps:

receive a gate input vector,

generate a respective intermediate gate output vector from the gate input vector in accordance with a respective set of parameters, and

apply a respective gating function to each component of the respective intermediate gate output vector to generate a respective final gate output vector,

wherein the respective gating function for at least one of the plurality of gates is a saturating gating function, and wherein a saturating gating function is a function that has a range that includes at least one of zero or one.

2. The system of claim 1 , wherein at least one of the saturating gating functions g satisfies:

g ( u )=min(0.5,max(−0.5, u ))+0.5,

wherein u is one of the components of an intermediate gate output vector.

3. The system of claim 1 , wherein the plurality of gates includes an input gate, and wherein generating the new cell state comprises:

processing the current cell input and the current cell output using a neural network layer having an activation function that is a saturating squashing function to generate an intermediate cell state update vector;

combining the respective final gate output vector of the input gate and the intermediate cell state update vector to generate a final cell state update vector; and

updating the cell state using the final cell state update vector.

4. The system of claim 3 , wherein the plurality of gates includes a forget gate, and wherein updating the cell state using the final cell state update vector comprises:

combining the respective final gate output vector of the forget gate and the current cell state to generate an intermediate new cell state; and

combining the intermediate cell state and the final cell state update vector to generate the new cell state.

5. The system of claim 3 , wherein a saturating squashing function is a function that maps inputs to a range of −1 to 1, including at least one of −1 or 1.

6. The system of claim 5 , wherein the saturating squashing function s satisfies:

s ( u )=min(1,max(−1, u )),

wherein u is an input to the saturating squashing function.

7. The system of claim 1 , wherein the plurality of gates includes an output gate, and wherein generating the new cell output comprises:

applying a saturating squashing function to each component of the new cell state to generate an intermediate new cell output; and

combining the respective final gate output vector of the output gate and the intermediate new cell output to generate the new cell output.

8. The system of claim 7 , wherein a saturating squashing function is a function that has a range that includes at least one of −1 or 1.

9. The system of claim 8 , wherein the saturating squashing function s satisfies:

s ( u )=min(1,max(−1, u )),

wherein u is an input to the saturating squashing function.

10. The system of claim 1 , wherein the first LSTM cell is configured to concatenate the current cell input and the current cell output to generate the gate input vector.

11. The system of claim 1 , wherein the first LSTM cell is configured to concatenate the current cell input, the current cell state, and the current cell output to generate the gate input vector.

12. One or more non-transitory computer storage media encoded with a computer program product, the computer program product comprising instructions that when executed by one or more computers cause the one or more computers to perform operations to implement:

a recurrent neural network, wherein the recurrent neural network is configured to receive a respective neural network input at each of a plurality of time steps and to generate a respective neural network output at each of the plurality of time steps, and wherein the recurrent neural network comprises:

a first Long Short-Term Memory (LSTM) cell, wherein the first LSTM cell is configured to, for each of the plurality of time steps, generate a new cell state and a new cell output by applying a plurality of gates to a current cell input, a current cell state, and a current cell output, each of the plurality of gates being configured to, for each of the plurality of time steps:

receive a gate input vector,

generate a respective intermediate gate output vector from the gate input vector in accordance with a respective set of parameters, and

apply a respective gating function to each component of the respective intermediate gate output vector to generate a respective final gate output vector,

wherein the respective gating function for at least one of the plurality of gates is a saturating gating function, and wherein a saturating gating function is a function that has a range that includes at least one of zero or one.