Multi-accent speech recognition

1. A computer-implemented method for training a hierarchical recurrent neural network (HRNN) having a plurality of parameters on a plurality of training acoustic sequences to perform automatic speech recognition, wherein the HRNN is configured to generate phoneme representations of received acoustic sequences, the method comprising, for each of the received training acoustic sequences:

processing the received acoustic sequence in accordance with current values of the parameters of the HRNN to generate a predicted grapheme representation of the received acoustic sequence;

processing an intermediate output generated by an intermediate layer of the HRNN during the processing of the received acoustic sequence to generate one or more predicted phoneme representations of the received acoustic sequence;

adjusting the current values of the parameters of the HRNN based on (i) the predicted grapheme representation and (ii) the one or more predicted phoneme representations; and

providing data specifying the trained HRNN for use in performing automatic speech recognition on input utterances.

2. The method of claim 1 , wherein the plurality of training acoustic sequences comprises training acoustic sequences from one or more dialects of a same natural language.

3. The method of claim 1 , wherein the received acoustic sequence comprises a respective acoustic feature representation at each of a plurality of time steps, and

wherein the predicted grapheme representation of the received acoustic sequence comprises a set of grapheme scores for each of the plurality of time steps, the set of grapheme scores for a time step comprising a respective grapheme score for each of a plurality of vocabulary graphemes and a blank character grapheme, the grapheme score for each vocabulary grapheme representing a likelihood that the vocabulary grapheme represents an utterance representing the acoustic sequence at the time step and the score for the blank character grapheme representing a likelihood that the utterance at the time step represents a transition from one word to another word.

4. The method of claim 1 , wherein the received acoustic sequence comprises a respective acoustic feature representation at each of a plurality of time steps, and

wherein a respective phoneme representation for the acoustic feature representation comprises a respective set of phoneme scores for each of the plurality of time steps, the respective set of phoneme scores for a time step comprising a respective phoneme score for each of a plurality of vocabulary phonemes, the phoneme score for each vocabulary phoneme representing a likelihood that the vocabulary phoneme represents an utterance representing the acoustic sequence at the time step.

5. The method of claim 1 , wherein the HRNN comprises (i) one or more recurrent neural network layers, (ii) a grapheme output layer, and (iii) one or more phoneme output layers.

6. The method of claim 5 , wherein the one or more recurrent neural network layers comprise long short term memory (LSTM) neural network layers.

7. The method of claim 6 , wherein the LSTM neural network layers comprise bidirectional LSTM (BLSTM) neural network layers.

8. The method of claim 7 , wherein the BLSTM layers are stacked and at each depth two LSTM layers are fully connected to two LSTM layers at a next adjacent depth.

9. The method of claim 5 , wherein the one or more phoneme output layers comprise softmax output layers.

10. The method of claim 5 , wherein processing the received acoustic sequence in accordance with current values of the parameters of the HRNN to generate a predicted grapheme representation of the received acoustic sequence comprises processing the received acoustic sequence through each of the recurrent neural network layers and grapheme output layer to generate the predicted grapheme representation of the received acoustic sequence.

11. The method of claim 5 , wherein processing an intermediate output generated by an intermediate layer of the HRNN during the processing of the received acoustic sequence to generate one or more predicted phoneme representations of the received acoustic sequence comprises processing the intermediate output using one or more of the phoneme output layers to generate one or more respective predicted phoneme representations of the received acoustic sequence.

12. The method of claim 5 , wherein each of the one or more phoneme output layers is configured to generate as output a respective predicted phoneme representation of the received acoustic sequence corresponding to a different dialect of the same natural language.

13. The method of claim 5 , wherein the (i) grapheme output layer, and (ii) one or more of the phoneme output layers comprise a Connectionist Temporal Classification (CTC) loss function.

14. The method of claim 13 , wherein adjusting the current values of the parameters of the HRNN based on (i) the predicted grapheme representation and (ii) the one or more predicted phoneme representations comprises:

determining a dialect of the received acoustic sequence;

estimating a CTC loss from (i) the grapheme output layer, and (ii) a phoneme output layer corresponding to the determined dialect; and

backpropagating CTC loss gradients from (i) the grapheme output layer, and (ii) the phoneme output layer corresponding to the determined dialect to determine updated values of one or more HRNN parameters.

15. The method of claim 14 , wherein backpropagating CTC loss gradients comprises optimizing a total logarithmic probability of correct grapheme representations for the acoustic sequence.

16. The method of claim 14 , wherein backpropagating CTC loss gradients from (i) the grapheme output layer, and (ii) the phoneme output layer corresponding to the determined dialect comprises:

(i) backpropagating CTC loss gradients from the grapheme output layer to the intermediate recurrent neural network layer, and

(ii) backpropagating a weighted sum of the CTC loss gradients from the grapheme output layer and the CTC loss gradients from the phoneme output layer from the intermediate recurrent neural network layer to a first recurrent neural network layer.

17. The method of claim 16 , wherein the weighted sum is an equally weighted sum.

18. The method of claim 14 , further comprising initializing the one or more HRNN parameters with randomly selected initial values.

19. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a hierarchical recurrent neural network (HRNN) having a plurality of parameters on a plurality of training acoustic sequences to perform automatic speech recognition, wherein the HRNN is configured to generate phoneme representations of received acoustic sequences, the operations comprising, for each of the received training acoustic sequences:

processing the received acoustic sequence in accordance with current values of the parameters of the HRNN to generate a predicted grapheme representation of the received acoustic sequence;

processing an intermediate output generated by an intermediate layer of the HRNN during the processing of the received acoustic sequence to generate one or more predicted phoneme representations of the received acoustic sequence;

adjusting the current values of the parameters of the HRNN based on (i) the predicted grapheme representation and (ii) the one or more predicted phoneme representations; and

providing data specifying the trained HRNN for use in performing automatic speech recognition on input utterances.

20. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a hierarchical recurrent neural network (HRNN) having a plurality of parameters on a plurality of training acoustic sequences to perform automatic speech recognition, wherein the HRNN is configured to generate phoneme representations of received acoustic sequences, the operations comprising, for each of the received training acoustic sequences:

processing the received acoustic sequence in accordance with current values of the parameters of the HRNN to generate a predicted grapheme representation of the received acoustic sequence;

processing an intermediate output generated by an intermediate layer of the HRNN during the processing of the received acoustic sequence to generate one or more predicted phoneme representations of the received acoustic sequence;

adjusting the current values of the para meters of the HRNN based on (i) the predicted grapheme representation and (ii) the one or more predicted phoneme representations; and

providing data specifying the trained HRNN for use in performing automatic speech recognition on input utterances.