Speech recognition using neural networks
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for speech recognition using neural networks. A feature vector that models audio characteristics of a portion of an utterance is received. Data indicative of latent variables of multivariate factor analysis is received. The feature vector and the data indicative of the latent variables is provided as input to a neural network. A candidate transcription for the utterance is determined based on at least an output of the neural network.
1. A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising:
while a user is speaking a current utterance:
obtaining feature vectors indicative of audio characteristics of corresponding portions of the current utterance that have been spoken by the user;
for each predetermined duration of new audio received that characterizes a corresponding portion of the current utterance, calculating a corresponding i-vector;
providing, as input to a neural network acoustic model, the feature vectors and the corresponding i-vector calculated for each predetermined duration of new audio received; and
based on the feature vectors and the corresponding i-vector calculated for each predetermined duration of new audio received and provided as input to the neural network acoustic model, determining, as output from an output layer of the neural network acoustic model, a posterior probability distribution of possible speech units representing each feature vector.
2. The computer-implemented method of claim 1 , wherein the corresponding i-vector is calculated using a Gaussian mixture model (GMM).
3. The computer-implemented method of claim 2 , wherein the GMM comprises 39-dimensional Gaussians.
4. The computer-implemented method of claim 1 , wherein the speech units in the posterior probability distribution of possible speech units comprise components of phones.
5. The computer-implemented method of claim 1 , wherein the neural network acoustic model comprises an input layer, multiple hidden layers, and the output layer.
6. The computer-implemented method of claim 1 , wherein the operations further comprise determining a candidate transcription for the current utterance based on the posterior probability distribution of possible speech units representing each feature vector.
7. The computer-implemented method of claim 1 , wherein the operations further comprise:
providing, as input to a hidden markov model (HMM), the posterior probability distribution of possible speech units output from the output layer of the neural network acoustic model; and
determining, as output from the HMM, a word lattice associated with a transcription of the utterance.
8. The computer-implemented method of claim 7 , wherein the HMM is approximated by a set of weighted finite state transducers that represent a language model.
9. The computer-implemented method of claim 1 , wherein the corresponding i-vector calculated for each predetermined duration of new audio received indicates time-independent audio characteristics.
10. The computer-implemented method of claim 1 , wherein the operations further comprise projecting the corresponding i-vector calculated for each predetermined duration of new audio received using linear discriminant analysis (LDA).
11. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware causes the data processing hardware to perform operations comprising:
while a user is speaking a current utterance:
obtaining feature vectors indicative of audio characteristics of corresponding portions of the current utterance that have been spoken by the user;
for each predetermined duration of new audio received that characterizes a corresponding portion of the current utterance, calculating a corresponding i-vector;
providing, as input to a neural network acoustic model, the feature vectors and the corresponding i-vector calculated for each predetermined duration of new audio received; and
based on the feature vectors and the corresponding i-vector calculated for each predetermined duration of new audio received and provided as input to the neural network acoustic model, determining, as output from an output layer of the neural network acoustic model, a posterior probability distribution of possible speech units representing each feature vector.
12. The system of claim 11 , wherein the corresponding i-vector is calculated using a Gaussian mixture model (GMM).
13. The system of claim 11 , wherein the GMM comprises 39-dimensional Gaussians.
14. The system of claim 11 , wherein the speech units in the posterior probability distribution of possible speech units comprise components of phones.
15. The system of claim 11 , wherein the neural network acoustic model comprises an input layer, multiple hidden layers, and the output layer.
16. The system of claim 11 , wherein the operations further comprise determining a candidate transcription for the current utterance based on the posterior probability distribution of possible speech units representing each feature vector.
17. The system of claim 11 , wherein the operations further comprise:
providing, as input to a hidden markov model (HMM), the posterior probability distribution of possible speech units output from the output layer of the neural network acoustic model; and
determining, as output from the HMM, a word lattice associated with a transcription of the utterance.
18. The system of claim 17 , wherein the HMM is approximated by a set of weighted finite state transducers that represent a language model.
19. The system of claim 11 , wherein the corresponding i-vector calculated for each predetermined duration of new audio received indicates time-independent audio characteristics.
20. The system of claim 11 , wherein the operations further comprise projecting the corresponding i-vector calculated for each predetermined duration of new audio received using linear discriminant analysis (LDA).