Deep learning models for speech recognition
Presented herein are embodiments of state-of-the-art speech recognition systems developed using end-to-end deep learning. In embodiments, the model architecture is significantly simpler than traditional speech systems, which rely on laboriously engineered processing pipelines; these traditional systems also tend to perform poorly when used in noisy environments. In contrast, embodiments of the system do not need hand-designed components to model background noise, reverberation, or speaker variation, but instead directly learn a function that is robust to such effects. Neither a phoneme dictionary, nor even the concept of a “phoneme,” is needed. Embodiments include a well-optimized recurrent neural network (RNN) training system that can use multiple GPUs, as well as a set of novel data synthesis techniques that allows for a large amount of varied data for training to be efficiently obtained. Embodiments of the system can also handle challenging noisy environments better than widely used, state-of-the-art commercial speech systems.
1. A computer-implemented method for training a transcription neural network, the method comprising:
inputting an utterance that comprises a set of spectrogram frames covering time steps of the utterance into a first layer of the transcription neural network that evaluates, for each time step of a set of time steps, a spectrogram frame from the set of spectrogram frames and an associated context of one or more spectrogram frames;
obtaining predicted character probabilities for the utterance from the transcription neural network;
using the predicted character probabilities for the utterance and a corresponding ground truth transcription for the utterance to determine a loss in predicting the corresponding ground truth transcription for the utterance; and
updating one or more parameters of the transcription neural network using a gradient based upon the loss in predicating the utterance.
2. The computer-implemented method of claim 1 further comprising:
jittering the utterances prior to inputting into the transcription neural network.
3. The computer-implemented method of claim 2 wherein the step of jittering the utterances prior to inputting into the transcription neural network comprises:
generating a jitter set of utterances for the utterance by translating an audio file of the utterance by one or more time values;
converting the jitter set of utterances and the utterance into a set of spectrograms;
obtaining output results from the transcription neural network for the set of spectrograms; and
averaging the output results for the set of spectrograms to obtain an output for the utterance.
4. The computer-implemented method of claim 1 further comprising:
generating one or more utterances for a set of training data for use in training the transcription neural network.
5. The computer-implemented method of claim 4 wherein generating one or more utterances for a set of training data for use in training the transcription neural network comprises:
having a person wear headphones as the person records an utterance;
intentionally inducing a Lombard effect during data collection of the utterance by playing background noise through the headphones worn by the person; and
capturing the Lombard-effected utterance of the person via a microphone without capturing the background noise.
6. The computer-implemented method of claim 4 wherein generating one or more utterances for a set of training data for use in training the transcription neural network comprises:
adding one or more noise clips selected from a set of approved noise clips to form a noise track;
adjusting a signal-to-noise ratio of the noise track relative to an audio file;
adding the adjusted noise track to the audio file to form a synthesized noise audio file; and
adding the synthesized noise audio file to the set of training data.
7. The computer-implemented method of claim 6 wherein the set of approved noise clips are generated by performing the steps comprising:
collecting a set of candidate noise clips; and
repeating the following steps until each noise clip from the set of candidate noise clips has been evaluated:
selecting a candidate noise clip from the set of candidate noise clips for evaluation;
responsive to the candidate noise clip's average powers in frequency bands not differing significantly from average powers in frequency bands observed in real noise recordings, adding the candidate noise clip to the set of approved noise clips; and
responsive to the candidate noise clip's average powers in frequency bands differing significantly from average powers observed in real noise recordings, discarding the candidate noise clip.
8. The computer-implemented method of claim 1 further comprising using data parallelism by performing the steps comprises:
using several copies of the transcription neural network across multiple processing units with each processing unit processing a separate minibatch of utterances; and
combining a computed gradient from a processing unit with its peers during each iteration.
9. The computer-implemented method of claim 1 further comprising using data parallelism by performing the steps comprises:
having each processing unit process many utterances in parallel by concatenating many utterances into a single matrix; and
sorting utterances by length and combining similarly-sized utterances into minibatches and padding utterances with silence so that all utterances in a minibatch have the same length.
10. A computer-implemented method for transcribing speech comprising:
generating a set of spectrogram frames for an input audio;
inputting the set of spectrogram frames into a transcription neural network;
obtaining predicted character probabilities outputs from the transcription neural network; and
decoding a predicted transcription of the input audio using the predicted character probabilities outputs from the transcription neural network constrained by a language model that interprets a string of characters from the predicted character probabilities outputs as a word or words.
11. The computer-implemented method of claim 10 further comprising:
using the predicted transcription for the input audio and a corresponding ground truth transcription for the input audio to determine a loss in predicting the corresponding ground truth transcription for the input audio; and
updating one or more parameters of the transcription neural network using the loss in predicting the corresponding ground truth transcription for the input audio.
12. The computer-implemented method of claim 10 wherein the step of generating a set of spectrogram frames from the input audio comprises:
generating a set of spectrogram frames from a normalized version of the input audio or from a normalized and jitter version of the input audio.
13. The computer-implemented method of claim 10 wherein the step of inputting the set of spectrogram frames into a transcription neural network comprises:
inputting the set of spectrogram frames into the transcription neural network in which at least one layer of the transcription neural network operates on a context of spectrogram frames from the set of spectrogram frames.
14. The computer-implemented method of claim 10 wherein the step of decoding a predicted transcription of the input audio using the predicted character probabilities outputs from the transcription neural network constrained by a language model that interprets a string of characters from the predicted character probabilities outputs as a word or words comprises:
given the predicted character probabilities outputs from the transcription neural network, performing a search to find a sequence of characters that is most probable according to both the predicted character probabilities outputs and an N-gram language model output that interprets a string of characters from the predicted character probabilities outputs as a word or words.
15. The computer-implemented method of claim 10 wherein the transcription neural network comprises:
a first set of three layers that are non-recurrent;
a fourth layer that is a bi-directional recurrent network, which includes two sets of hidden units comprising a set with forward recurrence and a set with backward recurrence; and
a fifth layer that is a non-recurrent layer, which takes forward and backward units from the fourth layer as inputs and outputs the predicted character probabilities.
16. A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by one or more processors, causes steps to be performed comprising:
generating a set of spectrogram frames for an input audio;
inputting the set of spectrogram frames into a transcription neural network;
obtaining predicted character probabilities outputs from the transcription neural network; and
decoding a predicted transcription of the input audio using the predicted character probabilities outputs from the transcription neural network constrained by a language model that interprets a string of characters from the predicted character probabilities outputs as a word or words.
17. The non-transitory computer-readable medium or media of claim 16 further comprising one or more sequences of instructions which, when executed by one or more processors, causes steps to be performed comprising:
using the predicted transcription for the input audio and a corresponding ground truth transcription for the input audio to determine a loss in predicting the corresponding ground truth transcription for the input audio; and
updating one or more parameters of the transcription neural network using the loss in predicting the corresponding ground truth transcription for the input audio.
18. The non-transitory computer-readable medium or media of claim 16 wherein the step of inputting the set of spectrogram frames into a transcription neural network comprises:
inputting the set of spectrogram frames into the transcription neural network in which at least one layer of the transcription neural network operates on a context of spectrogram frames from the set of spectrogram frames.
19. The non-transitory computer-readable medium or media of claim 16 wherein the step of decoding a predicted transcription of the input audio using the predicted character probabilities outputs from the transcription neural network constrained by a language model that interprets a string of characters from the predicted character probabilities outputs as a word or words comprises:
given the predicted character probabilities outputs from the transcription neural network, performing a search to find a sequence of characters that is most probable according to both the predicted character probabilities outputs and an N-gram language model output that interprets a string of characters from the predicted character probabilities outputs as a word or words.
20. The non-transitory computer-readable medium or media of claim 16 wherein the transcription neural network comprises:
a first set of three layers that are non-recurrent;
a fourth layer that is a bi-directional recurrent network, which includes two sets of hidden units comprising a set with forward recurrence and a set with backward recurrence; and
a fifth layer that is a non-recurrent layer, which takes forward and backward units from the fourth layer as inputs and outputs the predicted character probabilities.