Code-switching speech recognition with end-to-end connectionist temporal classification model

1. A system for automatic speech recognition associated with a major language and a secondary language, comprising:

a computer processor; and

a memory storage device including instructions that when executed by the computer processor enable the system to:

during a decoding process for each of a series of frames associated with a speech waveform input, if silence dominates a current frame then emit a silence output token; and

if silence does not dominate the current frame, then:

multiply a major language output token posterior vector constructed from posteriors of major language tokens from a Code-Switching (“CS”) Connectionist Temporal Classification (“CTC”) model with a Language Identification (“LID”) model major language probability to create a probability vector from the major language,

wherein the CS CTC model was: (1) initialized from the major language CTC model by keeping network hidden weights and replacing output tokens of the major language CTC model with a union of major language output tokens, secondary language output tokens, and the silence output token and (2) the initialized CTC model was trained by updating parameters with training data from both the major language and the secondary language, and

further wherein the LID model was trained with the training data in connection with three-frame-by-frame outputs: (1) the major language probability, (2) a secondary language probability, and (3) a silence probability,

multiply a secondary language output token posterior vector from the CS CTC model with the LID secondary language probability to create a probability vector from the secondary language, and

emit the output token associated with the highest probability across all tokens from the major and secondary language.

2. The system of claim 1 , further comprising instructions that when executed by the computer processor enable the system to:

collapse the emitted output tokens using greedy decoding to generate an automatic speech recognition decoding hypothesis.

3. The system of claim 2 , wherein the greedy decoding removes silence output tokens and repetitive language output tokens.

4. The system of claim 1 , wherein the CTC model comprises bidirectional Long Short-Term Memory (“LSTM”) Recurrent Neural Networks (“RNNs”).

5. The system of claim 4 , wherein an objective function for the CTC model is CTC loss.

6. The system of claim 1 , wherein the LID model is associated with feed-forward Deep Neural Networks (“DNNs”) and Long Short-Term Memories (“LSTMs”) to build a frame-level LID model.

7. The system of claim 6 , wherein the LID model utilizes information from a context window that includes data external to the current frame.

8. A computer-implemented method for automatic speech recognition associated with a first language and a second language, comprising:

during a decoding process for each of a series of frames associated with a speech waveform input, if silence dominates a current frame, emitting a silence output token; and

if silence does not dominate the current frame:

multiplying a first language output token posterior vector constructed from posteriors of major language tokens from a Code-Switching (“CS”) Connectionist Temporal Classification (“CTC”) model with a Language Identification (“LID”) model first language probability to create a probability vector from the first language,

wherein the CS CTC model was: (1) initialized from the major language CTC model by keeping network hidden weights and replacing output tokens of the major language CTC model with a union of major language output tokens, secondary language output tokens, and the silence output token and (2) the initialized CTC model was trained by updating parameters with training data from both the major language and the secondary language, and

further wherein the LID model was trained with the training data in connection with three frame-by-frame outputs: (1) the major language probability, (2) a secondary language probability, and (3) a silence probability,

multiplying a second language output token posterior vector from the CS CTC model with the LID second language probability to create a probability vector from the second language, and

emitting the output token associated with the highest probability across all tokens from the major and secondary language.

9. The method of claim 8 , further comprising:

collapsing the emitted output tokens using greedy decoding to generate an automatic speech recognition decoding hypothesis.

10. The method of claim 9 , wherein the greedy decoding removes silence output tokens and repetitive language output tokens.

11. The method of claim 8 , wherein the CTC model comprises bidirectional Long Short-Term Memory (“LSTM”) Recurrent Neural Networks (“RNNs”).

12. The method of claim 11 , wherein an objective function for the CTC model is CTC loss.

13. The method of claim 8 , wherein the LID model is associated with feed-forward Deep Neural Networks (“DNNs”) and Long Short-Term Memories (“LSTMs”) to build a frame-level LID model.

14. The method of claim 13 , wherein the LID model utilizes information from a context window that includes data external to the current frame.

15. A non-transient, computer-readable medium storing instructions to be executed by a processor to perform a method for automatic speech recognition associated with a major language and a secondary language, the method comprising:

during a decoding process for each of a series of frames associated with a speech waveform input, if silence dominates a current frame, emitting a silence output token;

if silence does not dominate the current frame:

multiplying a first language output token posterior vector constructed from posteriors of major language tokens from a Code-Switching (“CS”) Connectionist Temporal Classification (“CTC”) model with a Language Identification (“LID”) model first language probability to create a probability vector from the first language,

wherein the CS CTC model was: (1) initialized from the major language CTC model by keeping network hidden weights and replacing output tokens of the major language CTC model with a union of major language output tokens, secondary language output tokens, and the silence output token and (2) the initialized CTC model was trained by updating parameters with training data from both the major language and the secondary language, and

further wherein the LID model was trained with the training data in connection with three frame-by-frame outputs: (1) the major language probability, (2) a secondary language probability, and (3) a silence probability,

multiplying a second language output token posterior vector from the CS CTC model with the LID second language probability to create a probability vector from the second language, and

emitting the output token associated with the highest probability across all tokens from the first and second language; and

collapsing the emitted output tokens using greedy decoding to generate an automatic speech recognition decoding hypothesis.

16. The medium of claim 15 , wherein the greedy decoding removes silence output tokens and repetitive language output tokens.

17. The medium of claim 15 , wherein the CTC model comprises bidirectional Long Short-Term Memory (“LSTM”) Recurrent Neural Networks (“RNNs”).

18. The medium of claim 17 , wherein an objective function for the CTC model is CTC loss.

19. The medium of claim 15 , wherein the LID model is associated with feed-forward Deep Neural Networks (“DNNs”) and Long Short-Term Memories (“LSTMs”) to build a frame-level LID model.

20. The medium of claim 19 , wherein the LID model utilizes information from a context window that includes data external to the current frame.