System and method for multi-spoken language detection

1. A method comprising:

performing, using at least one processor, feature extraction of input audio data to identify extracted features associated with the input audio data;

detecting, using the at least one processor, a language associated with each of multiple portions of the input audio data by processing the extracted features using a plurality of language models, each language model associated with a different language, at least two portions of the input audio data associated with at least two different languages; and

directing, using the at least one processor, each portion of the input audio data to one of a plurality of automatic speech recognition (ASR) models based on the language associated with the portion of the input audio data, the at least two portions of the input audio data directed to at least two different ones of the ASR models.

2. The method of claim 1 , wherein:

the language models are distilled from the ASR models; and

the language models use a smaller number of features compared to the ASR models.

3. The method of claim 1 , wherein each language model is used to determine a probability that each portion of the input audio data is from a particular language.

4. The method of claim 3 , wherein detecting the language associated with each portion of the input audio data further comprises:

concatenating the probabilities determined using the language models; and

processing the concatenated probabilities using a neural classification model.

5. The method of claim 1 , further comprising:

processing each portion of the input audio data using the ASR model at which the portion of the input audio data is directed; and

generating text for each portion of the input audio data using the ASR model at which the portion of the input audio data is directed.

6. A method comprising:

performing, using at least one processor, feature extraction of input audio data to identify extracted features associated with the input audio data;

detecting, using the at least one processor, a language associated with each of multiple portions of the input audio data by processing the extracted features using a plurality of language models, each language model associated with a different language, wherein each language model is used to determine a probability that each portion of the input audio data is from a particular language; and

directing, using the at least one processor, each portion of the input audio data to one of a plurality of automatic speech recognition (ASR) models based on the language associated with the portion of the input audio data;

wherein detecting the language associated with each portion of the input audio data comprises:

concatenating the probabilities determined using the language models; and

processing the concatenated probabilities using a neural classification model; and

wherein processing the concatenated probabilities using the neural classification model comprises:

processing the concatenated probabilities using a plurality of temporal convolutional neural networks, wherein outputs from at least one of the temporal convolutional neural networks are normalized;

averaging outputs or normalized outputs from a last of the temporal convolutional neural networks; and

determining probabilities that specific languages are associated with specific portions of the input audio data based on the averaged outputs.

7. The method of claim 6 , wherein detecting the language associated with each portion of the input audio data further comprises:

applying the probabilities that the specific languages are associated with the specific portions of the input audio data to a plurality of sub-network models tuned for transitions between different languages; and

utilizing outputs of the sub-network models as a feature vector for language classification of the specific portions of the input audio data.

8. An electronic device comprising:

at least one memory; and

at least one processor operatively coupled to the at least one memory and configured to:

perform feature extraction of input audio data to identify extracted features associated with the input audio data;

detect a language associated with each of multiple portions of the input audio data by processing the extracted features using a plurality of language models, each language model associated with a different language, at least two portions of the input audio data associated with at least two different languages; and

direct each portion of the input audio data to one of a plurality of automatic speech recognition (ASR) models based on the language associated with the portion of the input audio data such that the at least two portions of the input audio data are directed to at least two different ones of the ASR models.

9. The electronic device of claim 8 , wherein:

the language models are distilled from the ASR models; and

the language models use a smaller number of features compared to the ASR models.

10. The electronic device of claim 8 , wherein the at least one processor is configured to use each language model to determine a probability that each portion of the input audio data is from a particular language.

11. The electronic device of claim 10 , wherein, to detect the language associated with each portion of the input audio data, the at least one processor is further configured to:

concatenate the probabilities determined using the language models; and

process the concatenated probabilities using a neural classification model.

12. The electronic device of claim 8 , wherein the at least one processor is further configured to:

process each portion of the input audio data using the ASR model at which the portion of the input audio data is directed; and

generate text for each portion of the input audio data using the ASR model at which the portion of the input audio data is directed.

13. An electronic device comprising:

at least one memory; and

at least one processor operatively coupled to the at least one memory and configured to:

perform feature extraction of input audio data to identify extracted features associated with the input audio data;

detect a language associated with each of multiple portions of the input audio data by processing the extracted features using a plurality of language models, each language model associated with a different language; and

direct each portion of the input audio data to one of a plurality of automatic speech recognition (ASR) models based on the language associated with the portion of the input audio data;

wherein the at least one processor is configured to use each language model to determine a probability that each portion of the input audio data is from a particular language;

wherein, to detect the language associated with each portion of the input audio data, the at least one processor is configured to:

concatenate the probabilities determined using the language models; and

process the concatenated probabilities using a neural classification model; and

wherein, to process the concatenated probabilities using the neural classification model, the at least one processor is configured to:

process the concatenated probabilities using a plurality of temporal convolutional neural networks and normalize outputs from at least one of the temporal convolutional neural networks;

average outputs or normalized outputs from a last of the temporal convolutional neural networks; and

determine probabilities that specific languages are associated with specific portions of the input audio data based on the averaged outputs.

14. The electronic device of claim 13 , wherein, to detect the language associated with each portion of the input audio data, the at least one processor is further configured to:

apply the probabilities that the specific languages are associated with the specific portions of the input audio data to a plurality of sub-network models tuned for transitions between different languages; and

utilize outputs of the sub-network models as a feature vector for language classification of the specific portions of the input audio data.

15. A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to:

perform feature extraction of input audio data to identify extracted features associated with the input audio data;

detect a language associated with each of multiple portions of the input audio data by processing the extracted features using a plurality of language models, each language model associated with a different language, at least two portions of the input audio data associated with at least two different languages; and

direct each portion of the input audio data to one of a plurality of automatic speech recognition (ASR) models based on the language associated with the portion of the input audio data such that the at least two portions of the input audio data are directed to at least two different ones of the ASR models.

16. The non-transitory machine-readable medium of claim 15 , wherein:

the language models are distilled from the ASR models; and

the language models use a smaller number of features compared to the ASR models.

17. The non-transitory machine-readable medium of claim 15 , further containing instructions that when executed cause the at least one processor to use each language model to determine a probability that each portion of the input audio data is from a particular language.

18. The non-transitory machine-readable medium of claim 17 , wherein the instructions that when executed cause the at least one processor to detect the language associated with each portion of the input audio data further comprise:

instructions that when executed cause the at least one processor to:

concatenate the probabilities determined using the language models; and

process the concatenated probabilities using a neural classification model.

19. The non-transitory machine-readable medium of claim 15 , further containing instructions that when executed cause the at least one processor to:

process each portion of the input audio data using the ASR model at which the portion of the input audio data is directed; and

generate text for each portion of the input audio data using the ASR model at which the portion of the input audio data is directed.

20. A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to:

perform feature extraction of input audio data to identify extracted features associated with the input audio data;

detect a language associated with each of multiple portions of the input audio data by processing the extracted features using a plurality of language models, each language model associated with a different language, each language model used to determine a probability that each portion of the input audio data is from a particular language; and

direct each portion of the input audio data to one of a plurality of automatic speech recognition (ASR) models based on the language associated with the portion of the input audio data;

wherein the instructions that when executed cause the at least one processor to detect the language associated with each portion of the input audio data comprise instructions that when executed cause the at least one processor to:

concatenate the probabilities determined using the language models; and

process the concatenated probabilities using a neural classification model; and

wherein the instructions that when executed cause the at least one processor to process the concatenated probabilities using the neural classification model comprise instructions that when executed cause the at least one processor to:

process the concatenated probabilities using a plurality of temporal convolutional neural networks, wherein outputs from at least one of the temporal convolutional neural networks are normalized;

average outputs or normalized outputs from a last of the temporal convolutional neural networks; and

determine probabilities that specific languages are associated with specific portions of the input audio data based on the averaged outputs.

21. The non-transitory machine-readable medium of claim 20 , wherein the instructions that when executed cause the at least one processor to detect the language associated with each portion of the input audio data further comprise:

instructions that when executed cause the at least one processor to:

apply the probabilities that the specific languages are associated with the specific portions of the input audio data to a plurality of sub-network models tuned for transitions between different languages; and

utilize outputs of the sub-network models as a feature vector for language classification of the specific portions of the input audio data.