Generating acoustic models of alternative pronunciations for utterances spoken by a language learner in a non-native language
A non-transitory processor-readable medium storing code representing instructions to be executed by a processor includes code to cause the processor to receive acoustic data representing an utterance spoken by a language learner in a non-native language in response to prompting the language learner to recite a word in the non-native language and receive a pronunciation lexicon of the word in the non-native language. The pronunciation lexicon includes at least one alternative pronunciation of the word based on a pronunciation lexicon of a native language of the language learner. The code causes the processor to generate an acoustic model of the at least one alternative pronunciation in the non-native language and identify a mispronunciation of the word in the utterance based on a comparison of the acoustic data with the acoustic model. The code causes the processor to send feedback related to the mispronunciation of the word to the language learner.
1. A non-transitory processor-readable medium storing code representing instructions to be executed by a processor, the code comprising code to cause the processor to:
receive acoustic data representing an utterance spoken by a language learner in a non-native language in response to prompting the language learner to recite a word in the non-native language;
determine a pronunciation lexicon of the word in the non-native language including a set of N-best alternative pronunciations of the word determined based on phonological errors in the non-native language associated with a native language of the language learner using a machine translation engine to generate the set of N-best alternative pronunciations, the phonological errors including at least one of a substitution of an incorrect sound, a deletion of a sound and an insertion of a sound;
generate an acoustic model for the N-best pronunciations of the word from the pronunciation lexicon using the machine translation engine, wherein the acoustic model includes phonetic data associated with the phonological errors of the word in the non-native language;
identify a mispronunciation of the word in the utterance based on a comparison of the acoustic data with the acoustic model of the N-best alternative pronunciations, wherein when a first log likelihood of the acoustic data is greater than a first threshold value and when a difference between the first log likelihood and a second log likelihood of the acoustic model is less than a second threshold value, the acoustic model of the N-best alternative pronunciations is used to identify the mispronunciation of the word in the utterance, and wherein when the difference between the first log likelihood and the second log likelihood of the acoustic model is greater than the second threshold value, the acoustic data is used to identify the mispronunciation of the word in the utterance; and
send feedback related to the mispronunciation of the word to the language learner.
2. The non-transitory processor-readable medium of claim 1 , wherein the code to cause the processor to identify includes code to cause the processor to identify grammar data associated with the acoustic data that is different from grammar data associated with the acoustic model to produce a grammar inaccuracy, the feedback including the grammar inaccuracy.
3. The non-transitory processor-readable medium of claim 1 , wherein the set of N-best alternative pronunciations is determined based on a pronunciation lexicon of the native language and phonetically annotated data related to the native language.
4. The non-transitory processor-readable medium of claim 1 , further comprising code to cause the processor to:
generate the pronunciation lexicon of the word in the non-native language based on a pronunciation lexicon of the native language and phonetically annotated data related to the native language.
5. The non-transitory processor-readable medium of claim 1 , further comprising code to cause the processor to:
generate a speech model of the word based on the pronunciation lexicon of the word in the non-native language and the acoustic model,
the code to cause the processor to identify including code to cause the processor to identify the mispronunciation of the word in the utterance based on the speech model and the comparison.
6. The non-transitory processor-readable medium of claim 1 , wherein the code to cause the processor to generate the acoustic model includes code to cause the processor to:
generate a first lattice for the word in the non-native language; and
generate a second lattice for the N-best alternative pronunciations of the word based on phonetically annotated data associated with the word in the non-native language.
7. The non-transitory processor-readable medium of claim 6 , wherein the first lattice and the second lattice are part of a minimum phone error training process used to train the acoustic model.
8. The non-transitory processor-readable medium of claim 1 , wherein the pronunciation lexicon of the word in the non-native language is received from a machine translation module.
9. The non-transitory processor-readable medium of claim 1 , wherein the utterance is a first utterance, the code further comprising code to cause the processor to:
generate an acoustic model for a second pronunciation of the word;
identify the second pronunciation of the word in a second utterance based on a comparison of acoustic data representing the second utterance with the acoustic model for the second pronunciation of the word; and
send feedback related to the second pronunciation of the word to the language learner.
10. The non-transitory processor readable medium of claim 1 , wherein the code to cause the processor to determine the pronunciation lexicon includes code to cause the processor to receive the pronunciation lexicon.
11. The non-transitory processor readable medium of claim 1 , wherein the phonological error includes at least one of a deletion of a sound and an insertion of a sound.
12. The non-transitory processor-readable medium of claim 1 , wherein the phonological error includes one of a possible substitution, a deletion, or an insertion of at least one sound that a language learner that speaks the native language is expected to perform when reciting the word in the non-native language.
13. The non-transitory processor-readable medium of claim 1 , wherein the word is from a plurality of words, the acoustic model is generated based on phonetically annotated data associated with a native recitation of each word from the plurality of words.
14. The non-transitory processor-readable medium of claim 1 , wherein the code to cause the processor to generate the acoustic model includes code to cause the processor to generate the acoustic model prior to receiving the acoustic data.
15. A method, comprising:
receiving acoustic data representing an utterance spoken by a language learner in a non-native language in response to prompting the language learner to recite a word in the non-native language;
generating a set of N-best alternative pronunciations of the word based on phonological errors in the non-native language associated with a native language of the language learner using a machine translation engine to generate the set of N-best alternative pronunciations, the phonological errors including at least one of a substitution of an incorrect sound, a deletion of a sound and an insertion of a sound;
generating an acoustic model for the set of N-best alternative pronunciations of the word using the machine translation engine, wherein the acoustic model includes phonetic data associated with the phonological errors of the word in the non-native language;
identifying a mispronunciation of the word in the utterance in response to a speech recognition engine recognizing the acoustic data being similar to one of the N-best alternative pronunciations of the word that is included in the acoustic models, wherein when a first log likelihood of the acoustic data is greater than a first threshold value and when a difference between the first log likelihood and a second log likelihood of the acoustic model is less than a second threshold value, the acoustic model of the N-best alternative pronunciations is used to identify the mispronunciation of the word in the utterance, and wherein when the difference between the first log likelihood and the second log likelihood of the acoustic model is greater than the second threshold value, the acoustic data is used to identify the mispronunciation of the word in the utterance; and
sending feedback related to the mispronunciation of the word to the language learner in response to the identifying.
16. The method of claim 15 , when the utterance is a first utterance, the method further comprising:
generating an acoustic model for a second pronunciation of the word;
identifying the second pronunciation of the word in a second utterance of the word in response to the speech recognition engine recognizing the acoustic data representing the second utterance as part of the acoustic model for the second pronunciation of the word; and
sending feedback related to the second pronunciation of the word to the language learner in response to the identifying the second pronunciation of the word.
17. The method of claim 15 , further comprising:
identifying grammar data associated with the acoustic data that is different from grammar data associated with the acoustic model to produce a grammar inaccuracy, the feedback including the grammar inaccuracy.
18. The method of claim 15 , wherein the generating the acoustic model includes:
generating a first lattice for the word in the non-native language; and
generating a second lattice for the N-best alternative pronunciations of the word based on the phonetically annotated data with the native pronunciation of the word.
19. The method of claim 18 , wherein the first lattice and the second lattice are part of a minimum phone error training process used to train the acoustic model.
20. The method of claim 15 , further comprising:
generating a pronunciation lexicon of the word in the non-native language, the pronunciation lexicon of the word including a set of alternative pronunciations of the word; and
generating a speech model of the word based on the pronunciation lexicon of the word in the non-native language and the acoustic model, the identifying including identifying the mispronunciation of the word in the utterance in response to the speech recognition engine recognizing the acoustic data as part of the acoustic model and the speech model.
21. The method of claim 15 , wherein the acoustic model is generated based on phonetically annotated data.
22. A method, comprising:
receiving a phrase having a plurality of words from a language learning module in response to the language learning module prompting a language learner to recite the phrase in a non-native language, the language learner having a native language;
generating a set of N-best alternative phrases based on phonological errors in the non-native language associated with the native language using a machine translation engine, the phonological errors including at least one of a substitution of an incorrect sound, a deletion of a sound and an insertion of a sound;
generating an acoustic model using the machine translation engine for each of the N-best alternative phrases based on phonetically annotated data associated with a native recitation of each word from the plurality of words in the phrase;
identifying that the language learner recited one of the N-best alternative phrases based on a comparison of the acoustic model and acoustic data representing an utterance spoken by the language learner in response to the language learning module prompting the language learner to recite the phrase in the non-native language;
identifying at least one word from the plurality of words in the phrase that was incorrectly recited by the language learner, wherein when a first log likelihood of the at least one word in the phrase recited by the language learner is greater than a first threshold value and when a difference between the first log likelihood and a second log likelihood of the at least one word in the phrase in the acoustic model is less than a second threshold value, the at least one word in the phrase in the acoustic model is used to identify an incorrectly recited word, and wherein when the difference between the first log likelihood and the second log likelihood of the at least one word in the phrase in the acoustic model is greater than the second threshold value, the at least one word in the phase recited by the language learner is used to identify the incorrectly recited word; and
sending feedback to the language learner associated with the at least one word.
23. The method of claim 22 , wherein the generating the set of N-best alternative phrases is based on a lexicon of the native language and phonetically annotated data associated with the native recitation of each word from the plurality of words in the phrase.
24. The method of claim 22 , wherein the generating the set of N-best alternative phrases is based on a lexicon of the native language and phonetically annotated data associated with a native pronunciation of each word from the plurality of words.
25. The method of claim 22 , wherein the alternative phrase from the set of N-best alternative phrases includes at least one grammatical inaccuracy associated with the native recitation of the phrase.
26. The method of claim 22 , wherein the generating the acoustic model includes generating a first lattice for each word from the plurality of words in the phrase, and generating a second lattice for each of the set of N-best alternative phrases.