Systems and methods for assessment of non-native spontaneous speech
Computer-implemented systems and methods are provided for assessing non-native spontaneous speech pronunciation. Speech recognition on digitized speech is performed using a non-native acoustic model trained with non-native speech to generate word hypotheses for the digitized speech. Time alignment is performed between the digitized speech and the word hypotheses using a reference acoustic model trained with native-quality speech. Statistics are calculated regarding individual words and phonemes in the word hypotheses based on the alignment. A plurality of features for use in assessing pronunciation of the speech are calculated based on the statistics, an assessment score is calculated based on one or more of the calculated features, and the assessment score is stored in a computer-readable memory.
1. A computer-implemented method of assessing spontaneous speech pronunciation, comprising:
performing speech recognition on digitized speech using a non-native acoustic model trained with non-native speech using a processor to generate word hypotheses for the digitized speech;
performing time alignment between the digitized speech and the word hypotheses utilizing a reference acoustic model trained with native-quality speech to associate the word hypotheses with corresponding sounds of the digitized speech;
calculating statistics regarding individual words and phonemes of the word hypotheses using the processor based on said alignment;
calculating a plurality of features for use in assessing pronunciation of the digitized speech based on the statistics using the processor;
calculating an assessment score based on one or more of the calculated features; and
storing the assessment score in a computer-readable memory.
2. The method of claim 1 further comprising excluding words not reliably recognized in generating the word hypotheses from contributing to the assessment score.
3. The method of claim 2 , wherein a confidence level is associated with each word in the word hypotheses identifying a likelihood that a word in a word hypothesis was correctly recognized in generating the word hypotheses.
4. The method of claim 3 , wherein words having corresponding word hypotheses that do not meet a confidence threshold are not considered in calculating the assessment score.
5. The method of claim 1 wherein the features are based on at least one of: Hidden Markov Model probabilities, average phoneme duration, average word duration, phoneme duration distribution, word duration distribution, energy measurements, energy distributions, pitch measurements, pitch distributions, and pitch contours.
6. The method of claim 1 wherein speech samples are scored by a human, and a statistical model is built using the features and human scores;
wherein the assessment score is based on the scoring model and one or more of the calculated features.
7. The method of claim 6 , wherein the statistical model is built using multiple regression.
8. The method of claim 1 , wherein the reference acoustic model is trained with only native speech.
9. The method of claim 1 , wherein generating word hypotheses utilizes the non-native acoustic model, a dictionary that maps pronunciations to words, and a language model that identifies a likelihood that a word will follow a sequence of already hypothesized words in a speech recording.
10. The method of claim 9 , wherein the language model is an n-gram language model.
11. The method of claim 1 , wherein the assessment score is based on one or more features selected from the group consisting of:
an average likelihood across all letters: L 1 /m, where L 1 is a summation of likelihoods of all individual words:
L
1
=
∑
i
=
1
n
L
(
x
i
)
,
where L(x i ) is a likelihood of word x i being spoken given an observed audio signal, where n is a number of words in a response, where m is a number of letters in the response;
an average likelihood across all words: L 1 /n;
an average likelihood per second normalized by a rate of speech: L 4 /R, where L 4 =L 1 /T, where T is the summation of a duration of all words in the response, where R=m/T s , where T s is a duration of the response;
an average likelihood density across all words normalized by a rate of speech, L 5 /R, where
L
5
=
∑
i
=
1
n
L
(
x
i
)
t
i
n
,
where t i is a duration of word i in a response;
an average vowel duration shift:
S
_
=
∑
i
=
1
N
v
S
v
i
N
v
,
where N v is a total number of vowels, where S v i is the duration shift of vowel v i , which is measured as an absolute value of the difference between a duration of vowel v i and a standard value of a duration of a vowel estimated on native speech data, where S v i =|P v i −D v i |, where P v i is the duration of vowel v i and D v i is the standard average duration of vowel v i ; and
an average normalized vowel duration shifts:
S
_
n
=
∑
i
=
1
N
v
Sn
v
i
N
v
,
where Sn v i is a normalized duration shift of vowel v i , which is measured as an absolute value of a normalized difference between a duration of vowel v i and a standard value of a duration of a vowel estimated on the native speech data, where Sn v i is calculated as
P
v
i
P
-
D
v
i
D
,
where P v i is the duration of vowel v i , P is an average vowel duration across all vowels in the response being scored, D v i is a standard average duration of vowel v i , and D is a standard vowel duration estimated on all vowels in native speech data.
12. The method of claim 1 , wherein one or more of the features are utilized with related stress, intonation, vocabulary, or grammar to generate the assessment score indicating communicative competence or other construct of speaking proficiency that includes pronunciation proficiency.
13. The method of claim 1 , wherein second statistics derived from native-quality speech are utilized in calculating the plurality of features.
14. A computer-implemented system for assessing spontaneous speech pronunciation, comprising:
a processor;
a non-transitory computer-readable memory comprising instructions for causing the processor to perform steps including:
performing speech recognition on digitized speech using a non-native acoustic model trained with non-native speech to generate word hypotheses for the digitized speech;
performing time alignment between the digitized speech and the word hypotheses utilizing a reference acoustic model trained with native-quality speech to associate the word hypotheses with corresponding sounds of the digitized speech;
calculating statistics regarding individual words and phonemes of the word hypotheses based on said alignment;
calculating a plurality of features for use in assessing pronunciation of the speech based on the statistics;
calculating an assessment score based on one or more of the calculated features; and
storing the assessment score in a computer-readable memory.
15. The system of claim 14 wherein the steps further comprise excluding words not reliably recognized in generating the word hypotheses from contributing to the assessment score.
16. The system of claim 15 , wherein a confidence level is associated with each word in the word hypotheses identifying a likelihood that a word in a word hypothesis was correctly recognized in generating the word hypotheses.
17. The system of claim 16 , wherein words having corresponding hypotheses that do not meet a confidence threshold are not considered in calculating the assessment score.
18. The system of claim 14 wherein the features are based on at least one of: Hidden Markov Model probabilities, average phoneme duration, average word duration, phoneme duration distribution, word duration distribution, energy measurements, energy distribution, pitch measurements, pitch distribution, and pitch contours.
19. The system of claim 14 wherein speech samples are scored by a human, and a statistical model is built using the features and human scores;
wherein the assessment score is based on the scoring model and one or more of the calculated features.
20. The system of claim 19 , wherein the statistical model is built using multiple regression.
21. The system of claim 14 , wherein the reference acoustic model is trained with only native speech.
22. The system of claim 14 , wherein generating word hypotheses utilizes the non-native acoustic model, a dictionary that maps pronunciations to words, and a language model that identifies a likelihood that a word will follow a sequence of already hypothesized words in a speech recording.
23. The system of claim 22 , wherein the language model is an n-gram language model.
24. The system of claim 14 , wherein the assessment score is based on one or more features selected from the group consisting of:
an average likelihood across all letters: L 1 /m, where L 1 is a summation of likelihoods of all individual words:
L
1
=
∑
i
=
1
n
L
(
x
i
)
,
where L(x i ) is a likelihood of word x i being spoken given an observed audio signal, where n is a number of words in a response, where m is a number of letters in the response;
an average likelihood across all words: L 1 /n;
an average likelihood per second normalized by a rate of speech: L 4 /R, where L 4 =L 1 /T, where T is the summation of a duration of all words in the response, where R=m/T s , where T s is a duration of the response;
an average likelihood density across all words normalized by a rate of speech, L 5 /R, where
L
5
=
∑
i
=
1
n
L
(
x
i
)
t
i
n
,
where t i is a duration of word i in a response;
an average vowel duration shift:
S
_
=
∑
i
=
1
N
v
S
v
i
N
v
,
where N v is a total number of vowels, where S v i where is the duration shift of vowel v i , which is measured as an absolute value of the difference between a duration of vowel v i and a standard value of a duration of a vowel estimated on native speech data, where S v i =|P v i −D v i |, where P v i is the duration of vowel v i and D v i is the standard average duration of vowel v i ; and
an average normalized vowel duration shifts:
S
_
n
=
∑
i
=
1
N
v
Sn
v
i
N
v
,
where Sn v i is a normalized duration shift of vowel v i , which is measured as an absolute value of a normalized difference between a duration of vowel v i and a standard value of a duration of a vowel estimated on the native speech data, where Sn v i is calculated as
P
v
i
P
-
D
v
i
D
,
where P v i is the duration of vowel v i , P is an average vowel duration across all vowels in the response being scored, D v i is a standard average duration of vowel vi , and D is a standard vowel duration estimated on all vowels in native speech data.
25. The system of claim 14 , wherein one or more of the features are utilized with related stress, intonation, vocabulary, or grammar to generate the assessment score indicating communicative competence or other construct of speaking proficiency that includes pronunciation proficiency.
26. The system of claim 14 , wherein second statistics derived from native-quality speech are utilized in calculating the plurality of features.
27. A non-transitory computer-readable memory comprising computer-readable instructions, which when executed cause a processor to perform steps comprising:
performing speech recognition on digitized speech using a non-native acoustic model trained with non-native speech to generate word hypotheses for the digitized speech;
performing time alignment between the digitized speech and the word hypotheses utilizing a reference acoustic model trained with native-quality speech;
calculating statistics regarding individual words and phonemes of the word hypotheses based on said alignment;
calculating a plurality of features for use in assessing pronunciation of the speech based on the statistics;
calculating an assessment score based on one or more of the calculated features; and
storing the assessment score in a computer-readable memory.
28. The method of claim 1 , wherein the digitized speech is spontaneous, non-native speech of a non-native language speaker.
29. The system of claim 14 , wherein the digitized speech is spontaneous, non-native speech of a non-native language speaker.
30. The non-transitory computer-readable memory of claim 27 , wherein the instructions cause the processor to perform steps comprising:
excluding words not reliably recognized in generating the word hypotheses from contributing to the assessment score.
31. The non-transitory computer-readable memory of claim 30 , wherein a confidence level is associated with each word in the word hypotheses identifying a likelihood that a word in a word hypothesis was correctly recognized in generating the word hypotheses.
32. The non-transitory computer-readable memory of claim 31 , wherein words having corresponding word hypotheses that do not meet a confidence threshold are not considered in calculating the assessment score.
33. The non-transitory computer-readable memory of claim 27 , wherein the features are based on at least one of: Hidden Markov Model probabilities, average phoneme duration, average word duration, phoneme duration distribution, word duration distribution, energy measurements, energy distributions, pitch measurements, pitch distributions, and pitch contours.
34. The non-transitory computer-readable memory of claim 27 , wherein speech samples are scored by a human, and a statistical model is built using the features and human scores;
wherein the assessment score is based on the scoring model and one or more of the calculated features.
35. The non-transitory computer-readable memory of claim 34 , wherein the statistical model is built using multiple regression.
36. The non-transitory computer-readable memory of claim 27 , wherein the reference acoustic model is trained with only native speech.
37. The non-transitory computer-readable memory of claim 27 , wherein generating word hypotheses utilizes the non-native acoustic model, a dictionary that maps pronunciations to words, and a language model that identifies a likelihood that a word will follow a sequence of already hypothesized words in a speech recording.
38. The non-transitory computer-readable memory of claim 37 , wherein the language model is an n-gram language model.
39. The non-transitory computer-readable memory of claim 27 , wherein the assessment score is based on one or more features selected from the group consisting of:
an average likelihood across all letters: L 1 /m, where L 1 is a summation of likelihoods of all individual words:
L
1
=
∑
i
=
1
n
L
(
x
i
)
,
where L(x i ) is a likelihood of word x i being spoken given an observed audio signal, where n is a number of words in a response, where m is a number of letters in the response;
an average likelihood across all words: L 1 /n;
an average likelihood per second normalized by a rate of speech: L 4 /R, where L 4 =L 1 /T, where T is the summation of a duration of all words in the response, where R=m/T s , where T s is a duration of the response;
an average likelihood density across all words normalized by a rate of speech, L 5 /R, where
L
5
=
∑
i
=
1
n
L
(
x
i
)
t
i
n
,
where t i is a duration of word i in a response;
an average vowel duration shift:
S
_
=
∑
i
=
1
N
v
S
v
i
N
v
,
where N v is a total number of vowels, where S v i is the duration shift of vowel v i , which is measured as an absolute value of the difference between a duration of vowel v i and a standard value of a duration of a vowel estimated on native speech data, where S v i =|P v i −D v i |, where P v i is the duration of vowel v i and D v i is the standard average duration of vowel v i ; and
an average normalized vowel duration shifts:
S
_
n
=
∑
i
=
1
N
v
Sn
v
i
N
v
,
where Sn v i is a normalized duration shift of vowel v i , which is measured as an absolute value of a normalized difference between a duration of vowel v i and a standard value of a duration of a vowel estimated on the native speech data, where Sn v i is calculated as
P
v
i
P
-
D
v
i
D
,
where P v i is the duration of vowel v i , P is an average vowel duration across all vowels in the response being scored, D v i is a standard average duration of vowel v i , and D is a standard vowel duration estimated on all vowels in native speech data.
40. The non-transitory computer-readable memory of claim 27 , wherein one or more of the features are utilized with related stress, intonation, vocabulary, or grammar to generate the assessment score indicating communicative competence or other construct of speaking proficiency that includes pronunciation proficiency.
41. The non-transitory computer-readable memory of claim 27 , wherein second statistics derived from native-quality speech are utilized in calculating the plurality of features.
42. The non-transitory computer-readable memory of claim 27 , wherein the digitized speech is spontaneous, non-native speech of a non-native language speaker.