Methods for measuring speech intelligibility, and related systems and apparatus

1. A speech intelligibility scoring method,

comprising:

receiving an acoustic signal encoding an utterance of a user, wherein the utterance comprises a verbalization of a sample text by the user;

generating, by an automatic speech recognition (ASR) module, an N-best output corresponding to the utterance, wherein the N-best output comprises N recognition results generated by the ASR module for the utterance, wherein N is a positive integer equal or greater than 2;

calculating N conditional intelligibility values corresponding, respectively, to the N recognition results, wherein each of the N conditional intelligibility values represents a conditional intelligibility of the utterance assuming that the respective recognition result corresponding to the respective conditional intelligibility value is a correct transcription of the utterance generated by the ASR module with respect to the sample text;

determining, based on the N conditional intelligibility values, which of the N recognition results is most intelligible; and

generating an intelligibility score representing an intelligibility of the utterance based, at least in part, on (i) the conditional intelligibility of the most intelligible recognition result and (ii) a rank of the most intelligible recognition result within the N-best output,

wherein the intelligibility score is set to a combined value and

wherein the combined value is a weighted sum of the conditional intelligibility value of the most intelligible recognition result and the conditional intelligibility value of a 1-best recognition result.

2. The method of claim 1 , wherein calculating the N conditional intelligibility values comprises:

normalizing the sample text and the N recognition results; and

for each of the N normalized recognition results:

aligning the respective normalized recognition result to the normalized sample text on a word-for-word basis,

calculating an error rate of the aligned recognition result based on a number of errors in the aligned recognition result relative to the normalized sample text, and

calculating the respective conditional intelligibility value based on the error rate of the aligned recognition result.

3. The method of claim 2 , wherein aligning the respective normalized recognition result to the normalized sample text on a word-for-word basis comprises:

calculating a distance between the normalized sample text and the respective normalized recognition result.

4. The method of claim 3 , wherein the distance is an edit distance.

5. The method of claim 1 ,

wherein the intelligibility score is further based on (i) a confidence score, wherein the confidence score indicates a probability that a particular one of the N recognition results is a correct transcription of the utterance, and/or

(ii) a pronunciation score for the utterance, wherein the pronunciation score indicates an extent to which the utterance exhibits correct pronunciation of the sample text.

6. The method of claim 5 , wherein the confidence score indicates the probability that the most intelligible recognition result is the correct transcription of the utterance, and wherein calculating the intelligibility score further comprises adjusting the intelligibility score based on the confidence score and/or the pronunciation score.

7. The method of claim 6 , wherein adjusting the intelligibility score comprises changing the intelligibility score by a first penalty value, wherein the first penalty value is determined based on the larger of the confidence score and the pronunciation score.

8. The method of claim 7 , wherein adjusting the intelligibility score further comprises changing the intelligibility score by a second penalty value if the pronunciation score is less than a threshold value, wherein the second penalty value is determined based on the pronunciation score.

9. The method of claim 6 , wherein adjusting the intelligibility score comprises changing the intelligibility score by a penalty value if the pronunciation score is less than a threshold value, wherein the penalty value is determined based on the pronunciation score.

10. The method of claim 1 , further comprising:

generating, by the automatic speech recognition (ASR) module, respective confidence scores for each of the N recognition results, wherein each respective confidence score indicates a probability that the corresponding recognition result is a correct transcription of the utterance.

11. The method of claim 10 , further comprising:

for each of the confidence scores, determining whether the respective confidence score is less than a threshold value, and if so, setting the respective confidence score to the threshold value.

12. The method of claim 5 , further comprising:

generating, by a pronunciation assessment module, a pronunciation score for the utterance.

13. The method of claim 1 , wherein N is greater than 2 and less than or equal to 5.

14. A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

receiving an acoustic signal encoding an utterance of a user, wherein the utterance comprises a verbalization of a sample text by the user;

generating, by an automatic speech recognition (ASR) module, an N-best output corresponding to the utterance, wherein the N-best output comprises N recognition results generated by the ASR module for the utterance, wherein N is a positive integer equal to or greater than 2;

calculating N conditional intelligibility values corresponding, respectively, to the N recognition results, wherein each of the N conditional intelligibility values represents a conditional intelligibility of the utterance assuming that the respective recognition result corresponding to the respective conditional intelligibility value is a correct transcription of the utterance generated by the ASR module with respect to the sample text;

determining, based on the N conditional intelligibility values, which of the N recognition results is most intelligible; and

generating an intelligibility score representing an intelligibility of the utterance based, at least in part, on (i) the conditional intelligibility value of the most intelligible recognition result from the N-best output and (ii) a rank of the most intelligible recognition result within the N-best output,

wherein the intelligibility score is set to a combined value and

wherein the combined value is a weighted sum of the conditional intelligibility value of the most intelligible recognition result and the conditional intelligibility value of a 1-best recognition result.

15. The system of claim 14 , wherein generating the intelligibility score comprises:

initializing the intelligibility score based, at least in part, on the conditional intelligibility value of the most intelligible recognition result.

16. The system of claim 14 ,

wherein the intelligibility score is further based on (i) a confidence score, wherein the confidence score indicates a probability that a particular one of the N recognition results is a correct transcription of the utterance, and/or

(ii) a pronunciation score for the utterance, wherein the pronunciation score indicates an extent to which the utterance exhibits correct pronunciation of the sample text.

17. The system of claim 16 , wherein the confidence score indicates the probability that the most intelligible recognition result is the correct transcription of the utterance, and wherein calculating the intelligibility score further comprises adjusting the intelligibility score based on the confidence score and/or the pronunciation score.

18. The system of claim 17 , wherein adjusting the intelligibility score comprises:

changing the intelligibility score by a first penalty value, wherein the first penalty value is determined based on the larger of the confidence score and the pronunciation score; and

changing the intelligibility score by a second penalty value if the pronunciation score is less than a threshold value, wherein the second penalty value is determined based on the pronunciation score.

19. A computer storage medium having instructions stored thereon that, when executed by a data processing apparatus, cause the data processing apparatus to perform operations comprising:

receiving an acoustic signal encoding an utterance of a user, wherein the utterance comprises a verbalization of a sample text by the user;

generating, by an automatic speech recognition (ASR) module, an N-best output corresponding to the utterance, wherein the N-best output comprises N recognition results generated by the ASR module for the utterance, wherein N is a positive integer equal to or greater than 2;

calculating N conditional intelligibility values corresponding, respectively, to the N recognition results, wherein each of the N conditional intelligibility values represents a conditional intelligibility of the utterance assuming that the respective recognition result corresponding to the respective conditional intelligibility value is a correct transcription of the utterance generated by the ASR module with respect to the sample text;

determining, based on the N conditional intelligibility values, which of the N recognition results is most intelligible; and

generating an intelligibility score representing an intelligibility of the utterance based, at least in part, on (i) the conditional intelligibility value of the most intelligible recognition result from the N-best output and (ii) a rank of the most intelligible recognition result within the N-best output,

wherein the intelligibility score is set to a combined value and

wherein the combined value is a weighted sum of the conditional intelligibility value of the most intelligible recognition result and the conditional intelligibility value of a 1-best recognition result.