Speech processing system and method
View Patent ↗A speech processing system includes an input for receiving an input utterance spoken by a user and a word alignment unit configured to align different sequences of acoustic speech models with the input utterance spoken by the user. Each different sequence of acoustic speech models corresponds to a different possible utterance that a user might make. The system identifies any parts of a read prompt text that the user skipped; any parts of the read prompt text that the user repeated; and any speech sounds that the user inserted between words of the read prompt text. The information from the word alignment unit can be used to assess the proficiency and/or fluency of the user's speech.
1. A speech processing system comprising:
an input for receiving an input utterance spoken by a user;
a speech recognition system that recognizes the input utterance spoken by the user and that outputs a recognition result comprising a sequence of recognized textual words, and an additional sequence of recognized sub-word units corresponding to the input utterance;
an acoustic model store that stores acoustic speech models;
a word alignment unit configured to receive the sequence of recognized textual words and the sequence of recognized sub-word units output by the speech recognition system and to align a sequence of said acoustic speech models corresponding to the received sequence of recognized textual words and the received sequence of recognized sub-word units with a sequence of acoustic feature vectors representing the input utterance spoken by the user and to output an alignment result identifying:
a time alignment between each word of the received sequence of recognized textual words and the sequence of acoustic feature vectors representing the input utterance spoken by the user that identifies where each recognized textual word is located within the sequence of acoustic feature vectors representing the input utterance spoken by the user;
a time alignment between each sub-word unit in the sequence of recognized sub-word units and the sequence of acoustic feature vectors representing the input utterance spoken by the user that identifies where each sub-word unit is located within the sequence of acoustic feature vectors representing the input utterance spoken by the user;
a speech scoring feature determining unit configured to use:
i) the time alignment between each word of the received sequence of recognized textual words and the sequence of acoustic feature vectors representing the input utterance spoken by the user that identifies where each recognized textual word is located within the sequence of acoustic feature vectors representing the input utterance spoken by the user; and
ii) the time alignment between each sub-word unit in the sequence of recognized sub-word units and the sequence of acoustic feature vectors representing the input utterance spoken by the user that identifies where each sub-word unit is located within the sequence of acoustic feature vectors representing the input utterance spoken by the user;
output from the word alignment unit to determine a plurality of speech scoring feature values for the input utterance; and
a scoring unit operable to use the plurality of speech scoring feature values for the input utterance determined by the speech scoring feature determining unit, to generate a score representing language ability of the user.
2. The speech processing system of claim 1 , wherein the word alignment unit is configured to output a sequence of sub-word units corresponding to a dictionary pronunciation of the recognized input utterance.
3. The speech processing system of claim 1 , wherein the word alignment unit is configured to output a sequence of sub-word units corresponding to a dictionary pronunciation of the matching possible utterance.
4. A speech processing system according to claim 3 , further comprising a sub-word alignment unit configured to receive the sequence of sub-word units corresponding to the dictionary pronunciation and configured to align the sequence of sub-word units corresponding to the dictionary pronunciation received from the word alignment unit with the input utterance spoken by the user whilst allowing for sub-word units to be inserted between words and for sub-word units of a word to be replaced by other sub-word units to determine where the input utterance spoken by the user differs from the dictionary pronunciation and to output a sequence of sub-word units corresponding to an actual pronunciation of the input utterance spoken by the user.
5. A speech processing system according to claim 4 , wherein the sub-word alignment unit is configured to use the sequence of sub-word units corresponding to the dictionary pronunciation of the recognized input utterance to generate a network having a plurality of paths allowing for sub-word units to be inserted between recognized words and for sub-word units of a recognized word to be replaced by other sub-word units and wherein the sub-word alignment unit is configured to align acoustic speech models for the different paths defined by the network with the input utterance spoken by the user.
6. A speech processing system according to claim 5 , wherein the sub-word alignment unit is configured to maintain a score representing the closeness of the match between the acoustic speech models for the different paths defined by the second network and input utterance spoken by the user.
7. A speech processing system according to claim 4 , further comprising a speech scoring feature determining unit configured to receive and to determine a measure of similarity between the sequence of sub-word units output by the word alignment unit and the sequence of sub-word units output by the sub-word alignment unit.
8. A speech processing system according to claim 1 , further comprising a free align unit configured to align acoustic speech models with the input utterance spoken by the user and to output an alignment result including a sequence of sub-word units that matches with the input utterance spoken by the user.
9. A speech processing system according to claim 1 , wherein the score represents the fluency and/or proficiency of the user's spoken utterance.
10. A speech processing method comprising:
receiving an input utterance spoken by a user;
using a speech recognition system to recognize the input utterance spoken by the user and to output a recognition result comprising a sequence of recognized textual words and an additional sequence of sub-word units corresponding to the input utterance; and
receiving the sequence of recognized textual words and the sequence of recognized sub-word units output by the speech recognition system and aligning a sequence of acoustic speech models corresponding to the received sequence of recognized textual words and the sequence of recognized sub-word units with a sequence of acoustic feature vectors representing the input utterance spoken by the user; and
outputting an alignment result identifying:
a time alignment between each word of the received sequence of recognized textual words and the sequence of acoustic feature vectors representing the input utterance spoken by the user that identifies where each recognized textual word is located within the sequence of acoustic feature vectors representing the input utterance spoken by the user; and
a time alignment between each sub-word unit in the sequence of recognized sub-word units and the sequence of acoustic feature vectors representing the input utterance spoken by the user that identifies where each sub-word unit is located within the sequence of acoustic feature vectors representing the input utterance spoken by the user;
using;
i) the time alignment between each word of the received sequence of recognized textual words and the sequence of acoustic feature vectors representing the input utterance spoken by the user that identifies where each recognized textual word is located within the sequence of acoustic feature vectors representing the input utterance spoken by the user; and
ii) the time alignment between each sub-word unit in the sequence of recognized sub-word units and the sequence of acoustic feature vectors representing the input utterance spoken by the user that identifies where each sub-word unit is located within the sequence of acoustic feature vectors representing the input utterance spoken by the user;
to determine a plurality of speech scoring feature values for the input utterance; and
using the determined plurality of speech scoring feature values for the input utterance to generate a score representing language ability of the user.