Psycho-physical state sensitive voice dialogue system

1. A system, comprising:

a psycho-physical state detection mechanism for detecting a psycho-physical state of a user based on input speech data from the user;

a spoken dialogue mechanism for carrying on a dialogue with the user based on the psycho-physical stale of the user, detected by the psycho-physical detection mechanism from the input speech data from the user;

a speech understanding mechanism for understanding the input speech data from the user based on the psycho-physical state of the user to generate a literal meaning of the speech data; and

a voice response generation mechanism for generating a voice response to the user based on the literal meaning of the input speech data and the psycho-physical state of the user, wherein the voice response to the user is linguistically and acoustically adjusted according to the detected psycho-physical state of the user.

2. The system according to claim 1 , wherein the speech understanding mechanism comprises:

at least one acoustic model for characterizing acoustic properties of the input speech data, each of the at least one acoustic model corresponding to some distinct characteristic related to a psycho-physical state of a speaker;

an acoustic model selection mechanism for selecting an acoustic model that is appropriate to according to the psycho-physical state detected by the psycho-physical state detection mechanism;

a speech recognizer for generating a transcription of spoken words recognized from the input speech data using the acoustic model selected by the acoustic model selection mechanism; and

a language understanding mechanism for interpreting the literal meaning of the input speech data based on the transcription.

3. The system according to claim 1 , wherein the voice response generation mechanism comprises:

a natural language response generator for generating a response based on an understanding of the transcription, the response being generated appropriately according to the psycho-physical state of the user; a prosodic pattern determining mechanism for determining a prosodic pattern to be applied to the response that is considered as appropriate according to the psycho-physical state; and

a text-to-speech engine for synthesizing the voice response based on the response and the prosodic pattern.

4. The system according to claim 1 , wherein the psycho-physical state detection mechanism comprises:

an acoustic feature extractor for extracting acoustic features from the input speech data to generate at least one acoustic feature; and

a psycho-physical state classifier for classifying the input speech data into one or more psycho-physical states based on the at least one acoustic feature.

5. The system according to claim 4 , further comprising:

at least one psycho-physical state model, each of the at least one psycho-physical state model corresponding to a single psycho-physical state and characterizing the acoustic properties of the single psycho-physical state; and

an off-line training mechanism for establishing the at least one psycho-physical model based on labeled training speech data.

6. The system according to claim 1 , further comprising a dialogue manager to control the dialogue flow.

7. A method, comprising:

receiving, by a psycho-physical state detection mechanism, input speech data from a user;

detecting a psycho-physical state of the user from the input speech data;

understanding, by a speech understanding mechanism, a literal meaning of spoken words recognized from the input speech data based on the psycho-physical state of the user, detected by the detecting; and

generating, by a voice response generation mechanism, a voice response to the user based on the literal meaning of the input speech data and the psycho-physical state of the user, wherein the voice response to the user is linguistically and acoustically adjusted according to the detected psycho-physical state of the user.

8. The method according to claim 7 , wherein the detecting comprises:

extracting, by an acoustic feature extractor, at least one acoustic feature from the input speech data; and

classifying, by a psycho-physical state classifier and based on said at least one acoustic feature, the input speech data into the psycho-physical state according to at least one psycho-physical state model.

9. The method according o claim 8 , further comprising: receiving, by an off-line training mechanism, labeled training data, wherein each of the data items in the labeled training data is labeled by a psycho-physical state; and

building the at least one psycho-physical state model using the labeled training data, each of the at least one psycho-physical state model corresponding to a single psycho-physical state and being established based on the data items in the labeled training data that have a label corresponding to the single psycho-physical state.

10. The method according to claim 7 , wherein the understanding comprises:

selecting, by an acoustic model selection mechanism, an acoustic model, from at least one acoustic model, that is appropriate to according to the psycho-physical state, detected by the detecting, each of the at least one acoustic model corresponding to some distinct speech characteristic related to the psycho-physical state;

recognizing, by a speech recognizer, the spoken words from the input speech data using the acoustic model, selected by the selecting, to generate a transcription; and

interpreting, by a language understanding mechanism, the literal meaning of the spoken words based on the transcription.

11. The method according to claim 7 , wherein the generating comprises:

constructing, by a natural language response generator, a natural language response based on an understanding of the transcription, the natural language response being constructed appropriately according to the psycho-physical state of the user;

determining, by a prosodic pattern determining mechanism, a prosodic pattern to be applied to said natural language response, wherein the prosodic pattern is considered to be appropriate according to the psycho-physical state; and

synthesizing, by a text-to-speech engine, the voice response based on the natural language response and the prosodic pattern.

12. A computer-readable medium encoded with a program, said program comprising instructions that when executed by a computer cause the computer to:

receive, by a psycho-physical state detection mechanism, input speech data from a user;

detect a psycho-physical state of the user from the input speech data;

understand, by a speech understanding mechanism, a literal meaning of spoken words recognized from the input speech data based on the psycho-physical state of the user; and

generate, by a voice response generation mechanism, a voice response to the user based on the literal meaning of the input speech data and the psycho-physical state of the user, wherein the voice response to the user is linguistically and acoustically adjusted according to the detected psycho-physical state of the user.

13. The medium according to claim 12 , wherein the to detect comprises instructions that when executed by the computer cause the computer to:

extract, by a acoustic feature extractor, at least one acoustic feature from the input speech data; and

classify, by a psycho-physical state classifier and based on the at least one feature, the input speech data into the psycho-physical state according to at least one psycho-physical state model.

14. The medium according to claim 13 , further comprising instructions that when executed by the computer cause the computer to:

receive, by an off-line training mechanism, labeled training data, wherein each of the data items in the labeled training data is labeled by a psycho-physical state; and

building the at least one psycho-physical state model using the labeled training data, each of the at least one psycho-physical state model corresponding to a single psycho-physical state and being established based on the data items in the labeled training data that have a label corresponding to the single psycho-physical state.

15. The medium according to claim 12 , wherein the understanding comprises instructions that when executed by the computer cause the computer to:

select, by an acoustic model selection mechanism, an acoustic model, from at least one acoustic model, that is appropriate according to the psycho-physical state, each of the at least one acoustic model corresponding to some distinct speech characteristic related to a psycho-physical state;

reconize, by a speech recognizer, the spoken words from the input speech data using the acoustic model, to generate a transcription; and

interpret, by a language understanding mechanism, the literal meaning of the spoken words based on the transcription.

16. The medium according to claim 12 , wherein the to generate comprises instructions that when executed by the computer cause the computer to:

construct, by a natural language response generator, a natural language response based on an understanding of the transcription, the natural language response being constructed appropriately according to the psycho-physical state of the user;

determine, by a prosodic pattern determining mechanism, a prosodic pattern to be applied to the natural language response, wherein the prosodic pattern is considered to be appropriate according to the psycho-physical state; and

synthesize, by a text-to-speech engine, the voice response based on the natural language response and the prosodic pattern.