Emotional detection and processing for barge-in speech
A method for managing an interaction between a user and a driver interaction system in a vehicle, the method comprising presenting a first audio output to a user from an output device of the driver interaction system, and, while presenting the first audio output to the user, receiving sensed input at the driver interaction system, processing the sensed input including determining an emotional content of the driver, and controlling the interaction based at least in part on the emotional content of the sensed input.
1 . A method for managing an interaction between a user and a driver-interaction system in a vehicle, the method comprising:
presenting a first audio output to a user from an output device of the driver-interaction system, and
while presenting the first audio output to the user, receiving sensed input at the driver-interaction system, processing the sensed input, wherein processing the sensed input comprises determining whether the sensed input includes speech of the driver defining a barge-in event, and determining an emotional content of the driver from the speech of the driver, and controlling the interaction based at least in part on the emotional content responsive to the sensed input being determined to include speech of the driver defining the barge-in event.
2 . The method of claim 1 , wherein controlling the interaction includes aborting presentation of the first audio output according to the processing, determining a dialog state according to the processing, and presenting a subsequent audio output based on the determined dialog state.
3 . The method of claim 1 , wherein the sensed input comprises spoken input and wherein processing the sensed input comprises determining an amplitude of the spoken input.
4 . The method of claim 1 , wherein processing the sensed input comprises determining the presence of speech in the sensed input based on frequency content of the sensed input.
5 . The method of claim 1 , wherein determining the emotional content of the sensed input includes classifying features of the sensed input using an emotion detector.
6 . The method of claim 1 , wherein determining the emotional content of the sensed input comprises classifying features of the sensed input into discrete emotion categories.
7 . The method of claim 1 , wherein determining emotion content of the sensed input comprises classifying the features of the sensed output into emotions from a discrete set of emotions and assigning scores to each of the emotions.
8 . The method of claim 1 , wherein determining emotion content of the sensed input comprises assigning scores to each emotion in a discrete set of emotions.
9 . The method of claim 1 , wherein the sensed input is a barge-in event and wherein the barge-in event occurs when the driver begins speaking during presentation of the first audio output by the driver-interaction system, wherein the first audio output comprises a first verbal output.
10 . The method of claim 1 , wherein processing the sensed input comprises determining the presence of speech in the sensed input based on periodicity of the sensed input.
11 . The method of claim 1 , wherein processing the sensed input comprises determining the presence of speech in the sensed input based on energy of the sensed input.
12 . The method of claim 1 , wherein determining the emotional content of the sensed input includes processing the sensed input to determine a dimensional representation of the emotional content of the sensed input and wherein the dimensional representation of the emotional content includes a scalar representation of the emotional content in a substantially continuous range of scalar values corresponding to a range of emotions.
13 . The method of claim 1 , wherein the sensed input comprises spoken input and wherein processing the sensed input comprises determining a pitch of the spoken input.
14 . The method of claim 1 , wherein the sensed input comprises spoken input and wherein processing the sensed input comprises processing spectral features of the spoken input.
15 . The method of claim 1 , wherein the sensed input includes one or more of camera input, physiological sensor input, radar sensor input, proximity sensor input, location information, and temperature input.
16 . The method of claim 1 , wherein controlling the interaction includes aborting presentation of the first audio output in response to having determined that the emotional content of the spoken input indicates a negative emotion toward the first audio output.
17 . The method of claim 1 , wherein controlling the interaction includes aborting presentation of the first audio output according to the processing, wherein aborting the presentation is carried out in response to determining that the emotional content of the spoken input indicates a lack of understanding of the first audio input.
18 . The method of claim 1 , wherein controlling the interaction includes continuing presentation of the first audio output based on a determination that the emotional content of the spoken input indicates a positive emotion toward the first audio output.
19 . The method of claim 1 , wherein controlling the interaction includes aborting presentation of the first audio output in response to having detected a negative emotion towards the first audio output in said sensed input, said sensed input being a barge-in event that occurred when the driver began speaking during the first audio output.
20 . The method of claim 1 , wherein the driver-interaction system constantly senses for sensed input.
21 . The method of claim 1 , wherein controlling the interaction comprises determining that presentation of the first audio output is to continue notwithstanding having received the sensed input.
22 . The method of claim 1 , wherein the sensed input is a first sensed input, wherein the emotional content is a first emotional content, wherein the method further comprises,
based on the first emotional content, aborting the presentation of the first audio output,
after having aborted presentation of the first audio output, presenting a second audio output to the user, and
while presenting the second audio output to the user, receiving a second sensed input at the driver-interaction system and processing the second sensed input,
wherein processing the second sensed input comprises determining a second emotional content of the driver based on the second sensed input and, based on the second emotional content, continuing presentation of the second audio output.
23 . A driver-interaction system for interacting with a driver in a vehicle, the driver-interaction system comprising:
a microphone that senses sensed signals from the driver, wherein the sensed signals includes speech;
an output device through which the driver-interaction system presents audio output to the driver;
a speech detector, wherein the speech detector processes the sensed signals and generates output speech signals corresponding to speech from the driver;
a speech recognizer, wherein the speech recognizer processes the sensed signals to generate a transcript of the speech signals;
an emotion detector, wherein the emotion detector processes the sensed signals to generate a classified emotion of the driver; and
a barge-in detector, wherein, while the driver-interaction system is presenting audio output to the driver, the barge-in detector carries out processing steps to determine that a barge-in event has occurred involving speech of the driver and to determine whether or not to interrupt the audio output being presented in response to the barge-in event based on the classified emotion of the driver determined from the speech of the driver defining the barge-in event, wherein the processing steps carried out by the barge-in detector rely on one or more of: the sensed signals, the output speech signals, the transcript, and the classified emotion.
24 . The driver-interaction system of claim 23 , further comprising an interaction-control module that is configured for controlling an interaction with the driver by determining a dialog state based on processing carried out by one or more of the barge-in detector, the driver-sensing components, the speech detector, and the speech recognizer and presenting a subsequent audio output to the driver based on the determined dialog state.
25 . The driver-interaction system of claim 23 , wherein the barge-in event occurs when the driver begins speaking during presentation of the verbal output by the driver-interaction system.
26 . A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by a processor of a driver-interaction system, cause the driver-interaction system to, as the driver-interaction system outputs a first audio output to a driver, execute a first action, a second action, and a third action, wherein: the first action is that of receiving sensed input; the second action is that of processing the sensed input including determining whether the sensed input includes speech of the driver defining a barge-in event, and determining an emotional content of the driver from the speech of the driver, and the third action is that of controlling an interaction with the driver based at least in part on the emotional content of the sensed input responsive to the sensed input being determined to include speech of the driver defining the barge-in event.