Emotion detection in audio interactions

1. A method comprising:

receiving a plurality of audio segments comprising a speech signal, wherein said audio segments represent a plurality of verbal interactions;

receiving labels associated with an emotional state expressed in each of said audio segments;

dividing each of said audio segments into a plurality of frames, based on a specified frame duration;

extracting a plurality of acoustic features from each of said frames;

computing statistics over said acoustic features with respect to sequences of frames representing phoneme boundaries in said audio segments;

at a training stage, training a machine learning model on a training set comprising:

(i) said statistics associated with said audio segments, and

(ii) said labels; and

at an inference stage, applying said trained machine learning model to one or more target audio segments comprising a speech signal, to detect an emotional state expressed in said target audio segments and a confidence score related to a probability of the detected emotional state being correct;

wherein the confidence score is calculated to represent contextual emotion given emotional states of neighboring audio segments of the target audio segment.

2. The method of claim 1 , wherein said audio segments are arranged in temporal sequences based on their association with a specified interaction of said verbal interactions.

3. The method of claim 2 , wherein boundaries of said temporal sequences are determined based, at least in part, on a continuity of said speech signal in said audio segments.

4. The method of claim 2 , wherein said statistics are computed with respect to said temporally-sequenced audio segments, and wherein said labels are associated with an emotional state expressed in said audio segments.

5. The method of claim 1 , wherein said training set further comprises vector representations of phonemes defined by said phoneme boundaries;

wherein the confidence score is calculated to represent the contextual emotion given the emotional states of the neighboring audio segments using a forward-backward algorithm.

6. The method of claim 5 , wherein said emotional state is one of neutral and negative; and

wherein the forward-backward algorithm comprises calculating the probability for the confidence score of the target audio segment as being equal to a sum of confidence scores of all paths going through the target audio segment divided by a sum of confidence scores of all possible paths for a sequence including the target audio segment and the neighboring audio segments.

7. The method of claim 6 , wherein said acoustic features are selected from the group consisting of: Mel-frequency cepstral coefficients (MFCC), Probability-of-Voicing (POV) features; pitch features; cutoff frequency; signal-to-noise-ratio (SNR) characteristics; speech descriptors; vocal tract characteristics; volume; signal energy; spectral distribution; tilt; sharpness; spectral flux; Chroma features; and zero-crossing rate (ZCR); and

wherein the emotional states of the labels of the received plurality of audio segments comprise a negative-to-neutral ratio of 4:1.

8. The method of claim 1 , wherein said statistics are selected from the group consisting of mean and standard deviation; and

wherein the machine learning model comprises a neural network architecture configured to take into account sequential characteristics of each of the audio segments, with a basic unit thereof being a phoneme, and a sequential nature of the audio segments in a corresponding one of the verbal interactions.

9. The method of claim 1 , wherein said phoneme boundaries are obtained based on applying a speech-to-text machine learning model to said audio segments; and

wherein the machine learning model comprises a neural network having a bidirectional long short-term memory with a conditional random field (BiLSTM-CRF) architecture.

10. The method of claim 1 , wherein said extracting further comprises a feature normalization stage, and wherein said normalization is performed with respect to at least one of: features associated with all of said frames; features associated with frames representing speech by a particular speaker within an interaction of said verbal interactions; and features associated with said sequences of frames representing phoneme boundaries associated with speech by a particular speaker within an interaction of said verbal interactions.

11. The method of claim 1 , wherein said verbal interactions represent conversations between a customer and a call center agent.

12. The method of claim 11 , wherein those of said audio segments comprising a speech signal representing speech by said agent are removed from said training set.

13. The method of claim 1 , wherein said target audio segments are a temporal sequence of audio segments from an individual verbal interaction.

14. A system comprising:

at least one hardware processor; and

a non-transitory computer-readable storage medium having stored thereon program instructions, the program instructions executable by the at least one hardware processor to:

receive a plurality of audio segments comprising a speech signal, wherein said audio segments represent a plurality of verbal interactions;

receive labels associated with an emotional state expressed in each of said audio segments;

divide each of said audio segments into a plurality of frames, based on a specified frame duration;

extract a plurality of acoustic features from each of said frames;

compute statistics over said acoustic features with respect to sequences of frames representing phoneme boundaries in said audio segments;

at a training stage, train a machine learning model on a training set comprising:

(i) said statistics associated with said audio segments, and

(ii) said labels; and

at an inference stage, apply said trained machine learning model to one or more target audio segments comprising a speech signal, to detect an emotional state expressed in said target audio segments and a confidence score related to a probability of the detected emotional state being correct;

wherein the confidence score is calculated to represent contextual emotion given emotional states of neighboring audio segments of the target audio segment.

15. The system of claim 14 , wherein said audio segments are arranged in temporal sequences based on their association with a specified interaction of said verbal interactions.

16. The system of claim 15 , wherein boundaries of said temporal sequences are determined based, at least in part, on a continuity of said speech signal in said audio segments.

17. The system of claim 15 , wherein said statistics are computed with respect to said temporally-sequenced audio segments, and wherein said labels are associated with an emotional state expressed in said audio segments.

18. The system of claim 14 , wherein said training set further comprises vector representations of phonemes defined by said phoneme boundaries;

wherein the confidence score is calculated to represent the contextual emotion given the emotional states of the neighboring audio segments using a forward-backward algorithm.

19. The system of claim 18 , wherein said emotional state is one of neutral and negative; and

wherein the forward-backward algorithm comprises calculating the probability for the confidence score of the target audio segment as being equal to a sum of confidence scores of all paths going through the target audio segment divided by a sum of confidence scores of all possible paths for a sequence including the target audio segment and the neighboring audio segments.

20. The system of claim 19 , wherein said acoustic features are selected from the group consisting of: Mel-frequency cepstral coefficients (MFCC), Probability-of-Voicing (POV) features; pitch features; cutoff frequency; signal-to-noise-ratio (SNR) characteristics; speech descriptors; vocal tract characteristics; volume; signal energy; spectral distribution; tilt; sharpness; spectral flux; Chroma features; and zero-crossing rate (ZCR); and

wherein the emotional states of the labels of the received plurality of audio segments comprise a negative-to-neutral ratio of 4:1.

21. The system of claim 14 , wherein said statistics are selected from the group consisting of mean and standard deviation deviation; and

wherein the machine learning model comprises a neural network architecture configured to take into account sequential characteristics of each of the audio segments, with a basic unit thereof being a phoneme, and a sequential nature of the audio segments in a corresponding one of the verbal interactions.

22. The system of claim 14 , wherein said phoneme boundaries are obtained based on applying a speech-to-text machine learning model to said audio segments; and

wherein the machine learning model comprises a neural network having a bidirectional long short-term memory with a conditional random field (BiLSTM-CRF) architecture.

23. The system of claim 14 , wherein said extracting further comprises a feature normalization stage, and wherein said normalization is performed with respect to at least one of: features associated with all of said frames; features associated with frames representing speech by a particular speaker within an interaction of said verbal interactions; and features associated with said sequences of frames representing phoneme boundaries associated with speech by a particular speaker within an interaction of said verbal interactions.

24. The system of claim 14 , wherein said verbal interactions represent conversations between a customer and a call center agent.

25. The system of claim 24 , wherein those of said audio segments comprising a speech signal representing speech by said agent are removed from said training set.

26. The system of claim 14 , wherein said target audio segments are a temporal sequence of audio segments from an individual verbal interaction.