Detecting non-verbal, audible communication conveying meaning
Various embodiments of the invention provide methods, systems, and computer-program products for analyzing an audio to capture semantic and non-semantic characteristics of the audio and corresponding relationships between the semantic and non-semantic characteristics. In particular embodiments, the audio is segmented into a set of utterance segments containing a party speaking on the audio and a set of noise segments containing the party not speaking on the audio. The semantic and non-semantic characteristics are then captured for each of the utterance segments. Specifically, speech analytics is performed on each segment to identify the words spoken by the party in the segment as semantic characteristics. Further, laughter, emotion, and sentence boundary detection is performed on each segment to identify occurrences of such in the segment as non-semantic characteristics. Once identified for each segment, various embodiments of the invention involve constructing a transcript based on the identified semantic and non-semantic characteristics.
1. A method for analyzing an audio to capture semantic characteristics and non-semantic characteristics of the audio and corresponding relationships between the semantic characteristics and the non-semantic characteristics comprising:
segmenting the audio into a set of utterance segments and a set of noise segments by measuring one or more prosodic features of the audio to identify the set of utterance segments comprising segments of the audio containing a party speaking on the audio and the set of noise segments comprising segments of the audio containing the party not speaking on the audio;
capturing the semantic characteristics and the non-semantic characteristics for each of the utterance segments in the set of utterance segments by:
performing speech analytics on the utterance segment to identify words spoken by the party in the utterance segment;
extracting lexical features with respect to each of the words spoken by the party in the utterance segment;
extracting prosodic features across the utterance segment with respect to each of the words spoken by the party in the utterance segment and each inter-word boundary lying between two consecutive words spoken by the party in the utterance segment;
performing laughter detection by:
dividing the utterance segment into a set of overlapping windows comprising feature frames; and
for each of the overlapping windows, using at least some of the prosodic features extracted at locations in the audio corresponding to each of the feature frames in the overlapping window as input to a first predictive model to identify whether laughter exists in the utterance segment;
performing emotion detection by:
comparing each of the words spoken by the party in the utterance segment to an emotion lexicon comprising a list of emotions and words that are indicative of one or more emotions in the list of emotions to identify a first result comprising one or more emotions in the list of emotions for at least one of the words spoken by the party in the utterance segment;
using at least some of the prosodic features extracted for each of the words spoken by the party in the utterance segment as input to a second predictive model to identify a second result comprising one or more emotions expressed by the party in the utterance segment; and
using the first result and the second result as input to an ensemble comprising a third predictive model to arrive at a conclusion as to which of the one or more emotions for the first result and the one or more emotions for the second result is the emotion expressed by the party in the utterance segment; and
performing sentence boundary detection by using the lexical features and at least some of the prosodic features extracted for each of the words spoken by the party in the utterance segment and each of the inter-word boundaries lying between two consecutive words as input to a fourth predictive model to identify where in the utterance segment a sentence boundary exists with respect to each of the inter-word boundaries lying between two consecutive words; and
constructing a transcript identifying the semantic characteristics as the words spoken by the party in the audio and the non-semantic characteristics as occurrences of laughter, emotions, and sentence boundaries and the corresponding relationships between the semantic characteristics and the non-semantic characteristics as locations of the words and the occurrences of laughter, emotions, and sentence boundaries within the audio relative to one another.
2. The method of claim 1 further comprising:
segmenting a second audio into a set of utterance segments and a set of noise segments by measuring one or more prosodic features of the second audio to identify the set of utterance segments comprising segments of the second audio containing a second party speaking on the second audio and the set of noise segments comprising segments of the second audio containing the second party not speaking on the second audio;
capturing semantic characteristics and non-semantic characteristics for each of the utterance segments in the set of utterance segments by:
performing speech analytics on the utterance segment to identify words spoken by the second party in the utterance segment;
extracting lexical features with respect to each of the words spoken by the second party in the utterance segment;
extracting prosodic features across the utterance segment with respect to each of the words spoken by the second party in the utterance segment and each inter-word boundary lying between two consecutive words spoken by the second party in the utterance segment;
performing laughter detection by:
dividing the utterance segment into a set of overlapping windows comprising feature frames; and
for each of the overlapping windows, using at least some of the prosodic features extracted at locations in the second audio corresponding to each of the feature frames in the overlapping window as input to the first predictive model to identify whether laughter exists in the utterance segment;
performing emotion detection by:
comparing each of the words spoken by the second party in the utterance segment to the emotion lexicon to identify a first result comprising one or more emotions in the list of emotions for at least one of the words spoken by the second party in the utterance segment;
using at least some of the prosodic features extracted for each of the words spoken by the second party in the utterance segment as input to the second predictive model to identify a second result comprising one or more emotions expressed by the second party in the utterance segment; and
using the first result and the second result as input to the ensemble to arrive at a conclusion as to which of the one or more emotions for the first result and the one or more emotions for the second result is the emotion expressed by the second party in the utterance segment; and
performing sentence boundary detection by using the lexical features and at least some of the prosodic features extracted for each of the words spoken by the second party in the utterance segment and each of the inter-word boundaries lying between two consecutive words as input to the fourth predictive model to identify where in the utterance segment a sentence boundary exists with respect to each of the inter-word boundaries lying between two consecutive words;
constructing a second transcript identifying the semantic characteristics as the words spoken by the second party in the second audio and the non-semantic characteristics as occurrences of laughter, emotions, and sentence boundaries and the corresponding relationships between the semantic characteristics and the non-semantic characteristics as locations of the words and the occurrences of laughter, emotions, and sentence boundaries within the second audio relative to one another; and
constructing a combined transcript comprising the first transcript and the second transcript, wherein the combined transcript represents a conversation that had taken place between the first party and the second party.
3. The method of claim 2 further comprising:
displaying the combined transcript on a graphical user interface (“GUI”) on a computer monitor, the GUI comprising a timeline representing the conversation that had taken place between the first party and the second party;
displaying the semantic characteristics as the words spoken by the first party and the words spoken by the second party on the GUI at locations with respect to the timeline proximate to a time when the words spoken by the first party occurred and the words spoken by the second party occurred during the conversation; and
displaying the non-semantic characteristics for the first party and the second party as a plurality of markers on the GUI in conjunction with the timeline to demonstrate the corresponding relationships between the semantic characteristics and the non-semantic characteristics for the first party and the second party, wherein each of the markers represents an occurrence of laugher by the first party or the second party, an emotion expressed by the first party or the second party, or a sentence boundary for a sentence spoken by the first party or the second party, and each marker is displayed on the GUI at a location with respect to the timeline proximate to a time when the occurrence of laugher by the first party or the second party, the emotion expressed by the first party or the second party, or the sentence boundary for the sentence spoken by the first party or the second party took place during the conversation.
4. The method of claim 3 , wherein the GUI identifies at least one of the first party and the second party as being associated with each marker.
5. The method of claim 1 , wherein the first predictive model comprises one or more neural networks, the second predictive model comprises one or more support vector machines, the third predictive model comprises one or more multinomial logistic regression models, and the fourth predictive model comprises conditional random fields.
6. The method of claim 1 further comprising, for each sentence boundary detected in the utterance segment, using the words spoken by the first party that comprise a sentence associated with the sentence boundary as input to a fifth predictive model to identify whether the sentence associated with the sentence boundary is a question.
7. The method of claim 6 , wherein the fifth predictive model comprises one or more logistic regression models.
8. A non-transitory, computer-readable medium comprising computer-executable instructions for analyzing an audio to capture semantic characteristics and non-semantic characteristics of the audio and corresponding relationships between the semantic characteristics and the non-semantic characteristics that when executed by at least one computer processor are configured to cause the at least one computer processor to:
segment the audio into a set of utterance segments and a set of noise segments by measuring one or more prosodic features of the audio to identify the set of utterance segments comprising segments of the audio containing a party speaking on the audio and the set of noise segments comprising segments of the audio containing the party not speaking on the audio;
to capture the semantic characteristics and the non-semantic characteristics for each of the utterance segments in the set of utterance segments:
have speech analytics performed on the utterance segment to identify words spoken by the party in the utterance segment;
extract lexical features with respect to each of the words spoken by the party in the utterance segment;
extract prosodic features across the utterance segment with respect to each of the words spoken by the party in the utterance segment and each inter-word boundary lying between two consecutive words spoken by the party in the utterance segment;
to perform laughter detection:
divide the utterance segment into a set of overlapping windows comprising feature frames; and
for each of the overlapping windows, use at least some of the prosodic features extracted at locations in the audio corresponding to each of the feature frames in the overlapping window as input to a first predictive model to identify whether laughter exists in the utterance segment;
to perform emotion detection:
compare each of the words spoken by the party in the utterance segment to an emotion lexicon comprising a list of emotions and words that are indicative of one or more emotions in the list of emotions to identify a first result comprising one or more emotions in the list of emotions for at least one of the words spoken by the party in the utterance segment;
use at least some of the prosodic features extracted for each of the words spoken by the party in the utterance segment as input to a second predictive model to identify a second result comprising one or more emotions expressed by the party in the utterance segment; and
use the first result and the second result as input to an ensemble comprising a third predictive model to arrive at a conclusion as to which of the one or more emotions for the first result and the one or more emotions for the second result is the emotion expressed by the party in the utterance segment; and
to perform sentence boundary detection:
use the lexical features and at least some of the prosodic features extracted for each of the words spoken by the party in the utterance segment and each of the inter-word boundaries lying between two consecutive words as input to a fourth predictive model to identify where in the utterance segment a sentence boundary exists with respect to each of the inter-word boundaries lying between two consecutive words; and
construct a transcript identifying the semantic characteristics as the words spoken by the party in the audio and the non-semantic characteristics as occurrences of laughter, emotions, and sentence boundaries and the corresponding relationships between the semantic characteristics and the non-semantic characteristics as locations of the words and the occurrences of laughter, emotions, and sentence boundaries within the audio relative to one another.
9. The non-transitory, computer-readable medium of claim 8 , wherein the computer-executable instructions cause the at least one computer processor to:
segment a second audio into a set of utterance segments and a set of noise segments by measuring one or more prosodic features of the second audio to identify the set of utterance segments comprising segments of the second audio containing a second party speaking on the second audio and the set of noise segments comprising segments of the second audio containing the second party not speaking on the second audio;
to capture semantic characteristics and non-semantic characteristics for each of the utterance segments in the set of utterance segments:
perform speech analytics on the utterance segment to identify words spoken by the second party in the utterance segment;
extract lexical features with respect to each of the words spoken by the second party in the utterance segment;
extract prosodic features across the utterance segment with respect to each of the words spoken by the second party in the utterance segment and each inter-word boundary lying between two consecutive words spoken by the second party in the utterance segment;
to perform laughter detection:
divide the utterance segment into a set of overlapping windows comprising feature frames; and
for each of the overlapping windows, use at least some of the prosodic features extracted at locations in the second audio corresponding to each of the feature frames in the overlapping window as input to the first predictive model to identify whether laughter exists in the utterance segment;
to perform emotion detection:
compare each of the words spoken by the second party in the utterance segment to the emotion lexicon to identify a first result comprising one or more emotions in the list of emotions for at least one of the words spoken by the second party in the utterance segment;
use at least some of the prosodic features extracted for each of the words spoken by the second party in the utterance segment as input to the second predictive model to identify a second result comprising one or more emotions expressed by the second party in the utterance segment; and
use the first result and the second result as input to the ensemble to arrive at a conclusion as to which of the one or more emotions for the first result and the one or more emotions for the second result is the emotion expressed by the second party in the utterance segment; and
to perform sentence boundary detection:
use the lexical features and at least some of the prosodic features extracted for each of the words spoken by the second party in the utterance segment and each of the inter-word boundaries lying between two consecutive words as input to the fourth predictive model to identify where in the utterance segment a sentence boundary exists with respect to each of the inter-word boundaries lying between two consecutive words;
construct a second transcript identifying the semantic characteristics as the words spoken by the second party in the second audio and the non-semantic characteristics as occurrences of laughter, emotions, and sentence boundaries and the corresponding relationships between the semantic characteristics and the non-semantic characteristics as locations of the words and the occurrences of laughter, emotions, and sentence boundaries within the second audio relative to one another; and
construct a combined transcript comprising the first transcript and the second transcript, wherein the combined transcript represents a conversation that had taken place between the first party and the second party.
10. The non-transitory, computer-readable medium of claim 9 , wherein the computer-executable instructions cause the at least one computer processor to:
display the combined transcript on a graphical user interface (“GUI”) on a computer monitor for a user, the GUI comprising a timeline representing the conversation that had taken place between the first party and the second party;
display the semantic characteristics as the words spoken by the first party and the words spoken by the second party on the GUI at locations with respect to the timeline proximate to a time when the words spoken by the first party occurred and the words spoken by the second party occurred during the conversation; and
display the non-semantic characteristics for the first party and the second party as a plurality of markers on the GUI in conjunction with the timeline to demonstrate the corresponding relationships between the semantic characteristics and the non-semantic characteristics for the first party and the second party, wherein each of the markers represents an occurrence of laugher by the first party or the second party, an emotion expressed by the first party or the second party, or a sentence boundary for a sentence spoken by the first party or the second party, and each marker is displayed on the GUI at a location with respect to the timeline proximate to a time when the occurrence of laugher by the first party or the second party, the emotion expressed by the first party or the second party, or the sentence boundary for the sentence spoken by the first party or the second party took place during the conversation.
11. The non-transitory, computer-readable medium of claim 10 , wherein the GUI identifies at least one of the first party and the second party as being associated with each marker.
12. The non-transitory, computer-readable medium of claim 8 , wherein the first predictive model comprises one or more neural networks, the second predictive model comprises one or more support vector machines, the third predictive model comprises one or more multinomial logistic regression models, and the fourth predictive model comprises conditional random fields.
13. The non-transitory, computer-readable medium of claim 8 , wherein the computer-executable instructions cause the at least one computer processor to, for each sentence boundary detected in the utterance segment, use the words spoken by the first party that comprise a sentence associated with the sentence boundary as input to a fifth predictive model to identify whether the sentence associated with the sentence boundary is a question.
14. The non-transitory, computer-readable medium of claim 13 , wherein the fifth predictive model comprises one or more logistic regression models.
15. A system for analyzing an audio to capture semantic characteristics and non-semantic characteristics of the audio and corresponding relationships between the semantic characteristics and the non-semantic characteristics comprising:
at least one computer processor configured to:
segment the audio into a set of utterance segments and a set of noise segments by measuring one or more prosodic features of the audio to identify the set of utterance segments comprising segments of the audio containing a party speaking on the audio and the set of noise segments comprising segments of the audio containing the party not speaking on the audio;
to capture the semantic characteristics and the non-semantic characteristics for each of the utterance segments in the set of utterance segments:
have speech analytics performed on the utterance segment to identify words spoken by the party in the utterance segment;
extract lexical features with respect to each of the words spoken by the party in the utterance segment;
extract prosodic features across the utterance segment with respect to each of the words spoken by the party in the utterance segment and each inter-word boundary lying between two consecutive words spoken by the party in the utterance segment;
to perform laughter detection:
divide the utterance segment into a set of overlapping windows comprising feature frames; and
for each of the overlapping windows, use at least some of the prosodic features extracted at locations in the audio corresponding to each of the feature frames in the overlapping window as input to a first predictive model to identify whether laughter exists in the utterance segment;
to perform emotion detection:
compare each of the words spoken by the party in the utterance segment to an emotion lexicon comprising a list of emotions and words that are indicative of one or more emotions in the list of emotions to identify a first result comprising one or more emotions in the list of emotions for at least one of the words spoken by the party in the utterance segment;
use at least some of the prosodic features extracted for each of the words spoken by the party in the utterance segment as input to a second predictive model to identify a second result comprising one or more emotions expressed by the party in the utterance segment; and
use the first result and the second result as input to an ensemble comprising a third predictive model to arrive at a conclusion as to which of the one or more emotions for the first result and the one or more emotions for the second result is the emotion expressed by the party in the utterance segment; and
to perform sentence boundary detection:
use the lexical features and at least some of the prosodic features extracted for each of the words spoken by the party in the utterance segment and each of the inter-word boundaries lying between two consecutive words as input to a fourth predictive model to identify where in the utterance segment a sentence boundary exists with respect to each of the inter-word boundaries lying between two consecutive words; and
construct a transcript identifying the semantic characteristics as the words spoken by the party in the audio and the non-semantic characteristics as occurrences of laughter, emotions, and sentence boundaries and the corresponding relationships between the semantic characteristics and the non-semantic characteristics as locations of the words and the occurrences of laughter, emotions, and sentence boundaries within the audio relative to one another.
16. The system of claim 15 , wherein the at least one computer processor is configured to:
segment a second audio into a set of utterance segments and a set of noise segments by measuring one or more prosodic features of the second audio to identify the set of utterance segments comprising segments of the second audio containing a second party speaking on the second audio and the set of noise segments comprising segments of the second audio containing the second party not speaking on the second audio;
to capture semantic characteristics and non-semantic characteristics for each of the utterance segments in the set of utterance segments:
perform speech analytics on the utterance segment to identify words spoken by the second party in the utterance segment;
extract lexical features with respect to each of the words spoken by the second party in the utterance segment;
extract prosodic features across the utterance segment with respect to each of the words spoken by the second party in the utterance segment and each inter-word boundary lying between two consecutive words spoken by the second party in the utterance segment;
to perform laughter detection:
divide the utterance segment into a set of overlapping windows comprising feature frames; and
for each of the overlapping windows, use at least some of the prosodic features extracted at locations in the second audio corresponding to each of the feature frames in the overlapping window as input to the first predictive model to identify whether laughter exists in the utterance segment;
to perform emotion detection:
compare each of the words spoken by the second party in the utterance segment to the emotion lexicon to identify a first result comprising one or more emotions in the list of emotions for at least one of the words spoken by the second party in the utterance segment;
use at least some of the prosodic features extracted for each of the words spoken by the second party in the utterance segment as input to the second predictive model to identify a second result comprising one or more emotions expressed by the second party in the utterance segment; and
use the first result and the second result as input to the ensemble to arrive at a conclusion as to which of the one or more emotions for the first result and the one or more emotions for the second result is the emotion expressed by the second party in the utterance segment; and
to perform sentence boundary detection:
use the lexical features and at least some of the prosodic features extracted for each of the words spoken by the second party in the utterance segment and each of the inter-word boundaries lying between two consecutive words as input to the fourth predictive model to identify where in the utterance segment a sentence boundary exists with respect to each of the inter-word boundaries lying between two consecutive words;
construct a second transcript identifying the semantic characteristics as the words spoken by the second party in the second audio and the non-semantic characteristics as occurrences of laughter, emotions, and sentence boundaries and the corresponding relationships between the semantic characteristics and the non-semantic characteristics as locations of the words and the occurrences of laughter, emotions, and sentence boundaries within the second audio relative to one another; and
construct a combined transcript comprising the first transcript and the second transcript, wherein the combined transcript represents a conversation that had taken place between the first party and the second party.
17. The system of claim 16 , wherein the at least one computer processor is configured to:
display the combined transcript on a graphical user interface (“GUI”) on a computer monitor for a user, the GUI comprising a timeline representing the conversation that had taken place between the first party and the second party;
display the semantic characteristics as the words spoken by the first party and the words spoken by the second party on the GUI at locations with respect to the timeline proximate to a time when the words spoken by the first party occurred and the words spoken by the second party occurred during the conversation; and
display the non-semantic characteristics for the first party and the second party as a plurality of markers on the GUI in conjunction with the timeline to demonstrate the corresponding relationships between the semantic characteristics and the non-semantic characteristics for the first party and the second party, wherein each of the markers represents an occurrence of laugher by the first party or the second party, an emotion expressed by the first party or the second party, or a sentence boundary for a sentence spoken by the first party or the second party, and each marker is displayed on the GUI at a location with respect to the timeline proximate to a time when the occurrence of laugher by the first party or the second party, the emotion expressed by the first party or the second party, or the sentence boundary for the sentence spoken by the first party or the second party took place during the conversation.
18. The system of claim 17 , wherein the GUI identifies at least one of the first party and the second party as being associated with each marker.
19. The system of claim 15 , wherein the first predictive model comprises one or more neural networks, the second predictive model comprises one or more support vector machines, the third predictive model comprises one or more multinomial logistic regression models, and the fourth predictive model comprises conditional random fields.
20. The system of claim 15 , wherein the at least one computer processor is configured to, for each sentence boundary detected in the utterance segment, use the words spoken by the first party that comprise a sentence associated with the sentence boundary as input to a fifth predictive model to identify whether the sentence associated with the sentence boundary is a question.
21. The system of claim 20 , wherein the fifth predictive model comprises one or more logistic regression models.