Granted Patent B2

US 10,592,611 · App. 15/332,766 · Granted Mar 17, 2020

Inventors: Jesse Vig (Los Altos, CA); Harish Arsikere (Bangalore, IN); Margaret H. Szymanski (Santa Clara, CA); Luke R. Plurkowski (Redwood City, CA); Kyle D. Dent (San Carlos, CA); Daniel G. Bobrow (Palo Alto, CA); Daniel Davies (Palo Alto, CA); Eric Saund (San Carlos, CA)

Assignee: Conduent Business Services, LLC

G06F17/279G06F17/277G10L15/26G10L25/48H04M3/51H04M2201/40H04M2203/357

Abstract

Claims (133)

1. A computer-implemented method for extracting conversational structure from a spoken conversation, comprising:

obtaining, by a computer system comprising a set of processors, a voice record of the spoken conversation;

dividing the voice record into at least three sequential utterances spoken by two different speakers;

extracting a lexical feature from a respective utterance of the three sequential utterances in the voice record using an automatic speech recognition;

extracting an acoustic feature from a respective utterance of the three sequential utterances in the voice record; and

determining, via a machine learning method and based on the extracted lexical and acoustic features, a coarse-level conversational structure of the spoken conversation, which involves:

determining a plurality of fine-level conversational activities based on the three sequential utterances, wherein a respective fine-level conversational activity is an activity identified based on respective words spoken by the two different speakers;

determining, based on the plurality of fine-level conversational activities, a plurality of coarse-level conversational activities, wherein a respective coarse-level conversational activity of the plurality of coarse-level conversational activities indicates a phase of the spoken conversation that includes multiple fine-level conversational activities of the plurality of fine-level conversational activities; and

generating the coarse-level conversational structure from the plurality of coarse-level conversational activities, wherein the coarse-level conversational structure indicates a high-level structure of the spoken conversation spanning the three sequential utterances.

2. The method of claim 1 :

wherein extracting the lexical feature further comprises generating a textual transcript of the voice record of the spoken conversation;

wherein extracting the acoustic feature further comprises identifying, based on the extracted acoustic feature and the textual transcript, three fine-level conversational activities associated respectively with the three sequential utterances; and

wherein determining the coarse-level conversational structure further comprises:

identifying that a first and a second fine-level conversational activities of the three fine-level conversational activities correspond to a first coarse-level conversational activity; and

identifying that a third fine-level activity of the three fine-level conversational activities transitions to a second coarse-level activity, which is different from the first coarse-level conversational activity.

3. The method of claim 1 , wherein the spoken conversation is a customer service conversation, and wherein the plurality of coarse-level conversational activities include one or more of:

opening;

detail gathering;

equipment identification;

security questions;

problem articulation;

diagnostics;

fix deployment;

customer satisfaction questions;

hold;

transfer;

pre-closing; and

closing.

4. The method of claim 1 , wherein the extracted acoustic feature includes one or more of:

speaking pitch;

speaking intensity;

timing or length of an utterance;

timing of silence or pauses;

overlap of utterances;

repetition of phrases, words, or word fragments;

speaking rhythm;

speaking rate;

speaking intonation;

laughter;

a Mel-frequency cepstral coefficient (MFCC); and

a derived acoustic feature.

5. The method of claim 1 , further comprising determining, via the machine learning method and based on the extracted lexical feature and acoustic feature, a fine-level activity structure of the spoken conversation, wherein the fine-level activity structure indicates a structure of one of the three sequential utterances.

6. The method of claim 5 , further comprising determining, via the machine learning method and based on the extracted lexical feature and acoustic feature, one or more intermediate-level structures of the spoken conversation.

7. The method of claim 5 , wherein the fine-level activity structure indicates a fine-level activity including one or more of:

an information request;

a clarification request;

a repetition request;

an action request;

pointing;

a future action request;

an alignment request;

a continuer;

a confirmation;

a sequence closer;

a correction;

information provision;

reporting activity status;

waiting;

reporting a future event; and

reciprocity.

8. The method of claim 1 :

wherein the machine learning method comprises a sequence model; and

wherein determining the coarse-level conversational structure further comprises tracking, using the sequence model, a global conversational state within the coarse-level conversational structure.

9. The method of claim 1 , further comprising:

computing for a user, via a business intelligence platform, an aggregate statistic, comprising a distribution over activities, categories, and/or entities, from a plurality of conversations that includes the spoken conversation; and

extracting for the user, via the business intelligence platform, targeted information about the spoken conversation.

10. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for extracting conversational structure from a spoken conversation, the method comprising:

obtaining a voice record of the spoken conversation;

dividing the voice record into at least three sequential utterances spoken by two different speakers;

extracting a lexical feature from a respective utterance of the at least three sequential utterances in the voice record using speech recognition;

extracting an acoustic feature from a respective utterance of the at least three sequential utterances in the voice record; and

determining, via a machine learning method and based on the extracted lexical and acoustic features, a coarse-level conversational structure of the spoken conversation, which involves:

determining a plurality of fine-level conversational activities based on the three sequential utterances, wherein a respective fine-level conversational activity is an activity identified based on respective words spoken by the two different speakers;

determining, based on the plurality of fine-level conversational activities, a plurality of coarse-level conversational activities, wherein a respective coarse-level conversational activity of the plurality of coarse-level conversational activities indicates a phase of the spoken conversation that includes multiple fine-level conversational activities of the plurality of fine-level conversational activities; and

generating the coarse-level conversational structure from the plurality of coarse-level conversational activities, wherein the coarse-level conversational structure indicates a high-level structure of the spoken conversation spanning the at least three sequential utterances.

11. The non-transitory computer-readable storage medium of claim 10 :

wherein extracting the lexical feature further comprises generating a textual transcript of the voice record of the spoken conversation;

wherein extracting the acoustic feature further comprises identifying, based on the extracted acoustic feature and the textual transcript, three fine-level conversational activities associated respectively with the three sequential utterances; and

wherein determining the coarse-level conversational structure further comprises:

identifying that a first and a second fine-level conversational activities of the three fine-level conversational activities correspond to a first coarse-level conversational activity; and

identifying that a third fine-level activity of the three fine-level conversational activities transitions to a second coarse-level activity, which is different from the first coarse-level conversational activity.

12. The non-transitory computer-readable storage medium of claim 10 , wherein the extracted acoustic feature includes one or more of:

speaking pitch;

speaking intensity;

timing or length of an utterance;

timing of silence or pauses;

overlap of utterances;

repetition of phrases, words, or word fragments;

speaking rhythm;

speaking rate;

speaking intonation;

laughter;

a Mel-frequency cepstral coefficient (MFCC); and

a derived acoustic feature.

13. The non-transitory computer-readable storage medium of claim 10 , wherein the method further comprises determining, via the machine learning method and based on the extracted lexical feature and acoustic feature, a fine-level activity structure of the spoken conversation, wherein the fine-level activity structure indicates a structure of one of the three sequential utterances.

14. The non-transitory computer-readable storage medium of claim 13 , wherein the method further comprises determining, via the machine learning method and based on the extracted lexical feature and acoustic feature, one or more intermediate-level structures of the spoken conversation.

15. The non-transitory computer-readable storage medium of claim 10 :

wherein the machine learning method comprises a sequence model; and

wherein determining the coarse-level conversational structure further comprises tracking, using the sequence model, a global conversational state within the coarse-level conversational structure.

16. A computing system for extracting conversational structure from a spoken conversation, the system comprising:

a set of processors; and

a non-transitory computer-readable medium coupled to the set of processors storing instructions thereon that, when executed by the processors, cause the processors to perform a method for extracting conversational structure from a spoken conversation, the method comprising:

obtaining a voice record of the spoken conversation;

dividing the voice record into at least three sequential utterances spoken by two different speakers;

extracting a lexical feature from a respective utterance of the three sequential utterances in the voice record using an automatic speech recognition;

extracting an acoustic feature from a respective utterance of the three sequential utterances the voice record; and

determining, via a machine learning method and based on the extracted lexical feature and acoustic feature, a coarse-level conversational structure of the spoken conversation, which involves:

determining a plurality of fine-level conversational activities based on the three sequential utterances, wherein a respective fine-level conversational activity is an activity identified based on respective words spoken by the two different speakers;

determining, based on the plurality of fine-level conversational activities, a plurality of coarse-level conversational activities, wherein a respective coarse-level conversational activity of the plurality of coarse-level conversational activities indicates a phase of the spoken conversation that includes multiple fine-level conversational activities of the plurality of fine-level conversational activities; and

generating the coarse-level conversational structure from the plurality of coarse-level conversational activities, wherein the coarse-level conversational structure indicates a high-level structure of the spoken conversation spanning the three sequential utterances.

17. The computing system of claim 16 :

wherein extracting the lexical feature from the voice record further comprises generating a textual transcript of the voice record of the spoken conversation;

wherein extracting the acoustic feature from the voice record further comprises identifying, based on the extracted acoustic feature and the textual transcript, three fine-level conversational activities associated respectively with the three sequential utterances; and

wherein determining the coarse-level conversational structure of the spoken conversation further comprises:

identifying that a first and a second fine-level conversational activities of the three fine-level conversational activities correspond to a first coarse-level conversational activity; and

identifying that a third fine-level activity of the three fine-level conversational activities transitions to a second coarse-level activity, which is different from the first coarse-level conversational activity.

18. The computing system of claim 16 , wherein the extracted acoustic feature includes one or more of:

speaking pitch;

speaking intensity;

timing or length of an utterance;

timing of silence or pauses;

overlap of utterances;

repetition of phrases, words, or word fragments;

speaking rhythm;

speaking rate;

speaking intonation; and

laughter;

a Mel-frequency cepstral coefficient (MFCC); and

a derived acoustic feature.

19. The computing system of claim 16 , wherein the method further comprises determining, via the machine learning method and based on the extracted lexical feature and acoustic feature, a fine-level activity structure of the spoken conversation, wherein the fine-level activity structure indicates a structure of one of the three sequential utterances.

20. The computing system of claim 16 :

wherein the machine learning method comprises a sequence model; and

wherein determining the coarse-level conversational structure further comprises tracking, using the sequence model, a global conversational state within the coarse-level conversational structure.

Assignments (9)

Continuity (1)

Cited By (1)

US 12,626,706