Granted Patent B1

US 12,573,401 · App. 17/246,250 · Granted Mar 10, 2026

Inventor: Shahab Jalalvand (Summit, NJ)

Assignee: Interactions LLC

G10L15/26G06N3/08G10L15/02G10L15/04G10L15/063G10L25/30G10L2015/0631

Abstract

Claims (88)

1 . A computer-implemented method for generating a machine learning based model for transcribing audio inputs, the computer-implemented method comprising:

receiving a set of audio inputs;

determining a set of conversations from the set of audio inputs, wherein a conversation comprises a plurality of rounds, each round representing an exchange of utterances between participants of the conversation;

for each conversation, determining a respective dialogue state for each utterance of the conversation, wherein each respective dialogue state corresponds to a set of rounds of the conversation that are related, the dialogue state corresponding to a cluster of utterance embeddings obtained by encoding utterances using a hierarchical encoding model comprising

(1) a plurality of low-level encoding models, each low-level encoding model configured to generate a context-independent embedding for a dialog round, and

(2) a high-level encoding model configured to receive as input a set of context-independent embeddings for a dialog round and output a contextual embedding for the dialog round, wherein training the low-level encoding model and the high-level encoding model includes using a set of positive and negative examples, wherein a positive example includes a dialog round and a positive response generated based on an actual response to the dialog round, and a negative example includes the dialog round and a negative response generated based on a random response from a corpus of dialogs;

training a hierarchical language model for transcribing audio inputs of a conversation using the set of conversations, the hierarchical language model comprising a top-level language model and a plurality of lower-level language models, the training comprising:

training the top-level language model using sequences of the respective dialogue states determined for the set of conversations, each sequence of dialogue states for a particular conversation in the set of conversations, the top-level language model trained to predict a probability of occurrence of a particular dialogue state, given a sequence of previous dialogue states in a corresponding conversation, and

for each dialogue state, training a plurality of lower-level language models, wherein each lower language model is trained for a source of utterance, each lower language model trained using utterances from the source having that dialogue state, the lower-level language model trained to predict the probability of occurrence of a word based on an input utterance;

receiving a new audio input for a new conversation; and

executing the hierarchical language model to transcribe the new audio input, the executing comprising, predicting the particular dialogue state using the top-level language model, and using a lower-level language model trained for a source of utterance, the lower-level language model corresponding to the predicted dialogue state to predict a word based on an utterance of the new audio input.

2 . The computer-implemented method of claim 1 , wherein each conversation is between a particular agent and a particular user, and wherein determining the particular dialogue state for an utterance of the conversation comprises:

segmenting the conversation into a set of rounds, wherein each round represents a pair of consecutive utterances comprising an utterance by the particular agent and an utterance by the particular user;

performing clustering of the set of rounds to determine a plurality of clusters; and

associating each cluster in the plurality of clusters with the particular dialogue state.

3 . The computer-implemented method of claim 2 , wherein performing clustering of the set of rounds comprises:

determining a set of embeddings, wherein an embedding is for a round; and

performing clustering of the set of embeddings.

4 . The computer-implemented method of claim 2 , further comprising:

identifying each dialogue state with a set of keywords based on a subset of utterances having the dialogue state.

5 . The computer-implemented method of claim 4 , further comprising: determining the set of keywords for the particular dialogue state based on term frequency-inverse document frequency of keywords occurring in utterances having the particular dialogue state.

6 . The computer-implemented method of claim 5 , wherein the particular dialogue state is associated with a particular cluster in the plurality of clusters, wherein determining the set of keywords for the particular dialogue state comprises:

receiving a set of conversations;

for each conversation in the set of conversations, for each round in the conversation, splitting the round into an agent utterance and a user utterance;

extracting a first set of keywords from agent utterances belonging to the particular cluster;

extracting a second set of keywords from user utterances belonging to the particular cluster; and

combining the first set of keywords and the second set of keywords as the set of keywords.

7 . The computer-implemented method of claim 1 , wherein the top-level language model is an n-gram model trained on sequences of dialogue states.

8 . A non-transitory computer-readable storage medium storing a hierarchical language model comprising a top-level language model and a plurality of lower-level language models, wherein the hierarchical language model is trained using steps comprising:

receiving a set of audio inputs;

determining a set of conversations from the set of audio inputs, wherein a conversation comprises a plurality of rounds, each round representing an exchange of utterances between participants of the conversation;

for each conversation, determining a respective dialogue state for each utterance of the conversation, wherein each respective dialogue state corresponds to a set of rounds of the conversation that are related, the dialogue state corresponding to a cluster of utterance embeddings obtained by encoding utterances using a hierarchical encoding model comprising

(1) a plurality of low-level encoding models, each low-level encoding model configured to generate a context-independent embedding for a dialog round, and

(2) a high-level encoding model configured to receive as input a set of context-independent embeddings for a dialog round and output a contextual embedding for the dialog round, wherein training the low-level encoding model and the high-level encoding model includes using a set of positive and negative examples, wherein a positive example includes a dialog round and a positive response generated based on an actual response to the dialog round, and a negative example includes the dialog round and a negative response generated based on a random response from a corpus of dialogs;

training a hierarchical language model for transcribing audio inputs of a conversation using the set of conversations, the hierarchical language model comprising a top-level language model and a plurality of lower-level language models, the training comprising:

training the top-level language model using sequences of the respective dialogue states determined for the set of conversations, each sequence of dialogue states for a particular conversation in the set of conversations, the top-level language model trained to predict a probability of occurrence of a particular dialogue state, given a sequence of previous dialogue states in a corresponding conversation, and

for each dialogue state, training a plurality of lower-level language models, wherein each lower language model is trained for a source of utterance, each lower language model trained using utterances from the source having that dialogue state, the lower-level language model trained to predict the probability of occurrence of a word based on an input utterance;

receiving a new audio input for a new conversation; and

executing the hierarchical language model to transcribe the new audio input, the executing comprising, predicting the particular dialogue state using the top-level language model, and using a lower-level language model trained for a source of utterance, the lower-level language model corresponding to the predicted dialogue state to predict a word based on an utterance of the new audio input.

9 . The non-transitory computer-readable storage medium of claim 8 , wherein each conversation is between a particular agent and a particular user, and wherein determining a corresponding dialogue state for each utterance of the conversation comprises:

segmenting the conversation into a set of rounds, wherein each round represents a pair of consecutive utterances comprising an utterance by the particular agent and an utterance by the particular user;

performing clustering of the set of rounds to determine a plurality of clusters; and

associating each cluster in the plurality of clusters with the corresponding dialogue state.

10 . The non-transitory computer-readable storage medium of claim 9 , wherein performing clustering of the set of rounds comprises:

determining a set of embeddings, wherein an embedding is for a round; and

performing clustering of the set of embeddings.

11 . The non-transitory computer-readable storage medium of claim 9 , wherein the hierarchical language model is trained using steps further comprising:

identifying each dialogue state with a set of keywords based on a subset of utterances having the dialogue state.

12 . The non-transitory computer-readable storage medium of claim 11 , wherein the hierarchical language model is trained using steps further comprising:

determining the set of keywords for a particular dialogue state based on term frequency-inverse document frequency of keywords occurring in utterances having the dialogue state.

13 . The non-transitory computer-readable storage medium of claim 12 , wherein the particular dialogue state is associated with a particular cluster in the plurality of clusters, wherein determining the set of keywords for the particular dialogue state comprises:

receiving a set of conversations;

for each conversation in the set of conversations, for each round in the conversation, splitting the round into an agent utterance and a user utterance;

extracting a first set of keywords from agent utterances belonging to the particular cluster;

extracting a second set of keywords from user utterances belonging to the particular cluster; and

combining the first set of keywords and the second set of keywords as the set of keywords.

14 . The non-transitory computer-readable storage medium of claim 8 , wherein the top-level language model is an n-gram model trained on sequences of dialogue states.

15 . A computer system comprising:

a computer processor; and

a non-transitory computer-readable storage medium storing instructions that when executed by the computer processor cause the computer processor to perform steps comprising:

receiving a set of audio inputs;

receiving a set of audio inputs;

determining a set of conversations from the set of audio inputs, wherein a conversation comprises a plurality of rounds, each round representing an exchange of utterances between participants of the conversation;

for each conversation, determining a respective dialogue state for each utterance of the conversation, wherein each respective dialogue state corresponds to a set of rounds of the conversation that are related, the dialogue state corresponding to a cluster of utterance embeddings obtained by encoding utterances using a hierarchical encoding model comprising

(1) a plurality of low-level encoding models, each low-level encoding model configured to generate a context-independent embedding for a dialog round, and

(2) a high-level encoding model configured to receive as input a set of context-independent embeddings for a dialog round and output a contextual embedding for the dialog round, wherein training the low-level encoding model and the high-level encoding model includes using a set of positive and negative examples, wherein a positive example includes a dialog round and a positive response generated based on an actual response to the dialog round, and a negative example includes the dialog round and a negative response generated based on a random response from a corpus of dialogs;

training a hierarchical language model for transcribing audio inputs of a conversation using the set of conversations, the hierarchical language model comprising a top-level language model and a plurality of lower-level language models, the training comprising:

training the top-level language model using sequences of the respective dialogue states determined for the set of conversations, each sequence of dialogue states for a particular conversation in the set of conversations, the top-level language model trained to predict a probability of occurrence of a particular dialogue state, given a sequence of previous dialogue states in a corresponding conversation, and

for each dialogue state, training a plurality of lower-level language models, wherein each lower language model is trained for a source of utterance, each lower language model trained using utterances from the source having that dialogue state, the lower-level language model trained to predict the probability of occurrence of a word based on an input utterance;

receiving a new audio input for a new conversation; and

executing the hierarchical language model to transcribe the new audio input, the executing comprising, predicting the particular dialogue state using the top-level language model, and using a lower-level language model trained for a source of utterance, the lower-level language model corresponding to the predicted dialogue state to predict a word based on an utterance of the new audio input.

16 . The computer system of claim 15 , wherein each conversation is between a particular agent and a particular user, and wherein determining the particular dialogue state for an utterance of the conversation comprises:

segmenting the conversation into a set of rounds, wherein each round represents a pair of consecutive utterances comprising an utterance by the particular agent and an utterance by the particular user;

performing clustering of the set of rounds to determine a plurality of clusters; and

associating each cluster in the plurality of clusters with the particular dialogue state.

17 . The computer system of claim 16 , wherein performing clustering of the set of rounds comprises:

determining a set of embeddings, wherein an embedding is for a round; and

performing clustering of the set of embeddings.

18 . The computer system of claim 16 , wherein the instructions further cause the computer processor to perform steps comprising:

identifying each dialogue state with a set of keywords based on a subset of utterances having the dialogue state.

19 . The computer system of claim 18 , wherein the instructions further cause the computer processor to perform steps comprising:

determining the set of keywords for the particular dialogue state based on term frequency-inverse document frequency of keywords occurring in utterances having the particular dialogue state.

20 . The computer system of claim 19 , wherein the particular dialogue state is associated with a particular cluster in the plurality of clusters, wherein determining the set of keywords for the particular dialogue state further comprises:

receiving a set of conversations;

for each conversation in the set of conversations, for each round in the conversation, splitting the round into an agent utterance and a user utterance;

extracting a first set of keywords from agent utterances belonging to the particular cluster;

extracting a second set of keywords from user utterances belonging to the particular cluster; and

combining the first set of keywords and the second set of keywords as the set of keywords.

Assignments (4)

References Cited (14)