Building a pragmatic action-item system
A method includes obtaining a transcript of multiple dialog acts during a multi-party communication session and extracting, from the transcript, a plurality of extractive action items each associated with a task expected to be completed within a timeframe to begin after the multi-party communication session ends. The method also includes generating, using an abstractive action item identification model configured to receive the extractive action items extracted from the transcript, one or more abstractive action items. Each abstractive action item is associated with a respective group of one or more of the extractive action items that are associated with a same respective task. For each of the one or more abstractive action items, the method also includes presenting, in one or more graphical user interfaces, information related to the corresponding abstractive action item.
1 . A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:
as each particular dialog act of multiple dialog acts during a multi-party communication session occurs:
obtaining corresponding audio data representing the particular dialog act;
processing, using a trained automatic speech recognition module, the corresponding audio data to obtain a corresponding transcript of the particular dialog act;
extracting, using an action item extractor model, from the corresponding transcript of the particular dialog act, a corresponding extractive action item associated with a task expected to be completed within a timeframe to begin after the multi-party communication session ends, the action item extractor model comprising a first machine learning (ML) model trained to:
receive, as input, the corresponding transcript of the particular dialog act and corresponding transcripts of a respective sequence of dialog acts occurring before and after the particular dialog act;
process, using a tokenizer, the corresponding transcript of the particular dialog act to generate a corresponding sequence of tokens representing the particular dialog act; and
process the corresponding sequence of tokens to generate, as output, a corresponding classification result indicating whether the particular dialog act includes an action item or does not include an action item;
generating, using an abstractive action item identification model configured to receive the corresponding extractive action items extracted by the action item extractor model, one or more abstractive action items, at least one abstractive action item of the one or more abstractive action items representing a respective group of two or more of the corresponding extractive action items, each extractive action item of the respective group of two or more extractive action items representing a respective aspect of a same respective action item task, at least two of the respective group of two or more extractive action items associated with different dialog acts, the abstractive action item identification model comprising a second ML model trained using a plurality of training transcripts, each training transcript of the plurality of training transcripts comprising:
a corresponding sequence of training dialog acts; and
one or more corresponding ground-truth abstraction actions for the corresponding sequence of training dialog acts, at least one ground-truth abstraction action of the one or more corresponding ground-truth abstraction actions indicating a span of sequential dialog acts that includes a first dialog act and a second dialog act both associated with a single task, and one or more intervening dialog acts between the first dialog act and the second dialog act that are not associated with the single task;
generating, using a text generation model, a summary of the at least one abstractive action item representing the respective group of two or more of the extractive action items based on lexical features of the respective group of two or more of the extractive action items, the summary concisely describing the respective action item task to be completed;
for each particular abstractive action item of the one or more abstractive action items, presenting, during the multi-party communication session in one or more graphical user interfaces, information related to the particular abstractive action item;
receiving user feedback indicating whether or not the presented information related to the particular abstractive action item is accurate, the user feedback corresponding to a positive or a negative training label for the presented information; and
fine-tuning the abstractive action item identification model based on the user feedback.
2 . The computer-implemented method of claim 1 , wherein the first ML model comprises a pre-trained Bidirectional Encoder representations from Transformers (BERT) model.
3 . The computer-implemented method of claim 1 , wherein the first ML model comprises an extended transformer construction (ETC) model having a global-local attention mechanism.
4 . The computer-implemented method of claim 1 , wherein each training transcript further comprises one or more ground-truth abstractive action items, each ground-truth abstractive action item indicating a respective span of sequential dialog acts from the sequence of training dialog acts that collectively indicate a respective task to be completed, one or more particular individuals responsible for completing the respective task, and a due date for completing the respective task.
5 . The computer-implemented method of claim 4 , wherein an initial dialog act and a last dialog act in the respective span of sequential dialog acts indicated by the ground-truth abstractive action item are each associated with extractive action items.
6 . The computer-implemented method of claim 5 , wherein at least one dialog act between an initial dialog act and a last dialog act in the respective span of sequential dialog acts indicated by the ground-truth abstractive item is not associated with an extractive action item.
7 . The computer-implemented method of claim 1 , wherein generating the one or more abstractive action items comprises clustering the corresponding extractive action items into one or more coherent groups, each coherent group forming a respective one of the one or more abstractive action items.
8 . The computer-implemented method of claim 1 , wherein presenting the information related to the at least one abstractive action item comprises presenting the summary of the at least one abstractive action item.
9 . The computer-implemented method of claim 8 , wherein the operations further comprise:
while presenting the summary of the at least one abstractive action item, receiving feedback from one or more users indicating whether or not the summary is accurate; and
updating the text generation model based on the received feedback.
10 . The computer-implemented method of claim 1 , wherein the operations further comprise:
predicting, using an assignee prediction model, based on the respective group of the two or more of the extractive action items associated with the same respective action item task, one or more individuals responsible for completing the respective action item task of the at least one abstractive action item,
wherein presenting the information related to the at least one abstractive action item comprises presenting the predicted one or more individuals responsible for completing the respective action item task.
11 . The computer-implemented method of claim 10 , wherein the operations further comprise:
while presenting the predicted one or more individuals, receiving feedback from one or more users indicating whether or not the prediction of the one or more individuals responsible for completing the respective action item task is accurate; and
updating the assignee prediction model based on the received feedback.
12 . The computer-implemented method of claim 1 , wherein the operations further comprise:
predicting, using a due-date prediction model, based on the respective group of the two or more of the extractive action items associated with the same respective action item task, a due date for completing the respective action item task of the at least one abstractive action item,
wherein presenting the information related to the at least one abstractive action item comprises presenting the predicted due date for completing the respective action item task.
13 . The computer-implemented method of claim 12 , wherein the operations further comprise:
while presenting the predicted due date, receiving feedback from one or more users indicating whether or not the prediction of the due date for completing the respective action item task is accurate; and
updating an assignee prediction model based on the received feedback.
14 . A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations comprising:
as each particular dialog act of multiple dialog acts during a multi-party communication session occurs:
obtaining corresponding audio data representing the particular dialog act;
processing, using a trained automatic speech recognition module, the corresponding audio data to obtain a corresponding transcript of the particular dialog act;
extracting, using an action item extractor model, from the corresponding transcript of the particular dialog act, a corresponding extractive action item associated with a task expected to be completed within a timeframe to begin after the multi-party communication session ends, the action item extractor model comprising a first machine learning (ML) model trained to:
receive, as input, the corresponding transcript of the particular dialog act and corresponding transcripts of a respective sequence of dialog acts occurring before and after the particular dialog act;
process, using a tokenizer, the corresponding transcript of the particular dialog act to generate a corresponding sequence of tokens representing the particular dialog act; and
process the corresponding sequence of tokens to generate, as output, a corresponding classification result indicating whether the particular dialog act includes an action item or does not include an action item;
generating, using an abstractive action item identification model configured to receive the extractive action items extracted by the action item extractor model, one or more abstractive action items, at least one abstractive action item of the one or more abstractive action items representing a respective group of two or more of the corresponding extractive action items, each extractive action item of the respective group of two or more extractive action items representing a respective aspect of a same respective action item task, at least two of the respective group of two or more extractive action items associated with different dialog acts, the abstractive action item identification model comprising a second ML model trained using a plurality of training transcripts, each training transcript of the plurality of training transcripts comprising:
a corresponding sequence of training dialog acts; and
one or more corresponding ground-truth abstraction actions for the corresponding sequence of training dialog acts, at least one ground-truth abstraction action of the one or more corresponding ground-truth abstraction actions indicating a span of sequential dialog acts that includes a first dialog act and a second dialog act both associated with a single task, and one or more intervening dialog acts between the first dialog act and the second dialog act that are not associated with the single task;
generating, using a text generation model, a summary of the at least one abstractive action item representing the respective group of two or more of the extractive action items based on lexical features of the respective group of two or more of the extractive action items, the summary concisely describing the respective action item task to be completed;
for each particular abstractive action item of the one or more abstractive action items, presenting, during the multi-party communication session in one or more graphical user interfaces, information related to the particular abstractive action item
receiving user feedback indicating whether or not the presented information related to the particular abstractive action item is accurate, the user feedback corresponding to a positive or a negative training label for the presented information; and
fine-tuning the abstractive action item identification model based on the user feedback.
15 . The system of claim 14 , wherein the first ML model comprises a pre-trained Bidirectional Encoder representations from Transformers (BERT) model.
16 . The system of claim 14 , wherein the first ML model comprises an extended transformer construction (ETC) model having a global-local attention mechanism.
17 . The system of claim 14 , wherein each training transcript further comprises one or more ground-truth abstractive action items, each ground-truth abstractive action item indicating a respective span of sequential dialog acts from the sequence of training dialog acts that collectively indicate a respective task to be completed, one or more particular individuals responsible for completing the respective task, and a due date for completing the respective task.
18 . The system of claim 17 , wherein an initial dialog act and a last dialog act in the respective span of sequential dialog acts indicated by the ground-truth abstractive action item are each associated with extractive action items.
19 . The system of claim 18 , wherein at least one dialog act between an initial dialog act and a last dialog act in the respective span of sequential dialog acts indicated by the ground-truth abstractive item is not associated with an extractive action item.
20 . The system of claim 14 , wherein generating the one or more abstractive action items comprises clustering the corresponding extractive action items into one or more coherent groups, each coherent group forming a respective one of the one or more abstractive action items.
21 . The system of claim 14 , wherein presenting the information related to the at least one abstractive action item comprises presenting the summary of the at least one abstractive action item.
22 . The system of claim 21 , wherein the operations further comprise:
while presenting the summary of the at least one abstractive action item, receiving feedback from one or more users indicating whether or not the summary is accurate; and
updating the text generation model based on the received feedback.
23 . The system of claim 14 , wherein the operations further comprise:
predicting, using an assignee prediction model, based on the respective group of the two or more of the extractive action items associated with the same respective action item task, one or more individuals responsible for completing the respective action item task of the at least one abstractive action item,
wherein presenting the information related to the at least one abstractive action item comprises presenting the predicted one or more individuals responsible for completing the respective action item task.
24 . The system of claim 23 , wherein the operations further comprise:
while presenting the predicted one or more individuals, receiving feedback from one or more users indicating whether or not the prediction of the one or more individuals responsible for completing the respective action item task is accurate; and
updating the assignee prediction model based on the received feedback.
25 . The system of claim 14 , wherein the operations further comprise:
predicting, using a due-date prediction model, based on the respective group of the two or more of the extractive action items associated with the same respective action item task, a due date for completing the respective action item task of the at least one abstractive action item,
wherein presenting the information related to the at least one abstractive action item comprises presenting the predicted due date for completing the respective action item task.
26 . The system of claim 25 , wherein the operations further comprise:
while presenting the predicted due date, receiving feedback from one or more users indicating whether or not the prediction of the due date for completing the respective action item task is accurate; and
updating an assignee prediction model based on the received feedback.