Multitask machine-learning model training and training data augmentation
Multitask machine-learning model training and training data augmentation techniques are described. In one example, training is performed for multiple tasks simultaneously as part of training a multitask machine-learning model using question pairs. Examples of the multiple tasks include question summarization and recognizing question entailment. Further, a loss function is described that incorporates a parameter sharing loss that is configured to adjust an amount that parameters are shared between corresponding layers trained for the first and second tasks, respectively. In an implementation, training data augmentation techniques are also employed by synthesizing question pairs, automatically and without user intervention, to improve accuracy in model training.
1 . A method comprising:
receiving, by a processing device, training data including a first set of training data associated with a first task involving summarization and a second set of training data associated with a second task involving classification;
training, by the processing device, a multitask machine-learning model having a first set of layers using the first set of training data associated with the first task involving summarization and a second set of layers using the second set of training data associated with the second task involving classification, the training using a multitask learning loss function defining differing amounts of parameter sharing that decreases through successive said layers between the first and second set of layers associated with the first and second tasks, respectively; and
outputting, by the processing device, a result of processing a subsequent input by the trained multitask machine-learning model.
2 . The method as described in claim 1 , wherein the first task involving summarization is a question summarization task and the second task involving classification is a recognizing question entailment task.
3 . The method as described in claim 2 , wherein the training of the recognizing question entailment task includes learning entailment between a first question in the first set of training data and a second question in the second set of training data.
4 . The method as described in claim 3 , wherein the entailment is defined between first question and the second question such that answers to the second question provide complete and correct answers to the first question and answers to the first question provide correct answers to the second question.
5 . The method as described in claim 1 , wherein the first set of training data includes a first collection of questions and answers and the second set of training data includes a second collection of questions and answers.
6 . The method as described in claim 5 , wherein at least one of the first or second set of training data is configured as frequently asked questions (FAQ).
7 . The method as described in claim 1 , wherein the first set of layers are implemented using a first decoder of a multitask machine learning model and the second set of layers are implemented using a second decoder of the multitask machine learning model.
8 . The method as described in claim 7 , wherein the multitask machine learning model further includes a shared encoder for the first and second tasks.
9 . The method as described in claim 1 , wherein the multitask learning loss function defines a parameter sharing loss as increasing or decreasing between sequential pairs of the layers.
10 . The method as described in claim 1 , wherein the result is a summary of text of the subsequent input, the result generated as part of long question understanding through use of a trained multitask machine learning model.
11 . The method as described in claim 10 , wherein the subsequent input as part of the long question understanding involves a plurality of sentences.
12 . A system comprising:
a processing device; and
a non-transitory computer-readable storage medium storing instructions that, responsive to execution by the processing device, causes the processing device to perform operations including:
receiving training data including a first set of training data associated with a first task involving summarization and a second set of training data associated with a second task involving classification;
training a multitask machine-learning model having a shared encoder, a first set of layers of a summarization decoder using the first set of training data associated with the first task involving summarization and a second set of layers of a classification decoder using the second set of training data associated with the second task involving classification, the training using a multitask learning loss function defining differing amounts of parameter sharing between corresponding said layers of the summarization decoder and the classification decoder associated with the first and second tasks, respectively; and
outputting a result of processing a subsequent input by the trained multitask machine-learning model.
13 . The system as described in claim 12 , wherein the first task is a question summarization task and the second task is a recognizing question entailment task.
14 . The system as described in claim 13 , wherein the training of the recognizing question entailment task includes learning entailment between a first question in the first set of training data and a second question in the second set of training data.
15 . The system as described in claim 13 , wherein the entailment is defined between first question and the second question such that answers to the second question provide complete and correct answers to the first question and answers to the first question provide correct answers to the second question.
16 . The system as described in claim 12 , wherein the first set of training data includes a first collection of questions and answers and the second set of training data includes a second collection of questions and answers.
17 . One or more non-transitory computer-readable storage media storing instructions that, responsive to execution by a processing device, causes the processing device to perform operations comprising:
receiving training data including a first set of training data associated with a first task involving summarization and a second set of training data associated with a second task involving classification;
training a multitask machine-learning model having a first set of layers of a summarization decoder using the first set of training data associated with the first task involving summarization and a second set of layers of a classification decoder using the second set of training data associated with the second task involving classification, the training using a multitask learning loss function defining differing amounts of parameter sharing that decreases through successive said layers between the first and second set of layers associated with the first and second tasks, respectively; and
outputting a result of processing a subsequent input by the trained multitask machine-learning model.
18 . The one or more non-transitory computer-readable storage media as described in claim 17 , wherein the first task is a question summarization task and the second task is a recognizing question entailment task.
19 . The one or more non-transitory computer-readable storage media as described in claim 18 , wherein the training of the recognizing question entailment task includes learning entailment between a first question in the first set of training data and a second question in the second set of training data.
20 . The one or more non-transitory computer-readable storage media as described in claim 19 , wherein the entailment is defined between first question and the second question such that answers to the second question provide complete and correct answers to the first question and answers to the first question provide correct answers to the second question.
21 . The one or more non-transitory computer-readable storage media as described in claim 17 , wherein the first set of training data includes a first collection of questions and answers and the second set of training data includes a second collection of questions and answers.