Multi-task machine learning architectures and training procedures

1. A method performed on a computing device, the method comprising:

evaluating a plurality of candidate multi-task teacher instances of a multi-task machine learning model, each candidate multi-task teacher instance having one or more shared layers, a first task-specific layer that performs a first task, and a second task-specific layer that performs a second task;

based at least on the evaluating, selecting one or more first multi-task teacher instances of the plurality of candidate multi-task teacher instances for the first task and one or more second multi-task teacher instances of the plurality of candidate multi-task teacher instances for the second task; and

training a multi-task student instance of the multi-task machine learning model, the multi-task student instance having the one or more shared layers, the first task-specific layer, and the second task-specific layer, the training comprising using first outputs of the one or more first multi-task teacher instances to train the first task-specific layer of the multi-task student instance and using second outputs of the one or more second multi-task teacher instances to train the second task-specific layer of the multi-task student instance.

2. The method of claim 1 , further comprising:

training the multi-task student instance using an objective function defined based at least on the first outputs and the second outputs.

3. The method of claim 2 , the objective function being further defined based at least on labels associated with labeled data processed to obtain the first outputs and the second outputs.

4. The method of claim 1 , wherein the selecting comprises:

training the plurality of candidate multi-task teacher instances to perform the first task and the second task;

selecting the one or more first multi-task teacher instances based at least on accuracy of the one or more first multi-task teacher instances at performing the first task; and

selecting the one or more second multi-task teacher instances based at least on accuracy of the one or more second multi-task teacher instances at performing the second task.

5. The method of claim 4 , wherein the training comprises:

averaging first outputs of multiple first multi-task teacher instances to obtain first averages; and

training the multi-task student instance to reproduce the first averages.

6. The method of claim 5 , wherein the multi-task student instance is trained to reproduce at least two different probabilities for a given category on a given instance of training data.

7. The method of claim 1 , wherein the plurality of candidate multi-task teacher instances and the multi-task student instance each include an encoder configured to output context embedding vectors to the first task-specific layer and the second task-specific layer.

8. A system comprising:

a hardware processing unit; and

a storage resource storing computer-readable instructions which, when executed by the hardware processing unit, cause the system to:

access a multi-task student instance, the multi-task student instance being a trained model that has been trained using first outputs of one or more first multi-task teacher instances to train a first task-specific layer of the multi-task student instance and using second outputs of one or more second multi-task teacher instances to train a second task-specific layer of the multi-task student instance;

receive input data;

provide the input data to the multi-task student instance;

obtain a first task-specific result produced by the first task-specific layer of the multi-task student instance; and

output the first task-specific result,

wherein the multi-task student instance, the one or more first multi-task teacher instances, and the one or more second multi-task teacher instances are instances of a multi-task machine learning model having one or more shared layers, the first task-specific layer, and the second task-specific layer.

9. The system of claim 8 , wherein the multi-task student instance is a natural language processing model and the input data comprises input text.

10. The system of claim 9 , wherein the one or more shared layers include a lexicon encoder and a transformer encoder that have been trained to map textual tokens into a vector space.

11. The system of claim 10 , wherein the input text comprises a sentence and the first task-specific result characterizes a sentiment of the sentence as positive or negative.

12. The system of claim 10 , wherein the input text comprises a pair of sentences and the first task-specific result characterizes a semantic similarity of the pair of sentences.

13. The system of claim 10 , wherein the input text comprises a pair of sentences and the first task-specific result characterizes the pair of sentences as having an entailment relationship, a contradiction relationship, or a neutral relationship.

14. The system of claim 10 , wherein the input text comprises a query and the first task-specific result comprises scores that reflect relevance of a plurality of documents to the query.

15. The system of claim 14 , wherein the computer-readable instructions, when executed by the hardware processing unit, cause the system to:

rank individual documents relative to one another based at least on the scores.

16. One or more computer-readable storage media storing instructions which, when executed by one or more hardware processing units, cause the one or more hardware processing units to perform acts comprising:

evaluating a plurality of candidate multi-task teacher instances of a multi-task machine learning model, each candidate multi-task teacher instance having one or more shared layers, a first task-specific layer that performs a first task, and a second task-specific layer that performs a second task;

based at least on the evaluating, selecting one or more first multi-task teacher instances of the plurality of candidate multi-task teacher instances for the first task and one or more second multi-task teacher instances of the plurality of candidate multi-task teacher instances for the second task; and

training a multi-task student instance of the multi-task machine learning model, the multi-task student instance being a trained model having the one or more shared layers, the first task-specific layer, and the second task-specific layer, the training comprising using first outputs of the one or more first multi-task teacher instances to train the first task-specific layer of the multi-task student instance and using second outputs of the one or more second multi-task teacher instances to train the second task-specific layer of the multi-task student instance.

17. The one or more computer-readable storage media of claim 16 , wherein a pretraining stage has been performed on the one or more shared layers of the one or more first multi-task teacher instances and the one or more second multi-task teacher instances.

18. The one or more computer-readable storage media of claim 17 , wherein the one or more shared layers comprise an encoder configured to map natural language tokens into token embeddings in a vector space.

19. The one or more computer-readable storage media of claim 18 , wherein a tuning stage has been performed on the first task-specific layer and the second task-specific layer of the one or more first multi-task teacher instances and the first task-specific layer and the second task-specific layer of the one or more second multi-task teacher instances, the tuning stage using different task-specific sets of textual training data to tune the first task-specific layer and the second task-specific layer.

20. The one or more computer-readable storage media of claim 18 , the encoder being a transformer encoder.