Multitask learning as question answering

1. A system for natural language processing, the system comprising:

one or more processors; and

a memory storing computer-executable instructions, which when executed by the one or more processors, cause the system to perform operations comprising:

encoding, using an input layer, first words from a context and second words from a question;

encoding and decoding, using a self-attention based transformer, an output of the input layer;

further encoding, using a first bi-directional long-term short-term memory (biLSTM), an output of the encoding;

generating, using a long-term short-term memory (LSTM), a context-adjusted hidden state from an output of the decoding and a first hidden state of a first time step;

generating, using an attention network, first attention weights based on an output of the first biLSTM and an output of the LSTM;

generating, using a vocabulary layer, a distribution over third words in a vocabulary based on the first attention weights;

generating, using a context layer, a distribution over the first words from the context based on the first attention weights; and

generating, using a switch, a weighting between the distribution over the third words from the vocabulary and the distribution over the first words from the context;

generating, using the switch, a composite distribution based on the weighting between the distribution over the third words from the vocabulary and the distribution over the first words from the context; and

selecting, using the switch, a word for inclusion in an answer using the composite distribution.

2. The system of claim 1 , wherein the input layer comprises one or more of a linear layer, a second biLSTM, a coattention layer, and a third biLSTM.

3. The system of claim 2 , wherein the coattention layer:

generates an affinity matrix between encoded representations of the context and the question;

generates second attention weights based on the affinity matrix; and

generates weighted sums of the context and the question using the second attention weights.

4. The system of claim 1 , wherein the vocabulary layer comprises:

a tanh layer for generating a second hidden state for a second time step subsequent to the first time step based on the first attention weights, the further encoding of the output of the encoding, and the context-adjusted hidden state; and

a softmax layer for generating the distribution over the third words in the vocabulary based on the second hidden state.

5. The system of claim 1 , wherein the self-attention based transformer, the LSTM, the attention network, the vocabulary layer, the context layer, and the switch iteratively select each word for the answer.

6. The system of claim 1 , wherein the self-attention based transformer comprises a plurality of transformer layers, each of the plurality of transformer layers comprising an encoder portion having a first multi-head self-attention network and a decoder portion having a second multi-head self-attention network and a third multi-head attention network.

7. The system of claim 1 , wherein the system is trained using a hybrid training strategy where the system is first trained against a plurality of task types using a sequential training strategy and is then trained against the plurality of task types using a joint training strategy.

8. The system of claim 7 , wherein each of the plurality of task types is a language translation task type, a classification task type, or a question answering task type.

9. A method for natural language processing, the method comprising:

receiving first words from a context and second words from a question;

encoding, using an input layer, the first words from the context and the second words from the question to generate a first encoding;

further encoding, using a self-attention based transformer, the first encoding to generate a second encoding of the first words from the context and the second words from the question;

decoding, using the self-attention based transformer, the second encoding to generate a decoder output;

further encoding, using a first bi-directional long-term short-term memory (biLSTM), the second encoding to generate a third encoding;

generating, using a long-term short-term memory (LSTM), a context-adjusted hidden state from the third encoding and a first hidden state of a first time step;

generating, using an attention network, first attention weights based on the third encoding and the context-adjusted hidden state;

generating, using a vocabulary layer, a distribution over third words in a vocabulary based on the first attention weights;

generating, using a context layer, a distribution over the first words from the context based on the first attention weights;

generating, using a switch, a weighting between the distribution over the third words from the vocabulary and the distribution over the first words from the context;

generating, using the switch, a composite distribution based on the weighting between the distribution over the third words from the vocabulary and the distribution over the first words from the context; and

selecting, using the switch, a word for inclusion in an answer using the composite distribution.

10. The method of claim 9 , further comprising:

generating, using a coattention layer, an affinity matrix between encoded representations of the context and the question;

generating, using the coattention layer, second attention weights based on the affinity matrix; and

generating, using the coattention layer, weighted sums of the context and the question using the second attention weights.

11. The method of claim 9 , further comprising:

generating, using a tanh layer of the vocabulary layer, a second hidden state of a second time step subsequent to the first time step based on the first attention weights, the second encoding, and the context-adjusted hidden state; and

generating, using a softmax layer of the vocabulary layer, the distribution over the third words in the vocabulary based on the second hidden state.

12. The method of claim 9 , further comprising iteratively selecting each word for the answer using the self-attention based transformer, the LSTM, the attention network, the vocabulary layer, the context layer, and the switch.

13. The method of claim 9 , wherein:

generating the second encoding comprises using one or more encoding layers, wherein each encoding layer has a first multi-head self-attention network; and

generating the decoder output comprises using one or more decoding layers, each decoding layer having a second multi-head self-attention network and a third multi-head attention network.

14. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions which when executed by one or more processors associated with a computing device are adapted to cause the one or more processors to perform a method comprising:

receiving first words from a context and second words from a question;

encoding the first words from the context and the second words from the question to generate a first encoding;

further encoding, using self-attention, the first encoding to generate a second encoding of the first words from the context and the second words from the question;

decoding, using self-attention, the second encoding to generate a decoder output;

further encoding the second encoding to generate a third encoding;

generating a context-adjusted hidden state from the third encoding and a first hidden state of a first time step;

generating first attention weights based on the third encoding and the context-adjusted hidden state;

generating a distribution over third words in a vocabulary based on the first attention weights;

generating a distribution over the first words from the context based on the first attention weights;

generating a weighting between the distribution over the third words from the vocabulary and the distribution over the first words from the context;

generating a composite distribution based on the weighting between the distribution over the third words from the vocabulary and the distribution over the first words from the context; and

selecting a word for inclusion in an answer using the composite distribution.

15. The non-transitory machine-readable medium of claim 14 , wherein the method further comprises:

generating an affinity matrix between encoded representations of the context and the question;

generating second attention weights based on the affinity matrix; and

generating weighted sums of the context and the question using the second attention weights.

16. The non-transitory machine-readable medium of claim 14 , wherein the method further comprises:

generating a second hidden state of a second time step subsequent to the first time step based on the first attention weights, the second encoding, and the context-adjusted hidden state; and

generating the distribution over the third words in the vocabulary based on the second hidden state.

17. The non-transitory machine-readable medium of claim 14 , wherein the method further comprises iteratively selecting each word for the answer.

18. The non-transitory machine-readable medium of claim 14 , wherein:

generating the second encoding comprises using one or more encoding layers, wherein each encoding layer has a first multi-head self-attention network; and

generating the decoder output comprises using one or more decoding layers, each decoding layer having a second multi-head self-attention network and a third multi-head attention network.

19. The non-transitory machine-readable medium of claim 14 , wherein the method further comprises training using a hybrid training strategy, the hybrid training strategy includes first training against a plurality of task types using a sequential training strategy and then training against the plurality of task types using a joint training strategy.

20. The non-transitory machine-readable medium of claim 19 , wherein each of the plurality of task types is a language translation task type, a classification task type, or a question answering task type.