Efficient transformer language models with disentangled attention and multi-step decoding
Systems and methods are provided for facilitating the building and use of natural language understanding models. The systems and methods identify a plurality of tokens and use them to generate one or more pre-trained natural language models using a transformer. The transformer disentangles the content embedding and positional embedding in the computation of its attention matrix. Systems and methods are also provided to facilitate self-training of the pre-trained natural language model by utilizing multi-step decoding to better reconstruct masked tokens and improve pre-training convergence.
1. A computing system configured to account for position bias while encoding data with a transformer to improve pre-training convergence of the transformer, the computing system comprising:
one or more processors; and
one or more computer-readable hardware storage devices that store computer executable instructions that are executable by the one or more processors to cause the computer system to at least:
identify a plurality of tokens to be encoded from a sequence;
obtain a transformer that includes an encoder, the encoder including a self-attention sub-layer, a feed forward sub-layer, and a final encoding layer;
embed the plurality of tokens to generate input data;
apply the input data to the encoder; and
generate and apply an attention score at the self-attention sub-layer, by at least disentangling position bias embedding from content embedding associated with the plurality of the tokens, the attention weight comprising a summation of:
a first attention score component comprising a product of a content embedding of a first token in a token pair, a first learn-able content parameter, a second learn-able content parameter, and a content embedding of a second token in a token pair;
a second attention score component comprising a product of the content embedding of the first token in the token pair, the first learn-able content parameter, a first learnable position parameter, and a relative position embedding of the first and second tokens in a token pair;
a third attention score component comprising a product of the relative position embedding of the first and second tokens in the token pair, a second learn-able position parameter, the second learn-able content parameter, and the second content embedding of a second token in a token pair; and
obtain and apply output of the final encoding layer as additional input to the final encoding layer to facilitate an improvement of the pre-training convergence of the transformer.
2. The computing system of claim 1 , wherein the attention weight includes the summation of a fourth attention score component comprising a product of an absolute position embedding of the first token in the token pair, the first learn-able position parameter, the second learn-able position parameter, and the relative positional embedding of the first and second tokens in the token pair.
3. The computing system of claim 1 , wherein the encoder includes a plurality of encoding layers within the encoder, and wherein the final encoding layer of the encoder is a task specific decoding layer.
4. The computing system of claim 3 , wherein the computing system applies the output of a decoding layer as additional input to the decoding layer for one or more iterations in order to generate new output from the decoding layer.
5. The computing system of claim 4 , wherein the computing system applies one or more hidden vector outputs associated with masked tokens from the decoding layer as additional input to the decoding layer to generate new output from the decoding layer.
6. The computing system of claim 5 , wherein the computing system applies a query vector output of the decoding layer as additional input to the decoding layer for one or more iterations in order to generate new output from the final decoding layer.
7. The computing system of claim 1 , wherein the computing system refrains from applying position bias embedding prior to the encoder.
8. The computing system of claim 1 , wherein the computing system generates an attention score for a plurality of token pairs.
9. The computing system of claim 8 , wherein the computing system applies a maximum relative distance between two tokens within a token pair.
10. The computing system of claim 1 , wherein the computing system utilizes one or more subsets of key, value or query vectors to extract the relative position embedding and calculate attention score for queries.
11. A computing system configured to improve pre-training convergence while encoding data with a transformer, the computing system comprising:
one or more processors; and
one or more computer-readable hardware storage devices that store computer executable instructions that are executable by the one or more processors to cause the computer system to at least:
identify a plurality of tokens to be encoded from a sequence;
obtain a transformer that includes an encoder with a plurality of encoding layers;
embed the plurality of tokens to generate input data;
apply the input data to the encoder by at least disentangling position bias embedding from content embedding associated with the plurality of tokens; and
apply output of a final encoding layer of the transformer as additional input to the final encoding layer for one or more iterations in order to generate new output from the final encoding layer to facilitate an improvement in the pre-training convergence of the transformer.
12. The computing system of claim 11 , wherein the final encoding layer is a decoding task layer.
13. The computing system of claim 12 , wherein a portion of the tokens are masked prior to generating the input data and a wherein the output of the final encoding layer corresponding to portion of the token that are masked prior to generating the input data is replaced with a corresponding absolute position embedding vector prior to being applied as the additional input.
14. The computing system of claim 12 , wherein the computing system applies one or more hidden vector outputs from the decoding layer as the additional input, wherein the one or more hidden vectors correspond to tokens that are masked prior to generating the input data.
15. The computing system of claim 13 , wherein the computing system applies a query vector output of the decoding layer as additional input to the decoding layer for one or more iterations in order to generate new output from the final decoding layer.
16. The computing system of claim 11 , wherein the encoder includes a self-attention sub-layer and a feed forward sub-layer.
17. The computing system of claim 16 , wherein the computing system generates and applies an attention score at the self-attention sub-layer, by at least disentangling position bias embedding from content embedding associated with the plurality of the tokens.
18. The computing system of claim 17 , wherein the attention score comprises a summation of:
a first attention score component comprising a product of a content embedding of a first token in a token pair, a first learn-able content parameter, a second learn-able content parameter, and a content embedding of a second token in a token pair;
a second attention score component comprising a product of the content embedding of the first token in the token pair, the first learn-able content parameter, a first learnable position parameter, and a relative position embedding of the first and second tokens in a token pair; and
a third attention score component comprising a product of the relative position embedding of the first and second tokens in the token pair, a second learn-able position parameter, the second learn-able content parameter, and the second content embedding of a second token in a token pair.
19. The computing system of claim 18 , wherein the computing system generates an attention score for a plurality of token pairs.
20. A method of using encoding data with a transformer that is configured to account for position bias and improve pre-training convergence of the transformer, the method including:
identifying a plurality of tokens to be encoded from a sequence;
obtaining a transformer that includes an encoder, comprising a plurality encoding layers within the encoder wherein each of the plurality of encoding layers includes a self-attention sub-layer and a feed forward sub-layer;
embedding the plurality of tokens to generate input data;
applying the input data to the encoder;
generating and applying an attention score at the self-attention sub-layer, by at least disentangling position bias embedding from content embedding associated with the plurality of the tokens, the attention score comprising a summation of:
a first attention score component comprising a product of a content embedding of a first token in a token pair, a first learn-able content parameter, a second learn-able content parameter, and a content embedding of a second token in a token pair;
a second attention score component comprising a product of the content embedding of the first token in the token pair, the first learn-able content parameter, a first learnable position parameter, and a relative position embedding of the first and second tokens in a token pair; and
a third attention score component comprising a product of the relative position embedding of the first and second tokens in the token pair, a second learn-able position parameter, the second learn-able content parameter, and the second content embedding of a second token in a token pair; and
applying output of a final encoding layer as additional input to the final encoding layer in order to generate new output from the final encoding layer to facilitate an improvement in the pre-training convergence of the transformer.