Unified natural language model with segmented and aggregate attention
Systems and methods are provided for training and using a novel unified language foundation model. An encoder-decoder natural language model is obtained and various training data is obtained and used for training. The training process integrates a combination of replaced token detection, corrupted span reconstruction, and disentangled attention methodologies to produce a unified encoder-decoder model. The trained model is trained for performing both natural language understanding (NLU) tasks and natural language generation (NLG) tasks. Attention applied to the model is applied discretely to segmented chunks of encoded data during processing to improve the efficiency of applying attention by the model.
1 . A computer-implemented method for training a natural language model to perform multiple different types of tasks rather than only a single type of task, said method comprising:
accessing a natural language model;
tokenizing a set of training data by creating tokens out of the set of training data;
masking a first subset of the tokens with a single sentinel token, resulting in generation of corrupted span data that is structured to facilitate training the natural language model to perform a natural language understanding task;
separating the set of training data into a set of chunks;
generating encoded data from the set of chunks, wherein each chunk is encoded independently relative to one another, such that chunk-wise encoding is performed;
applying relative attention to different encoded data in each chunk in the set of chunks, wherein applying the relative attention facilitates enabling the natural language model to perform a summarization task, wherein the relative attention is applied after the chunk-wise encoding;
after the relative attention is applied to the set of chunks, concatenating all chunks of the set of chunks into a long representation of the encoded data that includes aggregate attention based on the relative attention applied separately to the different encoded data in each chunk, such that the long representation of the encoded data includes the aggregate attention and such that the long representation is generated after the chunk-wise encoding and after application of the relative attention to the set of chunks; and
training the natural language model using the long representation of the encoded data and using the corrupted span data, resulting in the natural language model being trained to perform multiple different types of tasks.
2 . The computer-implemented method of claim 1 , further comprising:
using the trained natural language model to perform the summarization task.
3 . The computer-implemented method of claim 1 , wherein each chunk in the set of chunks is a subset of the training data.
4 . The computer-implemented method of claim 1 , wherein at least one of the relative attention or the aggregate attention is disentangled attention.
5 . The computer-implemented method of claim 1 , wherein the training data is a paragraph.
6 . The computer-implemented method of claim 1 , wherein the training data is a sentence.
7 . The computer-implemented method of claim 1 , wherein the training data is a partial sentence.
8 . The computer-implemented method of claim 1 , wherein the summarization task comprises abstractive document summarization.
9 . The computer-implemented method of claim 1 , wherein the summarization task comprises conversational summarization.
10 . The computer-implemented method of claim 1 , wherein the summarization task comprises multi-lingual question answering.
11 . A computing system that trains a natural language model to perform multiple different types of tasks rather than only a single type of task, said computing system comprising:
one or more hardware processors; and
one or more hardware storage devices that store instructions that are executable by the one or more hardware processors to cause the computing system to:
access a natural language model;
obtain a set of data;
tokenize the set of data by creating tokens out of the set of data;
mask a first subset of the tokens with a single sentinel token, resulting in generation of corrupted span data that is structured to facilitate training the natural language model to perform a natural language understanding task;
separate the set of data into a set of chunks;
generate encoded data from the set of chunks, wherein each chunk is encoded independently relative to one another, such that chunk-wise encoding is performed;
apply relative attention to different encoded data in each chunk in the set of chunks, wherein applying the relative attention facilitates enabling the natural language model to perform a summarization task, wherein the relative attention is applied after the chunk-wise encoding;
after the relative attention is applied to the set of chunks, concatenate all chunks of the set of chunks into a long representation of the encoded data that includes aggregate attention based on the relative attention applied separately to the different encoded data in each chunk, such that the long representation of the encoded data includes the aggregate attention and such that the long representation is generated after the chunk-wise encoding and after application of the relative attention to the set of chunks; and
train the natural language model using the long representation of the encoded data and using the corrupted span data, resulting in the natural language model being trained to perform multiple different types of tasks.
12 . The computing system of claim 11 , the method further comprising:
using the trained natural language model to perform the summarization task.
13 . The computing system of claim 11 , wherein each chunk in the set of chunks is a subset of the data.
14 . The computing system of claim 11 , wherein at least one of the relative attention or the aggregate attention is disentangled attention.
15 . The computing system of claim 11 , wherein the training data is a paragraph.
16 . The computing system of claim 11 , wherein the training data is a sentence.
17 . The computing system of claim 11 , wherein the training data is a partial sentence.
18 . One or more hardware storage devices that store instructions that are executable by one or more hardware processors to cause the one or more processors to:
access a natural language model;
obtain a set of data;
tokenize the set of data by creating tokens out of the set of data;
mask a first subset of the tokens with a single sentinel token, resulting in generation of corrupted span data that is structured to facilitate training the natural language model to perform a natural language understanding task;
separate the set of data into a set of chunks;
generate encoded data from the set of chunks, wherein each chunk is encoded independently relative to one another, such that chunk-wise encoding is performed;
apply relative attention to different encoded data in each chunk in the set of chunks, wherein applying the relative attention facilitates enabling the natural language model to perform a summarization task, wherein the relative attention is applied after the chunk-wise encoding;
after the relative attention is applied to the set of chunks, concatenate all chunks of the set of chunks into a long representation of the encoded data that includes aggregate attention based on the relative attention applied separately to the different encoded data in each chunk, such that the long representation of the encoded data includes the aggregate attention and such that the long representation is generated after the chunk-wise encoding and after application of the relative attention to the set of chunks; and
train the natural language model using the long representation of the encoded data and using the corrupted span data, resulting in the natural language model being trained to perform multiple different types of tasks.