Parameter utilization for language pre-training
Embodiments are directed to pre-training a transformer model using more parameters for sophisticated patterns (PSP++). The transformer model is divided into a held-out model and a main model. A forward pass and a backward pass are performed on the held-out model, where the forward pass determines self-attention hidden states of the held-out model and the backward pass determines loss of the held-out model. A forward pass on the main model is performed to determine a self-attention hidden states of the main model. The self-attention hidden states of the main model are concatenated with the self-attention hidden states of the held-out model. A backward pass is performed on the main model to determine a loss of the main model. The parameters of the held-out model are updated to reflect the loss of the held-out model and parameters of the main model are updated to reflect the loss of the main model.
1 . A method for training a neural network model, the method comprising:
dividing the neural network model stored in a memory into a held-out model and a main model;
during a first forward pass on the held-out model, determining, using a training dataset comprising words in a natural language, held-out model hidden states from attention heads of the held-out model;
determining, a first loss based on the first forward pass and a first backward pass on the held-out model;
during a second forward pass on the main model:
determining, using the training dataset, main model hidden states from attention heads of the main model;
concatenating the held-out model hidden states and the main model hidden states into concatenated hidden states; and
propagating the concatenated hidden states through a subset of layers of the main model, wherein the concatenated hidden states cause the main model to recognize language patterns different from the held-out model;
determining a second loss based on the second forward pass and a second backward pass on the main model;
updating parameters of the held-out model based on the first loss; and
updating parameters of the main model based on the second loss.
2 . The method of claim 1 , wherein:
the neural network model is a transformer model; and
the attention heads are of the transformer model and are from a portion of a number of lower layers of the transformer model, wherein the number of layers is set using a hyperparameter.
3 . The method of claim 2 , wherein the number of lower layers of the transformer model is less than a total number of layers in the transformer model.
4 . The method of claim 2 , wherein the held-out model includes a first portion of the number of lower layers and the main model includes a second portion of the number of lower layers, wherein the first portion is different from the second portion.
5 . The method of claim 1 , wherein a parameter size of the held-out model is smaller than a parameter size of the main model.
6 . The method of claim 1 , wherein:
the held-out model includes a plurality of held-out models; and
the first loss includes a plurality of first losses based on forward passes of the respective held-out models of the plurality of held-out models.
7 . The method of claim 6 , wherein updating the parameters of the held-out model based on the first loss includes updating parameters of the plurality of held-out models based on the respective losses of the plurality of first losses.
8 . A system for training a neural network model, the system comprising:
a non-transitory memory configured to store the neural network model as a held-out model and a main model; and
one or more processors coupled to the non-transitory memory and configured to read instruction from the non-transitory memory to cause the system to perform operations comprising:
during a first forward pass on the held-out model, determining, using a training dataset comprising words in a natural language, held-out model hidden states from attention heads of the held-out model;
determining, a first loss based on the first forward pass and a first backward pass on the held-out model;
during a second forward pass on the main model:
determining, using the training dataset, main model hidden states from attention heads of the main model;
concatenating the held-out model hidden states and the main model hidden states into concatenated hidden states; and
propagating the concatenated hidden states through a subset of layers of the main model, wherein the concatenated hidden states cause the main model to recognize language patterns different from the held-out model;
determining a second loss based on the second forward pass and a second backward pass on the main model;
updating parameters of the held-out model based on the first loss; and
updating parameters of the main model based on the second loss.
9 . The system of claim 8 , wherein:
the neural network model is a transformer model; and
the attention heads are of the transformer model and are from a portion of a number of lower layers of the transformer model, wherein the number of layers is set using a hyperparameter.
10 . The system of claim 9 , wherein the number of lower layers of the transformer model is less than a total number of layers in the transformer model.
11 . The system of claim 9 , wherein the held-out model includes a first portion of the number of lower layers and the main model includes a second portion of the number of lower layers, wherein the first portion is different from the second portion.
12 . The system of claim 8 , wherein a parameter size of the held-out model is smaller than a parameter size of the main model.
13 . The system of claim 8 , wherein:
the held-out model includes a plurality of held-out models; and
the first loss includes a plurality of first losses based on forward passes of the respective held-out models of the plurality of held-out models.
14 . The system of claim 13 , wherein updating parameters of the held-out model based on the first loss includes updating parameters of the plurality of held-out models based on the respective losses of the plurality of first losses.
15 . A non-transitory, machine-readable medium having stored thereon machine-readable instructions executable to cause a system to perform operations comprising:
dividing a neural network model stored in memory into a held-out model and a main model;
during a first forward pass on the held-out model determining, using a training dataset comprising words in a natural language, held-out model hidden states from attention heads of the held-out model;
determining, a first loss based on the first forward pass and a first backward pass on the held-out model;
during a second forward pass on the main model:
determining, using the training dataset, main model hidden states from attention heads of the main model;
concatenating the held-out model hidden states and the main model hidden states into concatenated hidden states; and
propagating the concatenated hidden states through a subset of layers of the main model, wherein the concatenated hidden states cause the main model to recognize language patterns different from the held-out model;
determining a second loss based on the second forward pass on the main model;
updating parameters of the held-out model based on the first loss; and
updating parameters of the main model based on the second loss.
16 . The non-transitory machine-readable medium of claim 15 , wherein:
the neural network model is a transformer model; and
the attention heads are of the transformer model from a portion of a number of lower layers of the transformer model, wherein the number of layers is set using a hyperparameter.
17 . The non-transitory machine-readable medium of claim 16 , wherein the number of lower layers of the transformer model is less than a total number of layers in the transformer model.
18 . The non-transitory machine-readable medium of claim 16 , wherein the held-out model includes a first portion of the number of lower layers and the main model includes a second portion of the number of lower layers, wherein the first portion is different from the second portion.
19 . The non-transitory machine-readable medium of claim 15 , wherein a parameter size of the held-out model is smaller than a parameter size of the main model.
20 . The non-transitory machine-readable medium of claim 15 , wherein:
the held-out model includes a plurality of held-out models; and
the first loss includes a plurality of first losses based on forward passes of the respective held-out models of the plurality of held-out models.