Parameter utilization for language pre-training

1. A method for pre-training a transformer model, the method comprising:

dividing the transformer model stored in memory into a held-out model and a main model, wherein the held-out model comprises attention heads of the transformer model from a portion of a predefined number of lower layers of the transformer model;

performing, using a training dataset, a forward pass on the held-out model, the forward pass determines self-attention hidden states of the held-out model at corresponding layers in the predefined number of lower layers;

performing, using the training dataset, a forward pass on the main model, wherein the forward pass comprises:

determining self-attention hidden states of the main model at a corresponding layer;

concatenating the self-attention hidden states of the main model at the corresponding layer with the self-attention hidden states of the held-out model at the corresponding layer, wherein the concatenated self-attention hidden states are inputs to a layer subsequent to the corresponding layer of the main model;

performing, a backward pass on the held-out model, the backward pass determines a loss of the held-out model;

performing, a backward pass on the main model, the backward pass determines a loss of the main model; and

updating parameters of the held-out model based on the loss of the held-out model and parameters of the main model based on the loss of the main model.

2. The method of claim 1 , wherein a parameter size of the held-out model is smaller than a parameter size of the main model.

3. The method of claim 1 , wherein the predefined number of lower layers of the transformer model is less than a total number of layers in the transformer model.

4. The method of claim 1 , further comprising:

adjusting a quantity of layers of the held-out model and a quantity of attention heads at each layer of the held-out model.

5. The method of claim 1 , further comprising:

dividing the held-out model into a plurality of held-out models, wherein a first held-out model has fewer parameters than a second held-out model in the plurality of held-out models.

6. The method of claim 5 , further comprising:

updating parameters of each of the plurality of held-out models based on a respective loss of the plurality of held-out models.

7. The method of claim 1 , wherein the held-out model includes a first portion of the predefined number of lower layers and the main model includes a second portion of the predefined number of lower layers.

8. A system for pre-training a transformer model, the system comprising:

a non-transitory memory; and

one or more processor coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations comprising:

dividing the transformer model stored in memory into a held-out model and a main model, wherein the held-out model comprises attention heads of the transformer model from a portion of a predefined number of lower layers of the transformer model;

performing, using a training dataset, a forward pass on the held-out model, the forward pass determines self-attention hidden states of the held-out model at corresponding layers in the predefined number of lower layers;

performing, using the training dataset, a forward pass on the main model, wherein the forward pass comprises:

determining self-attention hidden states of the main model at a corresponding layer;

concatenating the self-attention hidden states of the main model at the corresponding layer with the self-attention hidden states of the held-out model at the corresponding layer, wherein the concatenated self-attention hidden states are inputs to a layer subsequent to the corresponding layer of the main model;

performing, a backward pass on the held-out model, the backward pass determines a loss of the held-out model;

performing, a backward pass on the main model, the backward pass determines a loss of the main model; and

updating parameters of the held-out model based on the loss of the held-out model and parameters of the main model based on the loss of the main model.

9. The system of claim 8 , wherein a parameter size of the held-out model is smaller than a parameter size of the main model.

10. The system of claim 8 , further comprising:

adjusting a quantity of layers of the held-out model and a quantity of attention heads at each layer of the held-out model.

11. The system of claim 8 , further comprising:

dividing the held-out model into a plurality of held-out models, wherein a first held-out model has fewer parameters than a second held-out model in the plurality of held-out models.

12. The system of claim 11 , further comprising updating parameters of each of the plurality of held-out models based on a respective loss of the plurality of held-out models.

13. The system of claim 8 , wherein the held-out model includes a first portion of the predefined number of lower layers and the main model includes a second portion of the predefined number of lower layers.

14. The system of claim 8 , wherein the predefined number of lower layers of the transformer model is less than a total number of layers in the transformer model.

15. A non-transitory, machine-readable medium having stored thereon machine-readable instructions executable to cause a system to perform operations comprising:

dividing the transformer model stored in memory into a held-out model and a main model, wherein the held-out model comprises attention heads of the transformer model from a portion of a predefined number of lower layers of the transformer model;

performing, using a training dataset, a forward pass on the held-out model, the forward pass determines self-attention hidden states of the held-out model at corresponding layers in the predefined number of lower layers;

performing, using the training dataset, a forward pass on the main model, wherein the forward pass comprises:

determining self-attention hidden states of the main model at a corresponding layer;

concatenating the self-attention hidden states of the main model at the corresponding layer with the self-attention hidden states of the held-out model at the corresponding layer, wherein the concatenated self-attention hidden states are inputs to a layer subsequent to the corresponding layer of the main model;

performing, backward passes on the held-out model and the main model, the backward pass determine a loss of the held-out model and a loss of the main model; and

updating parameters of the held-out model based on the loss of the held-out model and parameters of the main model based on the loss of the main model.

16. The non-transitory, machine-readable medium of claim 15 , wherein a parameter size of the held-out model is smaller than a parameter size of the main model.

17. The non-transitory, machine-readable medium of claim 15 , further comprising:

adjusting a quantity of layers of the held-out model and a quantity of attention heads at each layer of the held-out model.

18. The non-transitory, machine-readable medium of claim 15 , further comprising dividing the held-out model into a plurality of held-out models, wherein a first held-out model has fewer parameters than a second held-out model in the plurality of held-out models.

19. The non-transitory, machine-readable medium of claim 18 , further comprising updating parameters of each of the plurality of held-out models based on a respective loss of the plurality of held-out models.

20. The non-transitory, machine-readable medium of claim 15 , wherein the held-out model includes a first portion of the predefined number of lower layers and the main model includes a second portion of the predefined number of lower layers.