SELF-SUPERVISED LEARNING WITH MODEL AUGMENTATION
A method for providing a neural network system includes performing contrastive learning to the neural network system to generate a trained neural network system. The performing the contrastive learning includes performing first model augmentation to a first encoder of the neural network system to generate a first embedding of a sample, performing second model augmentation to the first encoder to generate a second embedding of the sample, and optimizing the first encoder using a contrastive loss based on the first embedding and the second embedding. The trained neural network system is provided to perform a task.
1 . A method for providing a neural network system, comprising:
performing contrastive learning to the neural network system to generate a trained neural network system, wherein the performing the contrastive learning includes:
performing first model augmentation to a first encoder of the neural network system to generate a first embedding of a sample;
performing second model augmentation to the first encoder to generate a second embedding of the sample;
optimizing the first encoder using a contrastive loss based on the first embedding and the second embedding; and
providing the trained neural network system to perform a task.
2 . The method of claim 1 , wherein the performing the first model augmentation includes:
performing neuron masking by randomly masking one or more neurons associated with the first encoder;
performing layer dropping by dropping one or more layers associated with the first encoder; or
performing encoder complementing using a second encoder.
3 . The method of claim 2 , wherein the performing the neuron masking includes:
randomly masking the one or more neurons of one or more layers associated with the first encoder based on a masking probability.
4 . The method of claim 3 , where the same masking probability is applied to each layer.
5 . The method of claim 3 , wherein different masking probabilities are applied to different layers.
6 . The method of claim 2 , wherein the performing the layer dropping includes:
appending a plurality of appended layers to the first encoder; and
randomly dropping one or more of the plurality of appended layers.
7 . The method of claim 6 , wherein the neuron masking is performed to an original layer of the first encoder or one of the plurality of appended layers.
8 . The method of claim 2 , wherein the performing the encoder complementing includes:
providing a pre-trained encoder by pre-training a second encoder;
providing, by the first encoder, a first intermediate embedding of the sample;
providing, by the pre-trained encoder, a second intermediate embedding of the sample; and
combining the first intermediate embedding and a weighted second intermediate embedding for generating the first embedding for contrastive learning.
9 . The method of claim 1 , wherein the first encoder and the second encoder have different types.
10 . The method of claim 6 , wherein the first encoder is a Transformer-based encoder, and the second encoder is a recurrent neural network (RNN) based encoder.
11 . A non-transitory machine-readable medium comprising a plurality of machine-readable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform a method comprising:
performing contrastive learning to a neural network system to generate a trained neural network system, wherein the performing the contrastive learning includes:
performing first model augmentation to a first encoder of the neural network system to generate a first embedding of a sample;
performing second model augmentation to the first encoder to generate a second embedding of the sample;
optimizing the first encoder using a contrastive loss based on the first embedding and the second embedding; and
providing the trained neural network system to perform a task.
12 . The non-transitory machine-readable medium of claim 11 , wherein the performing the first model augmentation includes:
performing neuron masking by randomly masking one or more neurons associated with the first encoder;
performing layer dropping by dropping one or more layers associated with the first encoder; or
performing encoder complementing using a second encoder.
13 . The non-transitory machine-readable medium of claim 12 , wherein the performing the neuron masking includes:
randomly masking the one or more neurons of one or more layers associated with the first encoder based on a masking probability.
14 . The non-transitory machine-readable medium of claim 12 , wherein the performing the layer dropping includes:
appending a plurality of appended layers to the first encoder; and
randomly dropping one or more of the plurality of appended layers.
15 . The non-transitory machine-readable medium of claim 12 , wherein the performing the encoder complementing includes:
providing a pre-trained encoder by pre-training a second encoder;
providing, by the first encoder, a first intermediate embedding of the sample;
providing, by the pre-trained encoder, a second intermediate embedding of the sample; and
combining the first intermediate embedding and a weighted second intermediate embedding for generating the first embedding for contrastive learning.
16 . A system, comprising:
a non-transitory memory; and
one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform a method comprising:
performing contrastive learning to a neural network system to generate a trained neural network system, wherein the performing the contrastive learning includes:
performing first model augmentation to a first encoder of the neural network system to generate a first embedding of a sample;
performing second model augmentation to the first encoder to generate a second embedding of the sample;
optimizing the first encoder using a contrastive loss based on the first embedding and the second embedding; and
providing the trained neural network system to perform a task.
17 . The system of claim 16 , wherein the performing the first model augmentation includes:
performing neuron masking by randomly masking one or more neurons associated with the first encoder;
performing layer dropping by dropping one or more layers associated with the first encoder; or
performing encoder complementing using a second encoder.
18 . The system of claim 17 , wherein the performing the neuron masking includes:
randomly masking the one or more neurons of one or more layers associated with the first encoder based on a masking probability.
19 . The system of claim 17 , wherein the performing the layer dropping includes:
appending a plurality of appended layers to the first encoder; and
randomly dropping one or more of the plurality of appended layers.
20 . The system of claim 17 , wherein the performing the encoder complementing includes:
providing a pre-trained encoder by pre-training a second encoder;
providing, by the first encoder, a first intermediate embedding of the sample;
providing, by the pre-trained encoder, a second intermediate embedding of the sample; and
combining the first intermediate embedding and a weighted second intermediate embedding for generating the first embedding for contrastive learning.