Systems and methods for contrastive learning with self-labeling refinement
Embodiments described herein provide a contrastive learning mechanism with self-labeling refinement, which iteratively employs the network and data themselves to generate more accurate and informative soft labels for contrastive learning. Specifically, the contrastive learning framework includes a self-labeling refinery module to explicitly generate accurate labels, and a momentum mix-up module to increase similarity between a query and its positive, which in turn implicitly improves label accuracy.
1 . A method for training a machine learning model for detecting objects in an image, the method comprising:
receiving, via a communication interface, a training batch of images;
generating, by one or more processors, a first image query sample and a first positive instance corresponding to the first image query sample by randomly augmenting a same training image from the training batch into two views;
encoding the first image query sample, the first positive instance, and a set of positive instances and negative instances generated from the training batch into respective high-dimensional feature representations;
computing a first instance probability distribution based on a first semantic similarity between a first high-dimensional feature representation of the first positive instance and second high-dimensional feature representations of the set of positive instances and negative instances generated from the training batch;
removing the first positive instance from the set of positive instances and negative instances;
computing a second instance probability distribution based on a second semantic similarity between the first high-dimensional feature representation of the first positive instance and third high-dimensional feature representations corresponding to remaining instances in the set of positive instances and negative instances;
generating, for the first image query sample, a first self-label by combining a first one-hot label of the first image query sample, the first instance probability distribution and the second instance probability distribution;
generating, by a machine learning model implemented on the one or more processors, an encoded output based on a contrastive input of the set of positive instances and negative instances;
computing a contrastive loss objective based on the encoded output supervised by the generated first self-label;
training the machine learning model implemented on the one or more processors based on the contrastive loss objective; and
performing, by the trained machine learning model implemented on the one or more processors, an object detection task on an image.
2 . The method of claim 1 , wherein the set of positive instances and negative instances is obtained by:
generating a first number of positive instances corresponding to query samples in the training batch; and
generating a second number of negative instances corresponding to query samples in the training batch, wherein the second number is greater than the first number.
3 . The method of claim 1 , wherein the first instance probability distribution is computed by:
computing a similarity metric between the first positive instance and a first instance in the set of positive instances and negative instances; and
dividing the similarity metric by a sum of all similarity metrics over all instances in the set of positive instances and negative instances.
4 . The method of claim 3 , wherein the similarity metric is computed by:
encoding, by an online network of the machine learning model, the first query sample into a first feature representation;
encoding, by a target network of the machine learning model, the first instance into an encoded instance; and
computing an exponential similarity between the first feature representation and the encoded instance in an encoded space.
5 . The method of claim 1 , wherein the first self-label is generated by a linear combination with a first weight and a second weight, and the method comprises:
determining the first weight based on a first maximum probability among probability entries of the first instance probability distribution; and
determining the second weight based on a second maximum probability among probability entries of the second instance probability distribution.
6 . The method of claim 1 , further comprising:
constructing a virtual query sample by a first linear combination of the first query sample and a second positive instance corresponding to a second query sample; and
constructing a virtual label by a second linear combination of the first self-label and a second self-label corresponding to the second query sample.
7 . The method of claim 6 , wherein the second positive instance is randomly sampled from the set of positive instances.
8 . The method of claim 6 , wherein the first or the second linear combination includes a weight that is randomly sampled according to Beta distribution.
9 . The method of claim 6 , further comprising:
computing the contrastive loss objective supervised by the virtual label.
10 . The method of claim 1 , wherein the contrastive loss objective is computed as a cross-entropy between the first self-label and a similarity metric of the first query sample to the set of positive instances and negative instances.
11 . A system for training a machine learning model for detecting objects in an image, the system comprising:
a communication interface that receives a training batch of images;
a memory storing a machine learning model and a plurality of processor-executable instructions; and
one or more processors executing the plurality of processor-executable instructions to perform operations comprising:
generating a first image query sample and a first positive instance corresponding to the first image query sample by randomly augmenting a same training image from the training batch into two views;
encoding the first image query sample, the first positive instance, and a set of positive instances and negative instances generated from the training batch into respective high-dimensional feature representations;
computing a first instance probability distribution based on a first semantic similarity between a first high-dimensional feature representation of the first positive instance and second high-dimensional feature representations of the set of positive instances and negative instances generated from the training batch;
removing the first positive instance from the set of positive instances and negative instances;
computing a second instance probability distribution based on a second semantic similarity between the first high-dimensional feature representation of the first positive instance and third high-dimensional feature representations corresponding to remaining instances in the set of positive instances and negative instances;
generating, for the first image query sample, a first self-label by combining a first one-hot label of the first image query sample, the first instance probability distribution and the second instance probability distribution;
generating, by the machine learning model, an encoded output based on a contrastive input of the set of positive instances and negative instances;
computing a contrastive loss objective based on the encoded output supervised by the generated first self-label;
training the machine learning model based on the contrastive loss objective; and
performing, by the trained machine learning model, and object detection task on an image.
12 . The system of claim 11 , wherein the set of positive instances and negative instances is obtained by:
generating a first number of positive instances corresponding to query samples in the training batch; and
generating a second number of negative instances corresponding to query samples in the training batch, wherein the second number is greater than the first number.
13 . The system of claim 11 , wherein the first instance probability distribution is computed by:
computing a similarity metric between the first positive instance and a first instance in the set of positive instances and negative instances; and
dividing the similarity metric by a sum of all similarity metrics over all instances in the set of positive instances and negative instances.
14 . The system of claim 13 , wherein the similarity metric is computed by:
encoding, by a feature encoder of the machine learning model, the first query sample into a first feature representation;
encoding, by a projection head of the machine learning model, the first instance into an encoded instance; and
computing an exponential similarity between the first feature representation and the encoded instance in an encoded space.
15 . The system of claim 11 , wherein the first self-label is generated by a linear combination with a first weight and a second weight, and the operations further comprise:
determining the first weight based on a first maximum probability among probability entries of the first instance probability distribution; and
determining the second weight based on a second maximum probability among probability entries of the second instance probability distribution.
16 . The system of claim 11 , wherein the operations further comprise:
constructing a virtual query sample by a first linear combination of the first query sample and a second positive instance corresponding to a second query sample; and
constructing a virtual label by a second linear combination of the first self-label and a second self-label corresponding to the second query sample.
17 . The system of claim 16 , wherein the second positive instance is randomly sampled from the set of positive instances.
18 . The system of claim 16 , wherein the first or the second linear combination includes a weight that is randomly sampled according to Beta distribution.
19 . The system of claim 16 , wherein the operations further comprise:
computing the contrastive loss objective supervised by the virtual label.
20 . The system of claim 11 , wherein the contrastive loss objective is computed as a cross-entropy between the first self-label and a similarity metric of the first query sample to the set of positive instances and negative instances.