Unsupervised representation learning with contrastive prototypes
The system and method are directed to a prototypical contrastive learning (PCL). The PCL explicitly encodes the hierarchical semantic structure of the dataset into the learned embedding space and prevents the network from exploiting low-level cues for solving the unsupervised learning task. The PCL includes prototypes as the latent variables to help find the maximum-likelihood estimation of the network parameters in an expectation-maximization framework. The PCL iteratively performs an E-step for finding prototypes with clustering and M-step for optimizing the network on a contrastive loss.
1. A method for training a prototypical contrastive learning (PCL) framework, comprising:
receiving unstructured data at a momentum encoder of the PCL framework;
determining, using the momentum encoder, features from the unstructured data;
clustering, using the momentum encoder, the unstructured data into a number of clusters according to the features;
determining prototypes, assignments, and concentrations of the clusters, a prototype, an assignment, and a concentration for each cluster in the clusters;
determining, a contrastive loss function of the PCL framework from the prototypes, the assignments, and the concentrations of the clusters;
training an encoder of the PCL framework using the contrastive loss function and a subset of unstructured data; and
updating the momentum encoder using weights of the encoder.
2. The method of claim 1 , wherein the trained encoder is configured to determine second cluster for second unstructured data.
3. The method of claim 1 , wherein the momentum encoder and the encoder are convolutional neural networks.
4. The method of claim 3 , wherein a neural network structure of the momentum encoder is the same as a neural network structure of the encoder.
5. The method of claim 1 , wherein the updating further comprises:
updating weights of the momentum encoder with the weights of the encoder.
6. The method of claim 5 , wherein the updated weights of the momentum encoder are a moving average of the weights of the momentum encoder and the weights of the encoder.
7. The method of claim 1 , wherein the momentum encoder is updated at each iteration in an epoch.
8. The method of claim 1 , wherein the updated momentum encoder is trained to perform at least one task associated with processing the unstructured data.
9. The method of claim 1 , wherein the encoder is trained and the momentum encoder is updated over a configurable number of iterations.
10. A system for training a prototypical contrastive learning (PCL) framework, comprising:
a momentum encoder configured to:
receive images; and
determine features from the images;
a clustering module configured to:
cluster the images into a number of clusters according to the features;
determine prototypes, assignments, and concentrations of the clusters, a prototype, an assignment, and a concentration for each cluster in the clusters;
a ProtoNCE modules configured to determine, a contrastive loss function of the PCL framework from the prototypes, the assignments, and the concentrations of the clusters; and
an encoder configured to:
process a subset of the images using the contrastive loss function; and
update the momentum encoder using weights of the encoder.
11. The system of claim 10 , wherein the encoder is configured to determine second cluster for second images.
12. The system of claim 10 , wherein the momentum encoder and the encoder are convolutional neural networks.
13. The system of claim 12 , wherein a neural network structure of the momentum encoder is the same as a neural network structure of the encoder.
14. The system of claim 10 , wherein the encoder is further configured to update weights of the momentum encoder with the weights of the encoder.
15. The system of claim 14 , wherein the updated weights of the momentum encoder are a moving average of the weights of the momentum encoder and the weights of the encoder.
16. The system of claim 10 , wherein the momentum encoder is updated at each iteration in an epoch.
17. The system of claim 10 , wherein the updated momentum encoder is trained to perform at least one task associated with processing an image.
18. The system of claim 10 , wherein the encoder is trained and the momentum encoder is updated over a configurable number of iterations.
19. A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a machine to perform operations for training a prototypical contrastive learning (PCL) framework, the operations comprising:
receiving unstructured data at a momentum encoder of the PCL framework;
determining, using the momentum encoder, features from the unstructured data;
clustering, using the momentum encoder, the unstructured data into a number of clusters according to the features;
determining prototypes, assignments, and concentrations of the clusters, a prototype, an assignment, and a concentration for each cluster in the clusters;
determining, a contrastive loss function of the PCL framework from the prototypes, the assignments, and the concentrations of the clusters;
training an encoder of the PCL framework using the contrastive loss function and a subset of unstructured data; and
updating weights of the momentum encoder using weights of the encoder.
20. The non-transitory machine-readable medium of claim 19 , wherein a neural network structure of the momentum encoder is the same as a neural network structure of the encoder.