IP Library › Granted Patent US 12,254,413
Granted Patent B2
US 12,254,413 · App. 18/343,579 · Granted Mar 18, 2025

Systems and methods for contrastive learning of visual representations

Inventors: Ting Chen (Toronto, CA); Simon Komblith (Toronto, CA); Mohammad Norouzi (Toronto, CA); Geoffrey Everest Hinton (Toronto, CA); Kevin Jordan Swersky (Mississauga, CA)
Assignee: GOOGLE LLC
G06N3/084G06F18/2155G06F18/2178G06F18/241G06N3/08G06V10/764G06V10/7753G06V10/7788G06T2207/20081
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,254,413
App. No.
18/343,579
Filed
Jun 28, 2023
Granted
Mar 18, 2025
Kind
B2
Art Unit
2673
USPC
382/157
Abstract

Systems, methods, and computer program products for performing semi-supervised contrastive learning of visual representations are provided. For example, the present disclosure provides systems and methods that leverage particular data augmentation schemes and a learnable nonlinear transformation between the representation and the contrastive loss to provide improved visual representations. Further, the present disclosure also provides improvements for semi-supervised contrastive learning. For example, computer-implemented method may include performing semi-supervised contrastive learning based on a set of one or more unlabeled training data, generating an image classification model based on a portion of a plurality of layers in a projection head neural network used in performing the contrastive learning, performing fine-tuning of the image classification model based on a set of one or more labeled training data, and after performing the fine-tuning, distilling the image classification model to a student model comprising a relatively smaller number of parameters than the image classification model.

Claims (49)

1. A computing system to perform contrastive learning, the computing system comprising:

one or more processors; and

one or more non-transitory computer-readable media that collectively store:

a base encoder neural network configured to process an input to generate an intermediate representation of the input;

a projection head neural network configured to process the intermediate representation of the input to generate a projected representation of the input; and

instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:

obtaining one or more training inputs;

performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input;

separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input;

wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input;

respectively processing, with the base encoder neural network, the first augmented input and the second augmented input to respectively generate a first intermediate representation for the first augmented input and a second intermediate representation for the second augmented input;

respectively processing, with the projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented input and a second projected representation for the second augmented input;

evaluating a loss function that evaluates a difference between the first projected representation and the second projected representation; and

modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the loss function.

2. The computing system of claim 1 , wherein the one or more first augmentation operations further comprises a first resize operation that resizes the training input.

3. The computing system of claim 1 , wherein the one or more first augmentation operations further comprises a first random flip operation that randomly flips the training input.

4. The computing system of claim 1 , wherein the random color distortion operation has a color distortion strength of at least one half.

5. The computing system of claim 4 , wherein the random color distortion operation has a color distortion strength of one.

6. The computing system of claim 1 , wherein the one or more first augmentation operations further comprises a first random Gaussian blur operation that randomly applies a Gaussian blur to the training input.

7. The computing system of claim 1 , wherein the base encoder neural network comprises a ResNet convolutional neural network, and wherein the intermediate representation comprises an output of a final average pooling layer of the ResNet convolutional neural network.

8. The computing system of claim 1 , wherein the projection head neural network comprises a multi-layer perceptron that comprises one hidden layer and a rectified linear unit non-linear activation function.

9. The computing system of claim 1 , wherein the loss function comprises an L2 normalized cross entropy loss with an adjustable temperature parameter.

10. The computing system of claim 9 , wherein the adjustable temperature parameter has a value equal to or greater than 0.1 and equal to or less than 0.5.

11. The computing system of claim 1 , wherein the operations further comprise performing the operations described in claim 1 across a training batch of training inputs, wherein the training batch of training inputs comprises at least 256 training inputs.

12. The computing system of claim 11 , wherein the training batch comprises greater than 2000 training inputs.

13. The computing system of claim 12 , wherein the training batch comprises greater than 4000 training inputs.

14. The computing system of claim 11 , wherein the operations further comprise performing learning rate scaling based on a number of training inputs included in the training batch.

15. The computing system of claim 1 , wherein the operations further comprise after said modifying, generating a classification model from the base encoder neural network and the projection head neural network, the classification model comprising some but not all of the plurality of layers of the projection head neural network; and performing fine-tuning of the classification model based on a set of labeled data.

16. The computing system of claim 1 , wherein evaluating the loss function comprises evaluating the loss function based only on in-batch negative example sampling, whereby an instance class representation vector is not required to be stored in a memory bank.

17. The computing system of claim 1 , wherein evaluating the loss function comprises performing global batch normalization to aggregate mean and variance over a plurality of different devices.

18. A computer-implemented method to perform contrastive learning, method comprising:

obtaining one or more training inputs;

performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input;

separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input;

wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input;

respectively processing, with a base encoder neural network, the first augmented input and the second augmented input to respectively generate a first intermediate representation for the first augmented input and a second intermediate representation for the second augmented input;

respectively processing, with a projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented input and a second projected representation for the second augmented input, wherein the projection head neural network is configured to perform at least one non-linear transformation;

evaluating a loss function that evaluates a difference between the first projected representation and the second projected representation; and

modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the loss function.

19. The computer-implemented method of claim 18 , wherein the loss function comprises an L2 normalized cross entropy loss with an adjustable temperature parameter.

20. One or more non-transitory computer-readable media that collectively store a base encoder neural network that has been trained by a training method, the training method comprising:

obtaining one or more training inputs;

performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input;

separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input;

wherein at least one of the one or more second augmentation or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input;

respectively processing, with the base encoder neural network, the first augmented input and the second augmented input to respectively generate a first intermediate representation for the first augmented input and a second intermediate representation for the second augmented input;

respectively processing, with a projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented input and a second projected representation for the second augmented input, wherein the projection head neural network is configured to perform at least one non-linear transformation;

evaluating a loss function that evaluates a difference between the first projected representation and the second projected representation; and

modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the loss function.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 11, 2023
From: SWERSKY, KEVIN JORDAN
To: GOOGLE LLC
Reel/Frame 064561/0873 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 10, 2023
From: CHEN, TING; KORNBLITH, SIMON; HINTON, GEOFFREY EVEREST; NOROUZI, MOHAMMAD
To: GOOGLE LLC
Reel/Frame 064554/0388 →
Continuity (4)
Continuation 17863070 · Jul 12, 2022
Continuation 17018372 · Sep 11, 2020
Continuation In Part 16847163 · Apr 13, 2020
Related Publication 20230342616A1 · Oct 26, 2023
References Cited (109)
US 10628679B1 · Queen · 2020 [cited by examiner]
US 10902616B2 · Brown et al. · 2021 [cited by applicant]
US 11354778B2 · Chen · 2022 [cited by examiner]
US 11381888B2 · Krishnamurthy · 2022 [cited by applicant]
US 11386302B2 · Chen · 2022 [cited by examiner]
US 20160328644A1 · Lin · 2016 [cited by examiner]
US 20190228658A1 · Huang et al. · 2019 [cited by applicant]
US 20200134016A1 · Cao et al. · 2020 [cited by applicant]
US 20200184278A1 · Zadeh · 2020 [cited by examiner]
US 20210049346A1 · Skala · 2021 [cited by examiner]
US 20210089890A1 · Tang · 2021 [cited by examiner]
US 20210124881A1 · Li · 2021 [cited by examiner]
US 20210184278A1 · Kataoka et al. · 2021 [cited by applicant]
US 20210319266A1 · Chen · 2021 [cited by examiner]
US 20210327029A1 · Chen · 2021 [cited by examiner]
Abadi et al, “Tensorflow: A System for Large-Scale Machine Learning”, Symposium on Operating Systems Design and Implementation, Nov. 2-4, 2-16, Savannah, Georgia, United States, 20 pages. [cited by applicant]
Arora et al, “A Theoretical Analysis of Contrastive Unsupervised Representation Learning”, arXiv:1902v1, Feb. 25, 2019, 19 pages. [cited by applicant]
Bachman et al, “Learning Representations by Maximizing Mutual Information Across Views”, arXiv:1906v 2, Jul. 8, 2019, 11 pages. [cited by applicant]
Becker et al, “Self-organizing Neural Network that Discovers Surfaces in Random-Dot Stereograms”, Nature, vol. 355, Jan. 9, 1992, 3 pages. [cited by applicant]
Bengio et al, “Greedy Layer-Wise Training of Deep Networks”, Advances in Neural Information Processing Systems, 8 pages. [cited by applicant]
Berg et al, “Birdsnap: Large-Scale Fine-Grained Visual Categorization of Birds”, Conference on Computer Vision and Pattern Recognition, 8 pages. [cited by applicant]
Berthelot et al, “Mixmatch: A Holistic Approach to Semi-Supervised Learning”, arXiv:1905v2, Oct. 23, 2019, 14 pages. [cited by applicant]
Bossard et al, “Food-101—Mining Discriminative Components with Random Forests”, European Conference on Computer Vision, 16 pages. [cited by applicant]
Chapelle et al, “Semi-Supervised Learning”, MIT Press, 2006, 524 pages. [cited by applicant]
Chen et al., “Big Self Supervised Models are Strong Semi Supervised Learners”, Jun. 17, 2020. [cited by applicant]
Chen et al, “Improved Baselines with Momentum Contrastive Learning”, arXiv:2003v1, Mar. 9, 2020, 3 pages. [cited by applicant]
Chen et al, “A Simple Framework for Contrastive Learning of Visual Representations”, arXiv:2002v3, Jul. 1, 2020, 20 pages. [cited by applicant]
Chen et al, “On Sampling Strategies for Neural Network-Based Collaborative Filtering”, arXiv:1706v1, Jun. 23, 2017, 14 pages. [cited by applicant]
Chen et al, “Self-Supervised GANs via Auxiliary Rotation Loss”, Conference on Computer Vision and Pattern Recognition, 2019, pp. 12154-12163. [cited by applicant]
Cimpoi et al, “Describing Textures in the Wild”, Conference on Computer Vision and Pattern Recognition, 8 pages. [cited by applicant]
Cubuk et al, “Autoaugment: Learning Augmentation Strategies from Data”, Conference on Computer Vision and Pattern Recognition, 11 pages. [cited by applicant]
Dai et al, “Semi-Supervised Sequence Learning”, arXiv:1511v1, Nov. 4, 2015, 10 pages. [cited by applicant]
Devlin et al, “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding”, arXiv:1810v2, May 24, 2019, 16 pages. [cited by applicant]
DeVries et al, “Improved Regularization of Convolutional Neural Networks with Cutout”, arXiv:1708v2, Nov. 29, 2017, 8 pages. [cited by applicant]
Doersch et al, “Unsupervised Visual Representation Learning by Context Prediction”, arXiv:1505v3, Jan. 16, 2016, 10 pages. [cited by applicant]
Donahue et al, “Adversarial Feature Learning”, arXiv:1605v7, Apr. 3, 2017, 18 pages. [cited by applicant]
Donahue et al, “DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition”, arXiv:1310v1, Oct. 6, 2013, 10 pages. [cited by applicant]
Donahue et al, “Large Scale Adversarial Representation Learning”, arXiv:1907v1, Jul. 4, 2019, 29 pages. [cited by applicant]
Dosovitskiy et al, “Discriminative Unsupervised Feature Learning with Convolutional Neural Networks”, Advances in Neural Information Processing Systems, 9 pages. [cited by applicant]
Everingham et al, “The Pascal Visual Object Classes (VOC) Challenge”, International Journal of Computer Vision, 34 pages. [cited by applicant]
Fei-Fei et al, “Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories”, Conference on Computer Vision and Pattern Recognition, 12 pages. [cited by applicant]
Gidaris et al, “Unsupervised Representation Learning by Predicting Image Rotations”, arXiv:1803v1, Mar. 21, 2018, 16 pages. [cited by applicant]
Goodfellow et al, “Generative Adversarial Nets”, arXiv:1406v1, Jun. 10, 2014, 9 pages. [cited by applicant]
Goyal et al, “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”, arXiv:1706v2, Apr. 30, 2018, 12 pages. [cited by applicant]
Goyal et al, “Scaling and Benchmarking Self-Supervised Visual Representation Learning”, arXiv:1905v2, Jun. 6, 2019, 22 pages. [cited by applicant]
Grill et al., “Bootstrap Your Own a new Approach to Self Supervised Learning”, arxiv.org, Jun. 13, 2020. [cited by applicant]
Gross et al, “Training and Investigating Residual Nets”, The Torch Blog, http://torch.ch/blog/2016/02/04/resnets.html, retrieved on Sep. 16, 2020, 8 pages. [cited by applicant]
Gutmann et al, “Noise-Contrastive Estimation: A New Estimation Principle for Unnormalized Statistical Models”, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-30… [cited by applicant]
Hadsell et al, “Dimensionality Reduction by Learning an Invariant Mapping”, Conference on Computer Vision and Pattern Recognition, 8 pages. [cited by applicant]
Hamilton et al, “Inductive Representation Learning on Large Graphs”, arXiv:1706v4, Sep. 10, 2018, 19 pages. [cited by applicant]
He et al, “Bag of Tricks for Image Classification with Convolutional Neural Networks”, arXiv:1812v2, Dec. 5, 2018, 10 pages. [cited by applicant]
He et al, “Deep Residual Learning for Image Recognition”, arXiv:1512v1, Dec. 10, 2015, 12 pages. [cited by applicant]
He et al, “Momentum Contrast for Unsupervised Visual Representation Learning”, arXiv:1911v3, Mar. 23, 2020, 12 pages. [cited by applicant]
He et al, “Rethinking ImageNet Pre-Training”, arXiv:1811v1, Nov. 21, 2018, 10 pages. [cited by applicant]
Henaff et al, “Data-Efficient Image Recognition with Contrastive Predictive Coding”, arXiv:1905v3. Jul. 1, 2020, 13 pages. [cited by applicant]
Henriques, et al, “Invariant Information Clustering for Unsupervised Image Classification and Segmentation”, arXiv:1807v4, Aug. 22, 2019, 10 pages. [cited by applicant]
Hinton et al, “A Fast Learning Algorithm for Deep Belief Nets”, Neural Computation, vol. 18, 16 pages. [cited by applicant]
Hinton et al, “Distilling the Knowledge in a Neural Network”, arXiv:1503v1, Mar. 9, 2015, 9 pages. [cited by applicant]
Hjelm et al, “Learning Deep Representations by Mutual Information Estimation and Maximization”, arXiv:1808v5, Feb. 22, 2019, 24 pages. [cited by applicant]
Howard, “Some Improvements on Deep Convolutional Neural Network Based Image Classification”, arXiv:1312v1, 6 pages. [cited by applicant]
Ioffe et al, “Batch normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”, arXiv:1502v3, Mar. 2, 2015, 11 pages. [cited by applicant]
Kingma et al, “Auto-Encoding Variational Bayes”, arXiv:1312v10, May 1, 2014, 14 pages. [cited by applicant]
Kiros et al, “Skip-Thought Vectors”, arXiv:1506v1, Jun. 22, 2015, 11 pages. [cited by applicant]
Kolesnikov et al, “Big Transfer (BiT): General Visual Representation Learning”, arXiv:1912v3, May 5, 2020, 28 pages. [cited by applicant]
Kolesnikov et al, “Revisiting Self-Supervised Visual Representation Learning”, arXiv:1901v1, Jan. 25, 2019, 13 pages. [cited by applicant]
Kornblith et al, “Do Better ImageNet Models Transfer Better?”, arXiv:1805v3, Jun. 17, 2019, 24 pages. [cited by applicant]
Krause et al., “Collecting a Large-Scale Dataset of Fine-Grained Cars”, Second Workshop on Fine-Grained Visual Categorization, 2 pages. [cited by applicant]
Krizhevsky et al, “Imagenet Classification with Deep Convolutional Neural Networks”, Advances in Neural Information Processing Systems, 9 pages. [cited by applicant]
Lee, “Pseudo-Label: The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks”, International Council for Machinery Lubrication, Atlanta, Georgia, United States, 2013, 6 pages. [cited by applicant]
Li et al, “Selective Kernel Networks”, arXiv:1903v2, Mar. 18, 2019, 12 pages. [cited by applicant]
Linsker, “Self-Organization in a Perceptual Network”, Computer, Mar. 1988, pp. 105-117. [cited by applicant]
Loshchilov et al, “Decoupled Weight Decay Regularization”, arXiv:1711v3, Jan. 4, 2019, 19 pages. [cited by applicant]
Loshchilov et al, “SGDR: Stochastic Gradient Descent with Warm Restarts”, arXiv:1608v5, May 3, 2017, 16 pages. [cited by applicant]
Maaten et al, “Visualizing Data Using t-SNE” Journal of Machine Learning Research, vol. 9, 2008, 27 pages. [cited by applicant]
Maji et al, “Fine-Grained Visual Classification of Aircraft”, arXiv:1306v1, Jun. 21, 2013, 6 pages. [cited by applicant]
Mikolov et al, “Efficient Estimation of Word Representations in Vector Space”, arXiv:1301v3, Sep. 7, 2013, 12 pages. [cited by applicant]
Misra et al, “Self-Supervised Learning of Pretext-Invariant Representations”, arXiv:1912v1, Dec. 4, 2019, 14 pages. [cited by applicant]
Nilsback et al, “Automated Flower Classification Over a Large Number of Classes”, Computer Vision, Graphics & Image Processing, 2008, 8 pages. [cited by applicant]
Noroozi et al, “Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles”, arXiv:1603v3, Aug. 22, 2017, 19 pages. [cited by applicant]
Oliver et al, “Realistic Evaluation of Deep Semi-Supervised Learning Algorithms”, arXiv:1804v4, Jun. 17, 2019, 19 pages. [cited by applicant]
Oord et al, “Representation Learning with Contrastive Predictive Coding”, arXiv:1807v2, Jan. 22, 2019, 13 pages. [cited by applicant]
Parkhi et al, “Cats and Dogs”, Conference on Computer Vision and Pattern Recognition, 8 pages. [cited by applicant]
Peters, “Deep Contextualized Word Representations”, arXiv:1802v2, Mar. 22, 2018, 15 pages. [cited by applicant]
Pham et al, “Meta Pseudo Labels”, arXiv:2003v2, Apr. 23, 2020, 12 pages. [cited by applicant]
Radford et al, “Improving Language Understanding by Generative Pre-Training”, OpenAI, 2018, 12 pages. [cited by applicant]
Radford et al, “Language Models are Unsupervised Multitask Leamers”, OpenAI, 24 pages. [cited by applicant]
Rendle et al, “Bayesian Personalized Ranking from Implicit Feedback”, arXiv:1205, 10 pages. [cited by applicant]
Russakovsky et al, “Imagenet Large Scale Visual Recognition Challenge”, arXiv:1409v3, Jan. 30, 2015, 43 pages. [cited by applicant]
Schroff et al, “Facenet: A Unified Embedding for Face Recognition and Clustering”, arXiv:1503v3, Jun. 17, 2015, 10 pages. [cited by applicant]
Simonyan et al, “Very Deep Convolutional Networks for Largescale Image Recognition”, arXiv:1409v6, Apr. 10, 2015, 14 pages. [cited by applicant]
Sohn et al, “FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence”; arXiv:2001v1, Jan. 21, 2020, 14 pages. [cited by applicant]
Sohn, “Improved Deep Metric Learning with Multi-Class N-Pair Loss Objective”, Conference on Neural Information Processing Systems, 2016, 9 pages. [cited by applicant]
Szegedy et al, “Going Deeper with Convolutions”, arXiv:1409v1, Sep. 17, 2014, 12 pages. [cited by applicant]
Szegedy et al, “Rethinking the Inception Architecture for Computer Vision”, arXiv:1512v3, Dec. 11, 2015, 10 pages. [cited by applicant]
Tian et al, “Contrastive Multiview Coding”, arXiv:1906v4, Mar. 11, 2020, 16 pages. [cited by applicant]
Tian et al., “Contrastive Representation Distillation”, 2019. [cited by applicant]
Tschannen et al, “On Mutual Information Maximization for Representation Learning”, arXiv:1907v2, Jan. 23, 2020, 16 pages. [cited by applicant]
V.R. de Sa, “Learning Classification with Unlabeled Data”, Advances in Neural Information Processing Systems, pp. 112-119. [cited by applicant]
Verma et al, “Interpolation Consistency Training for Semi-Supervised Learning”, arXiv:1903v3, May 19, 2019, 10 pages. [cited by applicant]
Wu et al, “Unsupervised Feature Learning via Non-Parametric Instance Discrimination”, Conference on Computer Vision and Pattern Recognition, 10 pages. [cited by applicant]
Xiao et al, “SUN database: Large-Scale Scene Recognition from Abbey to Zoo”, Conference on Computer Vision and Pattern Recognition, 8 pages. [cited by applicant]
Xie et al, “Self-Training with Noisy Student Improves ImageNet Classification”, arXiv:1911v4, Jun. 19, 2020, 18 pages. [cited by applicant]
Xie et al, “Unsupervised Data Augmentation”, arXiv:1904, 23 pages. [cited by applicant]
Yalniz et al, “Billion-Scale Semi-SupervisedLearning for Image Classification”, arXiv:1905v1, May 2, 2019, 12 pages. [cited by applicant]
Ye et al, “Unsupervised Embedding Learning via Invariant and Spreading Instance Feature”, Conference on Computer Vision and Pattern Recognition, pp. 6210-6219. [cited by applicant]
You et al, “Large Batch Training of Convolutional Networks”, arXiv:1708v3, Sep. 13, 2017, 8 pages. [cited by applicant]
Zhai et al, “S4L: Self-Supervised Semi-Supervised Learning”, arXiv:1905v2, Jul. 23, 2019, 13 pages. [cited by applicant]
Zhang et al, “Colorful Image Colorization”, arXiv:1603v5, Oct. 5, 2016, 29 pages. [cited by applicant]
Zhuang et al, “Local Aggregation for Unsupervised Learning of Visual Embeddings”, arXiv:1903v2, Apr. 10, 2019, 13 pages. [cited by applicant]