IP Library › Granted Patent US 12,699,893

Granted Patent B2

US 12,699,893 · App. 17/338,777 · Granted Aug 4, 2026

Self-supervised representation learning using bootstrapped latent representations

Inventors: Jean-Bastien François Laurent Grill (Paris, FR); Florian Strub (Paris, FR); Florent Altché (Paris, FR); Corentin Tallec (Paris, FR); Pierre Richemond (London, GB); Bernardo Avila Pires (London, GB); Zhaohan Guo (London, GB); Mohammad Gheshlaghi Azar (London, GB); Bilal Piot (London, GB); Remi Munos (London, GB); Michal Valko (Paris, FR)

Assignee: GDM Holding LLC

G06N3/08

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12,699,893
App. No.: 17/338,777
Filed: Jun 4, 2021
Granted: Aug 4, 2026
Kind: B2
Examiner: GERMICK, JOHNATHAN R
Art Unit: 2122
USPC: 706/15

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

A computer-implemented method of training a neural network. The method comprises processing a first transformed view of a training data item, e.g. an image, with a target neural network to generate a target output, processing a second transformed view of the training data item, e.g. image, with an online neural network to generate a prediction of the target output, updating parameters of the online neural network to minimize an error between the prediction of the target output and the target output, and updating parameters of the target neural network based on the parameters of the online neural network. The method can effectively train an encoder neural network without using labelled training data items, and without using a contrastive loss, i.e. without needing “negative examples” which comprise transformed views of different data items.

Claims (70)

1 . A method performed by one or more computers, the method comprising:

training a target neural network by the one or more computers and using a machine learning technique, the training comprising:

processing a first transformation of a training data item with the target neural network to generate a target output comprising a first representation of the training data item;

processing a second transformation of the training data item with an online neural network to generate a prediction of the target output, wherein the online neural network comprises: (i) an encoder subnetwork having a same architecture as the target neural network but different parameter values from the target neural network, and (ii) a prediction subnetwork that comprises one or more neural network layers and that is separate from the encoder subnetwork,

wherein processing the second transformation of the training data item with the online neural network comprises:

processing the second transformation of the training data item using the encoder subnetwork of the online neural network to generate a second representation of the training data item; and

processing the second representation of the training data item using the prediction subnetwork of the online neural network to generate the prediction of the target output;

updating one or more parameters of the online neural network to minimize an error between the prediction of the target output and the target output; and

updating one or more parameters of the target neural network based on the parameters of the encoder subnetwork of the online neural network.

2 . The method of claim 1 , wherein updating one or more parameters of the target neural network comprises:

updating one or more parameters of the target neural network with a moving average of the parameters of the encoder subnetwork of the online neural network.

3 . The method of claim 1 , wherein updating one or more parameters of the target neural network comprises:

determining updated values of the one or more parameters of the target neural network in accordance with ξ←τξ+(1−τ)θ, where ξ represents the parameters of the target neural network, θ represents the parameters of the encoder subnetwork of the online neural network, and τ is a decay rate.

4 . The method of claim 1 , wherein the encoder subnetwork of the online neural network and the target neural network each comprise a respective residual neural network.

5 . The method of claim 1 , wherein:

the online neural network and the target neural network each comprise a respective projection neural network; and

processing the first transformation of the training data item with the target neural network further comprises:

processing the first representation of the training data item using the projection neural network of the target neural network to reduce a dimensionality of the first representation of the training data item; and

processing the second transformation of the training data item with the online neural network further comprises:

processing the second representation of the training data item using the projection neural network of the online neural network to reduce a dimensionality of the second representation of the training data item.

6 . The method of claim 5 , wherein the projection neural network of the target neural network and the projection neural network of the online neural network each comprise a respective multi-layer perceptron.

7 . The method of claim 1 , processing the second representation of the training data item using the prediction subnetwork of the online neural network to generate the prediction of the target output comprises:

processing the second representation of the training data item, using a regression model embodied by parameters of the prediction subnetwork of the online neural network, to generate the prediction of the target output.

8 . The method of claim 7 , wherein the prediction subnetwork of the online neural network comprises a multi-layer perceptron.

9 . The method of claim 1 , further comprising:

initializing the parameters of the online neural network and/or the target neural network to random values.

10 . The method of claim 1 , further comprising:

applying a first data item transformation to the training data item to generate the first transformed view of the training data item; and

applying a second data item transformation to the training data item to generate the second transformed view of the training data item, wherein the second data item transformation is different from the first data item transformation.

11 . The method of claim 1 , wherein updating one or more parameters of the online neural network comprises:

normalizing the prediction of the target output; and

minimizing a squared error between the normalized prediction of the target output and the target output.

12 . The method of claim 1 , further comprising iteratively performing the operations of processing the first transformed view of the training data item with the target neural network and processing the second transformed view of the training data item with the online neural network, for each training data item in a batch comprising a plurality of training data items; and then updating the one or more parameters of the online neural network and updating the one or more parameters of the target neural network.

13 . A computer-implemented method of processing a data item, the method comprising:

providing an input data item to part of a trained online neural network, the online neural network having been trained by performing operations comprising:

processing a first transformation of a training data item with the target neural network to generate a target output comprising a first representation of the training data item;

wherein processing the second transformation of the training data item with the online neural network comprises:

processing the second transformation of the training data item using the encoder subnetwork of the online neural network to generate a second representation of the training data item; and

processing the second representation of the training data item using the prediction subnetwork of the online neural network to generate the prediction of the target output;

updating one or more parameters of the online neural network to minimize an error between the prediction of the target output and the target output; and

updating one or more parameters of the target neural network based on the parameters of the encoder subnetwork of the online neural network

processing the input data item using the part of the trained online neural network;

outputting a representation of the input data item from the part of the trained online neural network; and

processing the representation of the input data item.

14 . The method of claim 13 , wherein the online neural network comprises a residual neural network configured to generate the representation of the input data item.

15 . The method of claim 13 , wherein processing the representation of the input data item includes:

classifying the input data item, using the representation of the input data item.

16 . The method of claim 13 , wherein the input data item comprises an input image.

17 . The method of claim 1 , wherein the training data item comprises an image, the method further comprising generating a representation of pixels of an input image using the encoder subnetwork of the trained online neural network; and using the representation to perform an image processing task.

18 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

training a target neural network by the one or more computers and using a machine learning technique, the training comprising:

processing a first transformation of a training data item with the target neural network to generate a target output comprising a first representation of the training data item;

wherein processing the second transformation of the training data item with the online neural network comprises:

processing the second transformation of the training data item using the encoder subnetwork of the online neural network to generate a second representation of the training data item; and

processing the second representation of the training data item using the prediction subnetwork of the online neural network to generate the prediction of the target output;

updating one or more parameters of the online neural network to minimize an error between the prediction of the target output and the target output; and

updating one or more parameters of the target neural network based on the parameters of the encoder subnetwork of the online neural network.

19 . One or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

training a target neural network by the one or more computers and using a machine learning technique, the training comprising:

processing a first transformation of a training data item with the target neural network to generate a target output comprising a first representation of the training data item;

wherein processing the second transformation of the training data item with the online neural network comprises:

processing the second transformation of the training data item using the encoder subnetwork of the online neural network to generate a second representation of the training data item; and

processing the second representation of the training data item using the prediction subnetwork of the online neural network to generate the prediction of the target output;

updating one or more parameters of the online neural network to minimize an error between the prediction of the target output and the target output; and

updating one or more parameters of the target neural network based on the parameters of the encoder subnetwork of the online neural network.

20 . The one or more computer storage media of claim 19 , wherein updating one or more parameters of the target neural network comprises:

updating one or more parameters of the target neural network with a moving average of the parameters of the encoder subnetwork of the online neural network.

Assignments (2)

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2025

From: DEEPMIND TECHNOLOGIES LIMITED

To: GDM HOLDING LLC

Reel/Frame 071498/0210 →

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 12, 2021

From: GRILL, JEAN-BASTIEN FRANCOIS LAURENT; STRUB, FLORIAN; ALTCHE, FLORENT; TALLEC, CORENTIN; RICHEMOND, PIERRE; PIRES, BERNARDO AVILA; GUO, ZHAOHAN; AZAR, MOHAMMAD GHESHLAGHI; PIOT, BILAL; MUNOS, REMI; VALKO, MICHAL

To: DEEPMIND TECHNOLOGIES LIMITED

Reel/Frame 057759/0223 →

Continuity (3)

Provisional Application 63039343 · Jun 15, 2020

Provisional Application 63035583 · Jun 5, 2020

Related Publication 20210383225A1 · Dec 9, 2021

References Cited (134)

US 20190050727A1 · Anderson · 2019 [cited by examiner]

US 20200143206A1 · Kartal et al. · 2020 [cited by applicant]

CN 108701253A · 2018 [cited by applicant]

CN 109492666A · 2019 [cited by applicant]

CN 110880036A · 2020 [cited by applicant]

CN 110909815A · 2020 [cited by applicant]

CN 111126599A · 2020 [cited by applicant]

CN 111159367A · 2020 [cited by applicant]

JP 2018055259 · 2018 [cited by applicant]

JP 2019528511 · 2019 [cited by applicant]

JP 2019219712A · 2019 [cited by applicant]

JP 2019220104 · 2019 [cited by applicant]

JP WO2020110775 · 2020 [cited by applicant]

X. Yao, T. Huang, C. Wu, R. Zhang and L. Sun, “Towards Faster and Better Federated Learning: A Feature Fusion Approach,” 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 2019, pp. 175-179, … [cited by examiner]

Antti Tarvainen, & Harri Valpola. (2018). Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. (Year: 2018). [cited by examiner]

Zhiqiang Shen, Zhankui He, & Xiangyang Xue. (2019). MEAL: Multi-Model Ensemble via Adversarial Learning. (Year: 2019). [cited by examiner]

International Preliminary Report on Patentability in International Appln. No. PCT/EP2021/065071, dated Dec. 15, 2022, 12 pages. [cited by applicant]

International Search Report and Written Opinion in International Appln. No. PCT/EP2021/065071, dated Oct. 8, 2021, 18 pages. [cited by applicant]

Zhou et al., “Breaking Transferability of Adversarial Samples with Randomness,” CoRR, Jun. 2018, arxiv.org/abs/1805.04613, 19 pages. [cited by applicant]

Office Action in Indian Appln. No. 202227053096, dated Feb. 8, 2023, 7 pages (with English translation). [cited by applicant]

Bachman et al., “Learning representations by maximizing mutual information across views,” CoRR, Jun. 2019, arxiv.org/abs/1906.00910, 11 pages. [cited by applicant]

Bachman et al., “Learning with pseudo-ensembles,” CoRR, Dec. 2014, arxiv.org/abs/1412.4864, 9 pages. [cited by applicant]

Becker et al., “Self-organizing neural network that discovers surfaces in random-dot stereograms,” Nature, Jan. 1992, 355(6356):161-163. [cited by applicant]

Berg et al., “Birdsnap: Large-scale fine-grained visual categorization of birds,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2011-2018. [cited by applicant]

Berthelot et al., “Mixmatch: A holistic approach to semi-supervised learning,” CoRR, May 2019, arxiv.org/abs/1905.02249, 14 pages. [cited by applicant]

Berthelot et al., “Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring,” CoRR, Nov. 2019, arxiv.org/abs/1911.09785, 13 pages. [cited by applicant]

Bossard et al., “Food-101—mining discriminative components with random forests,” European Conference on Computer Vision, 2014, pp. 446-461. [cited by applicant]

Brock et al., “Large scale GAN training for high fidelity natural image synthesis,” CoRR, Sep. 2018, arXiv:1809.11096, 35 pages. [cited by applicant]

Caron et al., “Deep clustering for unsupervised learning of visual features,” Proceedings of the European Conference on Computer Vision, 2018, pp. 132-149. [cited by applicant]

Chapelle et al., “Semi-supervised learning,” IEEE Transactions on Neural Networks, Mar. 2009, 20(3):542. [cited by applicant]

Chen et al., “A simple framework for contrastive learning of visual representations,” Proceedings of the 37th International Conference on Machine Learning, 2020, 119:1597-1607. [cited by applicant]

Chen et al., “Big self-supervised models are strong semi-supervised learners,” CoRR, Jun. 2020, arXiv:2006.10029, 18 pages. [cited by applicant]

Cimpoi et al., “Describing textures in the wild,” IProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 3606-3613. [cited by applicant]

Cubuk et al., “Randaugment: Practical automated data augmentation with a reduced search space,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020, pp. 702-703. [cited by applicant]

Doersch et al., “Multi-task self-supervised visual learning,” Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2051-2060. [cited by applicant]

Doersch et al., “Unsupervised visual representation learning by context prediction,” Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 1422-1430. [cited by applicant]

Donahue et al., “Adversarial feature learning,” CoRR, May 2016, arxiv.org/abs/1605.09782?context=cs, 18 pages. [cited by applicant]

Donahue et al., “Decaf: A deep convolutional activation feature for generic visual recognition,” Proceedings of the 31st International Conference on Machine Learning, 2014, 32(1):647-655. [cited by applicant]

Donahue et al., “Large scale adversarial representation learning,” CoRR, Jul. 2019, arxiv.org/abs/1907.02544, 32 pages. [cited by applicant]

Dosovitskiy et al., “Discriminative unsupervised feature learning with convolutional neural networks,” Neural Information Processing Systems, Jun. 2014, 13 pages. [cited by applicant]

Dumoulin et al., “Adversarially learned inference,” CoRR, Jun. 2016, arXiv:1606.00704, 18 pages. [cited by applicant]

Everingham et al., “The Pascal visual object classes (VOC) challenge,” International Journal of Computer Vision, Sep. 2009, 88(2):303-338. [cited by applicant]

Fei-Fei et al., “Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories,” 2004 Conference on Computer Vision and Pattern Recognition Workshop, Jul.… [cited by applicant]

Fukushima et al., “Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position,” Biological Cybernetics, 1980, 36(4):193-202. [cited by applicant]

Gidaris et al., “Unsupervised representation learning by predicting image rotations,” CoRR, Mar. 2018, arXiv:1803.07728, 16 pages. [cited by applicant]

Girshick et al., “Rich feature hierarchies for accurate object detection and semantic segmentation,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 580-587. [cited by applicant]

github.com/google/jax [online], “JAX: composable transformations of Python+NumPy programs,” Sep. 2020, retrieved on Jun. 21, 2021, retrieved from URL<https://github.com/google/jax>, 12 pages. [cited by applicant]

Gong et al., “Maxup: A simple way to improve generalization of neural network training,” CoRR, Feb. 2020, arXiv:2002.09024, 10 pages. [cited by applicant]

Goodfellow et al., “Generative adversarial nets,” CoRR, Jun. 2014, arxiv.org/abs/1406.2661, 9 pages. [cited by applicant]

Goyal et al., “Accurate, large minibatch sgd: Training imagenet in 1 hour,” CoRR, Jun. 2017, arXiv:1706.02677, 12 pages. [cited by applicant]

Guo et al., “Bootstrap latent-predictive representations for multitask reinforcement learning,” International Conference on Machine Learning, 2020, 119:3875-3886. [cited by applicant]

Harwood et al., “Smart mining for deep metric learning,” Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2821-2829. [cited by applicant]

Hasselt et al., “Deep reinforcement learning and the deadly triad,” CoRR, Dec. 2018, arxiv.org/abs/1812.02648, 13 pages. [cited by applicant]

He et al., “Deep residual learning for image recognition,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770-778. [cited by applicant]

He et al., “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” In Proceedings of the IEEE international conference on computer vision, 2015, pp. 1026-1034. [cited by applicant]

He et al., “Momentum contrast for unsupervised visual representation learning,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9729-9738. [cited by applicant]

Hénaff et al., “Data-efficient image recognition with contrastive predictive coding,” Proceedings of the 37th International Conference on Machine Learning, 2020, 119:4182-4192. [cited by applicant]

Hessel et al., “Rainbow: Combining improvements in deep reinforcement learning,” In AAAI Conference on Artificial Intelligence, Apr. 2018, 32(1):3215-3222. [cited by applicant]

Hinton et al., “A fast learning algorithm for deep belief nets,” Neural Computation, Jul. 2006, 18(7):1527-1554. [cited by applicant]

Hjelm et al., “Learning deep representations by mutual information estimation and maximization,” CORR, Aug. 2018, arXiv:1808.06670, 24 pages. [cited by applicant]

https://github.com/deepmind/dm-haiku [online], “Haiku: Sonnet for JAX,” Feb. 2020, retrieved on Jun. 30, 2021, retrieved from URL<https://github.com/deepmind/dm-haiku>, 11 pages. [cited by applicant]

Ioffe et al., “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” Proceedings of the 32nd International Conference on Machine Learning, 2015, 37:448-456. [cited by applicant]

Jain et al., “Improved baselines with momentum contrastive learning,” CoRR, Mar. 2020, arXiv:2003.04297, 3 pages. [cited by applicant]

Kingma et al., “Auto-encoding variational bayes,” CoRR, Dec. 2013, arxiv.org/abs/1312.6114, 14 pages. [cited by applicant]

Kingma et al., “Semi-supervised learning with deep generative models,” CoRR, Jun. 2014, arxiv.org/abs/1406.5298, 9 pages. [cited by applicant]

Kolesnikov et al., “Revisiting self-supervised visual representation learning,” CoRR, Jan. 2019, arxiv.org/abs/1901.09005, 13 pages. [cited by applicant]

Kornblith et al., “Do better ImageNet models transfer better?,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2661-2671. [cited by applicant]

Krause et al., “3D object representations for fine-grained categorization,” Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, 2013, pp. 554-561. [cited by applicant]

Krizhevsky, “Learning multiple layers of features from tiny images,” Technical report, University of Toronto, Apr. 8, 2009, 60 pages. [cited by applicant]

Laina et al., “Deeper depth prediction with fully convolutional residual networks,” 2016 Fourth International Conference on 3D Vision, Oct. 2016, 12 pages. [cited by applicant]

Laine et al., “Temporal ensembling for semi-supervised learning,” CoRR, Oct. 2016, arXiv:1610.02242, 13 pages. [cited by applicant]

Larsson et al., “Learning representations for automatic colorization,” European Conference on Computer Vision, Sep. 2016, pp. 577-593. [cited by applicant]

Ledig et al., “Photo-realistic single image super-resolution using a generative adversarial network,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4681-4690. [cited by applicant]

Lee et al., “Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks,” ICML 2013 Workshop: Challenges in Representation Learning, 2013, 6 pages. [cited by applicant]

Li et al., “Prototypical contrastive learning of unsupervised representations,” CoRR, May 2020, arXiv:2005.04966, 16 pages. [cited by applicant]

Lillicrap et al., “Continuous control with deep reinforcement learning,” CoRR, Sep. 2015, arXiv:1509.02971, 14 pages. [cited by applicant]

Long et al., “Fully convolutional networks for semantic segmentation,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3431-3440. [cited by applicant]

Loshchilov et al., “SGDR: stochastic gradient descent with warm restarts,” CoRR, Aug. 2016, arxiv.org/abs/1608.03983, 16 pages. [cited by applicant]

Madry et al., “Towards Deep Learning Models Resistant to Adversarial Attacks,” CoRR, Jun. 2017, arXiv:1706.06083, 28 pages. [cited by applicant]

Maji et al., “Fine-grained visual classification of aircraft,” CoRR, Jun. 2013, arXiv:1306.5151, 6 pages. [cited by applicant]

Maria-Elena et al., “Automated flower classification over a large number of classes,” 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, Dec. 2008, 8 pages. [cited by applicant]

Misra et al., “Self-supervised learning of pretext-invariant representations,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 6707-6717. [cited by applicant]

Miyato et al., “Virtual adversarial training: a regularization method for supervised and semi-supervised learning,” IEEE transactions on pattern analysis and machine intelligence, Aug. 2019, 41(8):1979-1993. [cited by applicant]

Mnih et al., “Asynchronous methods for deep reinforcement learning,” Proceedings of The 33rd International Conference on Machine Learning, 2016, 48:1928-1937. [cited by applicant]

Mnih et al., “Human-level control through deep reinforcement learning,” Nature, Feb. 2015, 518:529-533. [cited by applicant]

Nair et al., “Rectified linear units improve restricted boltzmann machines,” International Conference on Machine Learning, Jan. 2010, 8 pages. [cited by applicant]

Noroozi et al., “Unsupervised learning of visual representations by solving jigsaw puzzles,” European Conference on Computer Vision, Sep. 2016, 19 pages. [cited by applicant]

Oquab et al., “Learning and transferring mid-level image representations using convolutional neural networks,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1717-1724. [cited by applicant]

Owen, “A robust hybrid of lasso and ridge regression,” Contemporary Mathematics, Oct. 2006, 443(7):59-72. [cited by applicant]

Parkhi et al., “Cats and dogs,” 2012 Computer Vision and Pattern Recognition, Jun. 2012, 8 pages. [cited by applicant]

Pascanu et al., “On the difficulty of training recurrent neural networks,” Proceedings of the 30th International Conference on Machine Learning, 2013, 28(3):1310-1318. [cited by applicant]

Pathak et al., “Context encoders: Feature learning by inpainting,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2536-2544. [cited by applicant]

Pathak et al., “Learning features by watching objects move,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2701-2710. [cited by applicant]

Poole et al., “On variational bounds of mutual information,” CoRR, May 2019, arXiv:1905.06922, 14 pages. [cited by applicant]

Rasmus et al., “Semi-supervised learning with ladder networks,” CoRR, Jul. 2015, arxiv.org/abs/1507.02672, 19 pages. [cited by applicant]

Ren et al., “Faster R-CNN: Towards real-time object detection with region proposal networks,” CoRR, Jun. 2015, arxiv.org/abs/1506.01497, 14 pages. [cited by applicant]

Rezende et al., “Stochastic back-propagation and variational inference in deep latent gaussian models,” CoRR, Jan. 2014, arXiv:1401.4082, 14 pages. [cited by applicant]

Russakovsky et al., “ImageNet Large Scale Visual Recognition Challenge,” International Journal of Computer Vision, Apr. 2015, 115(3):211-252. [cited by applicant]

Saunshi et al., “A theoretical analysis of contrastive unsupervised representation learning,” Proceedings of the 36th International Conference on Machine Learning, 2019, 97:5628-5637. [cited by applicant]

Simonyan et al., “Very deep convolutional networks for large-scale image recognition,” CoRR, Sep. 2014, preprint arXiv:1409.1556, 14 pages. [cited by applicant]

Sohn et al., “Fixmatch: Simplifying semi-supervised learning with consistency and confidence,” CoRR, Jan. 2020, arXiv:2001.07685, 21 pages. [cited by applicant]

Szegedy et al., “Rethinking the inception architecture for computer vision,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2818-2826. [cited by applicant]

Tarvainen et al., “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” CoRR, Mar. 2017, arxiv.org/abs/1703.01780, 16 pages. [cited by applicant]

Tian et al., “Contrastive multiview coding,” CoRR, Jun. 2019, arXiv:1906.05849v4, 16 pages. [cited by applicant]

Tian et al., “What makes for good views for contrastive learning,” CoRR, May 2020, arXiv:2005.10243, 24 pages. [cited by applicant]

Van den Oord et al., “Representation learning with contrastive predictive coding,” CoRR, Jul. 2018, preprint arXiv:1807.03748, 13 pages. [cited by applicant]

Vincent et al., “Extracting and composing robust features with denoising autoencoders,” Proceedings of the 25th international conference on Machine learning, Jul. 2008, pp. 1096-1103. [cited by applicant]

Wiskott et al., “Slow feature analysis: Unsupervised learning of invariances,” Neural Computation, Apr. 2002, 14(4):715-770. [cited by applicant]

Wu et al., “Group normalization,” CoRR, Mar. 2018, arxiv.org/abs/1803.08494, 10 pages. [cited by applicant]

Wu et al., “Sampling matters in deep embedding learning,” Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2840-2848. [cited by applicant]

Wu et al., “Unsupervised feature learning via non-parametric instance discrimination,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 3733-3742. [cited by applicant]

Xiao et al., “Sun database: Large-scale scene recognition from abbey to zoo,” 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Jun. 2010, 9 pages. [cited by applicant]

You et al., “Scaling SGD batch size to 32k for imagenet training,” Electrical Engineering and Computer Sciences University of California at Berkeley, Sep. 16, 2017, Technical Report No. UCB/EECS-2017-156, 14 pages. [cited by applicant]

Zagoruyko et al., “Wide residual networks,” CoRR, May 2016, arXiv:1605.07146, 15 pages. [cited by applicant]

Zhai et al., “A large-scale study of representation learning with the visual task adaptation benchmark,” CoRR, Oct. 2019, arxiv.org/abs/1910.04867, 33 pages. [cited by applicant]

Zhai et al., “S4L: Self-supervised semi-supervised learning,” Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 1476-1485. [cited by applicant]

Zhang et al., “Colorful image colorization,” European Conference on Computer Vision, Sep. 2016, 29 pages. [cited by applicant]

Zhou et al., “Places: A 10 million image database for scene recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Jun. 2018, 40(6):1452-1464. [cited by applicant]

Zhu et al., “Introduction to semi-supervised learning,” Synthesis Lectures on Artificial Intelligence and Machine Learning, 2009, 3(1):1-130. [cited by applicant]

Zhuang et al., “Local aggregation for unsupervised learning of visual embeddings,” Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 6002-6012. [cited by applicant]

Decision to Grant Patent in Japanese Appln. No. 2022-560500, dated Apr. 22, 2024, 5 pages. [cited by applicant]

Huang et al., “Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization,” CoRR, Jul. 30, 2017, arxiv.org/abs/1703.06868v2, 11 pages. [cited by applicant]

Office Action in European Appln. No. 21731442.6, dated Nov. 29, 2024, 22 pages. [cited by applicant]

Office Action in Japanese Appln. No. 2024-082493, dated Dec. 2, 2024, 6 pages (with English translation). [cited by applicant]

Office Action in European Appln. No. 21731442.6, dated Nov. 29, 2023, 12 pages. [cited by applicant]

Office Action in Japanese Appln. No. 2022-560500, dated Oct. 30, 2023, 12 pages (with English translation). [cited by applicant]

Office Action in Canadian Appln. No. 3174279, dated Mar. 13, 2024, 6 pages. [cited by applicant]

Office Action in Japanese Appln. No. 2024-082493, dated Apr. 15, 2025, 6 pages (with English translation). [cited by applicant]

Khosla et al., “Supervised contrastive learning,” Advances in neural information processing systems, Apr. 23, 2020, 33:18661-18673. [cited by applicant]

Office Action in Canadian Appln. No. 3174279, dated Jun. 17, 2025, 8 pages. [cited by applicant]

Office Action in Korean Appln. No. 10-2022-7034239, mailed on Sep. 5, 2025, 18 pages (with English translation). [cited by applicant]

Office Action in Chinese Appln. No. 202180025878.9, mailed on Dec. 19, 2025, 16 pages (with English translation). [cited by applicant]

Extended Search Report in European Appln. 25226277.9, mailed on Apr. 21, 2026, 16 pages. [cited by applicant]

Notice of Allowance in Chinese Appln. No. 202180025878.9, mailed on Apr. 23, 2026, 7 pages (with English translation). [cited by applicant]