IP Library › Granted Patent US 12699893

Granted Patent B2

US 12699893 · App. 17/338,777 · Granted Aug 4, 2026

Self-supervised representation learning using bootstrapped latent representations

Inventors: Jean-Bastien François Laurent Grill (Paris, FR); Florian Strub (Paris, FR); Florent Altché (Paris, FR); Corentin Tallec (Paris, FR); Pierre Richemond (London, GB); Bernardo Avila Pires (London, GB); Zhaohan Guo (London, GB); Mohammad Gheshlaghi Azar (London, GB); Bilal Piot (London, GB); Remi Munos (London, GB); Michal Valko (Paris, FR)

Assignee: GDM Holding LLC

G06N3/08

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12699893
App. No.: 17/338,777
Granted: Aug 4, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

A computer-implemented method of training a neural network. The method comprises processing a first transformed view of a training data item, e.g. an image, with a target neural network to generate a target output, processing a second transformed view of the training data item, e.g. image, with an online neural network to generate a prediction of the target output, updating parameters of the online neural network to minimize an error between the prediction of the target output and the target output, and updating parameters of the target neural network based on the parameters of the online neural network. The method can effectively train an encoder neural network without using labelled training data items, and without using a contrastive loss, i.e. without needing “negative examples” which comprise transformed views of different data items.

Claims (70)

1 . A method performed by one or more computers, the method comprising:

training a target neural network by the one or more computers and using a machine learning technique, the training comprising:

processing a first transformation of a training data item with the target neural network to generate a target output comprising a first representation of the training data item;

processing a second transformation of the training data item with an online neural network to generate a prediction of the target output, wherein the online neural network comprises: (i) an encoder subnetwork having a same architecture as the target neural network but different parameter values from the target neural network, and (ii) a prediction subnetwork that comprises one or more neural network layers and that is separate from the encoder subnetwork,

wherein processing the second transformation of the training data item with the online neural network comprises:

processing the second transformation of the training data item using the encoder subnetwork of the online neural network to generate a second representation of the training data item; and

processing the second representation of the training data item using the prediction subnetwork of the online neural network to generate the prediction of the target output;

updating one or more parameters of the online neural network to minimize an error between the prediction of the target output and the target output; and

updating one or more parameters of the target neural network based on the parameters of the encoder subnetwork of the online neural network.

2 . The method of claim 1 , wherein updating one or more parameters of the target neural network comprises:

updating one or more parameters of the target neural network with a moving average of the parameters of the encoder subnetwork of the online neural network.

3 . The method of claim 1 , wherein updating one or more parameters of the target neural network comprises:

determining updated values of the one or more parameters of the target neural network in accordance with ξ←τξ+(1−τ)θ, where ξ represents the parameters of the target neural network, θ represents the parameters of the encoder subnetwork of the online neural network, and τ is a decay rate.

4 . The method of claim 1 , wherein the encoder subnetwork of the online neural network and the target neural network each comprise a respective residual neural network.

5 . The method of claim 1 , wherein:

the online neural network and the target neural network each comprise a respective projection neural network; and

processing the first transformation of the training data item with the target neural network further comprises:

processing the first representation of the training data item using the projection neural network of the target neural network to reduce a dimensionality of the first representation of the training data item; and

processing the second transformation of the training data item with the online neural network further comprises:

processing the second representation of the training data item using the projection neural network of the online neural network to reduce a dimensionality of the second representation of the training data item.

6 . The method of claim 5 , wherein the projection neural network of the target neural network and the projection neural network of the online neural network each comprise a respective multi-layer perceptron.

7 . The method of claim 1 , processing the second representation of the training data item using the prediction subnetwork of the online neural network to generate the prediction of the target output comprises:

processing the second representation of the training data item, using a regression model embodied by parameters of the prediction subnetwork of the online neural network, to generate the prediction of the target output.

8 . The method of claim 7 , wherein the prediction subnetwork of the online neural network comprises a multi-layer perceptron.

9 . The method of claim 1 , further comprising:

initializing the parameters of the online neural network and/or the target neural network to random values.

10 . The method of claim 1 , further comprising:

applying a first data item transformation to the training data item to generate the first transformed view of the training data item; and

applying a second data item transformation to the training data item to generate the second transformed view of the training data item, wherein the second data item transformation is different from the first data item transformation.

11 . The method of claim 1 , wherein updating one or more parameters of the online neural network comprises:

normalizing the prediction of the target output; and

minimizing a squared error between the normalized prediction of the target output and the target output.

12 . The method of claim 1 , further comprising iteratively performing the operations of processing the first transformed view of the training data item with the target neural network and processing the second transformed view of the training data item with the online neural network, for each training data item in a batch comprising a plurality of training data items; and then updating the one or more parameters of the online neural network and updating the one or more parameters of the target neural network.

13 . A computer-implemented method of processing a data item, the method comprising:

providing an input data item to part of a trained online neural network, the online neural network having been trained by performing operations comprising:

processing a first transformation of a training data item with the target neural network to generate a target output comprising a first representation of the training data item;

wherein processing the second transformation of the training data item with the online neural network comprises:

processing the second transformation of the training data item using the encoder subnetwork of the online neural network to generate a second representation of the training data item; and

processing the second representation of the training data item using the prediction subnetwork of the online neural network to generate the prediction of the target output;

updating one or more parameters of the online neural network to minimize an error between the prediction of the target output and the target output; and

updating one or more parameters of the target neural network based on the parameters of the encoder subnetwork of the online neural network

processing the input data item using the part of the trained online neural network;

outputting a representation of the input data item from the part of the trained online neural network; and

processing the representation of the input data item.

14 . The method of claim 13 , wherein the online neural network comprises a residual neural network configured to generate the representation of the input data item.

15 . The method of claim 13 , wherein processing the representation of the input data item includes:

classifying the input data item, using the representation of the input data item.

16 . The method of claim 13 , wherein the input data item comprises an input image.

17 . The method of claim 1 , wherein the training data item comprises an image, the method further comprising generating a representation of pixels of an input image using the encoder subnetwork of the trained online neural network; and using the representation to perform an image processing task.

18 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

training a target neural network by the one or more computers and using a machine learning technique, the training comprising:

processing a first transformation of a training data item with the target neural network to generate a target output comprising a first representation of the training data item;

wherein processing the second transformation of the training data item with the online neural network comprises:

processing the second transformation of the training data item using the encoder subnetwork of the online neural network to generate a second representation of the training data item; and

processing the second representation of the training data item using the prediction subnetwork of the online neural network to generate the prediction of the target output;

updating one or more parameters of the online neural network to minimize an error between the prediction of the target output and the target output; and

updating one or more parameters of the target neural network based on the parameters of the encoder subnetwork of the online neural network.

19 . One or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

training a target neural network by the one or more computers and using a machine learning technique, the training comprising:

processing a first transformation of a training data item with the target neural network to generate a target output comprising a first representation of the training data item;

wherein processing the second transformation of the training data item with the online neural network comprises:

processing the second transformation of the training data item using the encoder subnetwork of the online neural network to generate a second representation of the training data item; and

processing the second representation of the training data item using the prediction subnetwork of the online neural network to generate the prediction of the target output;

updating one or more parameters of the online neural network to minimize an error between the prediction of the target output and the target output; and

updating one or more parameters of the target neural network based on the parameters of the encoder subnetwork of the online neural network.

20 . The one or more computer storage media of claim 19 , wherein updating one or more parameters of the target neural network comprises:

updating one or more parameters of the target neural network with a moving average of the parameters of the encoder subnetwork of the online neural network.