Unsupervised pre-training of geometric vision models
A method includes: performing unsupervised pre-training of a model, the model including and a decoder including: obtaining a first image and a second image under different conditions or from different viewpoints; encoding, by the encoder, the first image into a representation of the first image and the second image into a representation of the second image; transforming the representation of the first image into a transformed representation; decoding, by the decoder, the transformed representation into a reconstructed image, where the transforming of the representation of the first image and the decoding of the transformed representation is based on the representation of the first image and the representation of the second image; and adjusting one or more parameters of at least one of the encoder and the decoder based on minimizing a loss; and fine-tuning the model, initialized with a set of task specific encoder parameters, for a geometric vision task.
1 . A computer-implemented machine learning method of training a task specific machine learning model for a downstream geometric vision task, the method comprising:
performing unsupervised pre-training of a machine learning model, the machine learning model comprising an encoder having a set of encoder parameters and a decoder having a set of decoder parameters,
wherein the performing of the unsupervised pre-training of the machine learning model includes:
obtaining a pair of unannotated images including a first image and a second image,
wherein the first and second images depict a same scene and are taken under different conditions or from different viewpoints;
encoding, by the encoder, the first image into a representation of the first image and the second image into a representation of the second image;
transforming the representation of the first image into a transformed representation;
decoding, by the decoder, the transformed representation into a reconstructed image,
wherein the transforming of the representation of the first image and the decoding of the transformed representation is based on the representation of the first image and the representation of the second image; and
adjusting one or more parameters of at least one of the encoder and the decoder based on minimizing a loss;
constructing the task specific machine learning model for the downstream geometric vision task based on the pre-trained machine learning model,
the task specific machine learning model comprising a task specific encoder having a set of task specific encoder parameters;
initializing the set of task specific encoder parameters with the set of encoder parameters of the pre-trained machine learning model; and
fine-tuning the task specific machine learning model, initialized with the set of task specific encoder parameters, for the downstream geometric vision task.
2 . The method of claim 1 , wherein the unsupervised pre-training of the machine learning model is a cross-view alignment pre-training, and wherein the transforming of the representation of the first image includes applying a transformation to the representation of the first image to generate the transformed representation, the transformation being determined based on the representation of the first image and the representation of the second image such that the transformed representation approximates the representation of the second image.
3 . The method of claim 1 , wherein the unsupervised pre-training of the machine learning model is a cross-view alignment pre-training, and wherein the loss is based on a metric quantifying a difference between the reconstructed image and the second image.
4 . The method of claim 1 , wherein the unsupervised pre-training of the machine learning model is a cross-view alignment pre-training, and wherein the loss is based on a metric quantifying a difference between the transformed representation and the representation of the second image.
5 . The method of claim 2 , wherein the representation of the first image is a first set of n vectors {x 1,i } i=1 . . . n , each x 1,i ∈ K , wherein the representation of the second image is a second set of n vectors {x 2,i } i=1 . . . n , each x 2,i ∈ K , wherein the applying of the transformation includes decomposing each vector of the first and second sets of vectors in a D-dimensional equivariant part and a (K−D)-dimensional invariant part and applying a (D×D)-dimensional transformation matrix Ω to the equivariant part of each vector of the first set of vectors, wherein 0<D≤K.
6 . The method of claim 5 , wherein the transformation is a D-dimensional rotation and Ω is a D-dimensional rotation matrix, and wherein Ω is set based on aligning the equivariant parts of the vectors of the first set of vectors with the equivariant parts of the respective vectors of the second set of vectors.
7 . The method of claim 5 , further comprising determining Ω based on the equation:
Ω
=
arg
min
Ω
^
∈
SO
(
D
)
∑
i
=
1
n
Ω
ˆ
x
1
,
i
e
q
u
i
v
-
x
2
,
i
e
q
u
i
v
2
where x 1,i equiv denotes the equivariant part of vector x 1,i , x 2,i equiv denotes the equivariant part of vector x 2,i , and SO(D) denotes the D-dimensional rotation group.
8 . The method of claim 1 , wherein the unsupervised pre-training is a cross-view completion pre-training, and wherein the performing of the cross-view completion pre-training of the machine learning model further comprises:
splitting the first image into a first set of non-overlapping patches and splitting the second image into a second set of non-overlapping patches; and
masking ones of the patches of the first set of patches,
wherein the encoding of the first image into the representation of the first image includes, encoding, by the encoder, each unmasked patch of the first set of patches into a corresponding representation of the respective unmasked patch, thereby generating a first set of patch representations,
wherein the encoding the second image into the representation of the second image includes, encoding, by the encoder, each patch of the second set of patches into a corresponding representation of the respective patch, thereby generating a second set of patch representations,
wherein the decoding of the transformed representation includes, generating, by the decoder, for each masked patch of the first set of patches, a predicted reconstruction for the respective masked patch based on the transformed representation and the second set of patch representations, and
wherein the loss function is based on a metric quantifying the difference between each masked patch and its respective predicted reconstruction.
9 . The method of claim 8 , wherein the transforming of the representation of the first image into the transformed representation further includes, for each masked patch of the first set of patches, padding the first set of patch representations with a respective learned representation of the masked patch.
10 . The method of claim 9 , wherein each learned representation includes a set of representation parameters.
11 . The method of claim 9 , wherein the generating of the predicted reconstruction of a masked patch of the first set of patches includes decoding, by the decoder, the learned representation of the masked patch into the predicted reconstruction of the masked patch, where the decoder receives the first and second sets of patch representations as input data and decodes the learned representation of the masked patch based on the input data, and wherein the method further includes adjusting the learned representations of the masked patches by adjusting the respective set of representation parameters.
12 . The method of claim 11 wherein the adjusting the respective set of representation parameters includes adjusting the set of representation parameters based on minimizing the loss.
13 . The method of claim 1 , wherein fine-tuning the task specific machine learning model includes performing supervised fine-tuning of the task specific machine learning model.
14 . The method of claim 13 wherein the supervised fine-tuning includes:
i) applying the task specific machine learning model to one or more annotated images to create output data according to the downstream geometric vision task,
wherein the annotated images are annotated with ground truth data for the downstream geometric vision task; and
ii) adjusting the task specific machine learning model by adjusting the set of task specific encoder parameters of the task specific machine learning model based on minimizing a task specific loss, the task specific loss being based on a metric quantifying a difference between the created output data and the ground truth data.
15 . The method of claim 14 further comprising repeating i) and ii) for a predetermined number of times or until a minimum value of the task specific loss is reached.
16 . The method of claim 14 further comprising applying the fine-tuned task specific machine learning model to one or more new images to extract prediction data from the one or more new images for the downstream geometric vision task.
17 . The method of claim 16 , wherein the downstream geometric vision task is relative pose estimation.
18 . The method of claim 17 wherein the applying of the fine-tuned task specific machine learning model includes applying the fine-tuned task specific machine learning model to a new pair of images to extract a relative rotation and a relative translation between the images of the new pair of images as the prediction data,
wherein the images of the pair of new images depict two views of a same scene.
19 . The method of claim 16 , wherein the downstream geometric vision task is depth estimation.
20 . The method of claim 19 wherein the applying of the fine-tuned task specific machine learning model includes applying the fine-tuned task specific machine learning model to one or more new images to extract one or more depth maps from the one or more new images as the prediction data.
21 . The method of claim 16 , wherein the downstream geometric vision task is optical flow estimation.
22 . The method of claim 21 wherein the applying of the fine-tuned task specific machine learning model includes applying the fine-tuned task specific machine learning model to a new image pair including a new first image and a new second image to generate a plurality of pixel pairs as the prediction data,
wherein each pixel pair includes a pixel in the new first image and a corresponding pixel in the new second image,
where the new first and second images depict a same scene under different conditions or from different viewpoints.
23 . A computer-implemented machine learning method for generating prediction data according to a downstream geometric vision task, the method comprising:
training a first task specific machine learning model for the downstream geometric vision task using cross-view alignment pre-training of a first machine learning model;
training a second task specific machine learning model for the downstream geometric vision task using cross-view completion pre-training of a second pretext machine learning model;
generating first prediction data according to the downstream geometric vision task by applying the trained first task specific machine learning model to at least one image;
generating second prediction data according to the downstream geometric vision task by applying the trained second task specific machine learning model to the at least one image;
determining a first confidence value for the first prediction data and a second confidence value for the second prediction data; and
generating resulting prediction data according to the geometric vision task by fusing together the first and second prediction data based on the first and second confidence values.
24 . A computing system comprising:
one or more processors; and
memory including code that, when executed by the one or more processors, perform to:
perform unsupervised pre-training of a machine learning model, the machine learning model comprising an encoder having a set of encoder parameters and a decoder having a set of decoder parameters,
wherein the performance of the unsupervised pre-training of the machine learning model includes:
obtaining a pair of unannotated images including a first image and a second image,
wherein the first and second images depict a same scene and are taken under different conditions or from different viewpoints;
encoding, by the encoder, the first image into a representation of the first image and the second image into a representation of the second image;
transforming the representation of the first image into a transformed representation;
decoding, by the decoder, the transformed representation into a reconstructed image,
wherein the transforming of the representation of the first image and the decoding of the transformed representation is based on the representation of the first image and the representation of the second image; and
adjusting one or more parameters of at least one of the encoder and the decoder based on minimizing a loss;
construct the task specific machine learning model for the downstream geometric vision task based on the pre-trained machine learning model,
the task specific machine learning model comprising a task specific encoder having a set of task specific encoder parameters;
initialize the set of task specific encoder parameters with the set of encoder parameters of the pre-trained machine learning model; and
fine-tune the task specific machine learning model, initialized with the set of task specific encoder parameters, for the downstream geometric vision task.