Single stream multi-level alignment for vision-language pretraining
A method is provided for pretraining vision and language models that includes receiving image-text pairs, each including an image and a text describing the image. The method encodes an image into a set of feature vectors corresponding to input image patches and a CLS token which represents a global image feature. The method parses, by a text tokenizer, the text into a set of feature vectors as tokens for each word in the text. The method encodes the CLS token from the NN based visual encoder and the tokens from the text tokenizer into a set of features by a NN based text and multimodal encoder that shares weights for encoding both the CLS token and the tokens. The method accumulates the weights from multiple iterations as an exponential moving average of the weights during the pretraining until a predetermined error threshold is reduced to be under a threshold amount.
1 . A computer-implemented method for pretraining vision and language models, comprising:
receiving image-text pairs, each including an image and a text describing the image;
encoding, by a neural network (NN) based visual encoder, an image into a set of feature vectors corresponding to input image patches and a Classification (CLS) token which represents a global feature of the image;
parsing, by a text tokenizer, the text into a set of feature vectors as tokens for each word in the text;
encoding the CLS token from the NN based visual encoder and the tokens from the text tokenizer into a set of features by a NN based text and multimodal encoder that shares weights for encoding both the CLS token and the tokens;
accumulating, by a NN based momentum encoder, the weights from multiple iterations as an exponential moving average of the weights during the pretraining until a predetermined error threshold is reduced to be under a threshold amount including minimizing a divergence between predicted pseudolabels and momentum pseudolabels that are concepts obtained by projecting the CLS token to a V-dimensional space with a fully-connected layer of the NN based momentum encoder to obtain a pre-trained NN-based visual and language model; and
controlling an autonomous vehicle to avoid collisions based on a trajectory generated with the pre-trained NN-based visual and language model that utilizes input images from a traffic scene.
2 . The computer-implemented method of claim 1 , wherein accumulating the weights further comprises performing for multiple iterations using an image-text contrastive loss to align image and text features, such that the image and the text features from a same pair are encouraged to be closer in a feature space and the image and the text features from a different pair are encouraged to be farther in the feature space.
3 . The computer-implemented method of claim 1 , wherein accumulating the weights further comprises performing for multiple iterations using an image-text matching loss implemented as a binary classification task that encourages a higher score for matching image-text pairs from the NN based text and multimodal encoder than for non-matching image-text pairs.
4 . The computer-implemented method of claim 1 , wherein wherein accumulating the weights further comprises performing for multiple iterations using a masked image and masked text modeling loss that comprises masking some parts of at least one of the image and the text to obtain masked parts and providing the masked parts to corresponding ones of the NN based visual encoder and the NN based text and multimodal encoder for reconstruction and minimization of a difference between the some parts and reconstruction versions of the some parts.
5 . The computer-implemented method of claim 1 , wherein accumulating the weights further comprises performing for multiple iterations using a concept alignment loss that encourages the NN based visual encoder to predict semantic concepts present in at least one of the image and the text by minimizing the divergence between the predicted pseudolabels and the momentum pseudolabels.
6 . The computer-implemented method of claim 1 , wherein the image-text pairs are noisy by at least one of missing one or more concepts, being abstract and being irrelevant.
7 . The computer-implemented method of claim 1 , wherein accumulating the weights further comprises masking the tokens from one modality selected from the image and the text and using cross-modal information to reconstruct masked tokens using an unmasked modality as context to obtain a fine-grained alignment between the image and the text.
8 . The computer-implemented method of claim 1 , further comprising:
computing a global alignment by capturing modality-invariant information; and
computing a loss value based on the modality-invariant information.
9 . A computer program product for pretraining vision and language models, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
receiving image-text pairs by a hardware processor of the computer, each including an image and a text describing the image;
encoding, by a neural network (NN) based visual encoder implemented by the hardware processor, an image into a set of feature vectors corresponding to input image patches and a Classification (CLS) token which represents a global feature of the image;
parsing, by a text tokenizer implemented by the hardware processor, the text into a set of feature vectors as tokens for each word in the text;
encoding the CLS token from the NN based visual encoder and the tokens from the text tokenizer into a set of features by a NN based text and multimodal encoder implemented by the hardware processor that shares weights for encoding both the CLS token and the tokens;
accumulating, by a NN based momentum encoder implemented by the hardware processor, the weights from multiple iterations as an exponential moving average of the weights during the pretraining until a predetermined error threshold is reduced to be under a threshold amount including minimizing a divergence between predicted pseudolabels and momentum pseudolabels that are concepts obtained by projecting the CLS token to a V-dimensional space with a fully-connected layer of the NN based momentum encoder to obtain a pre-trained NN-based visual and language model; and
controlling an autonomous vehicle to avoid collisions based on a trajectory generated with the pre-trained NN-based visual and language model that utilizes input images from a traffic scene.
10 . The computer program product of claim 9 , wherein accumulating the weights further comprises performing for multiple iterations using an image-text contrastive loss to align features of the image and the text, such that the image and the text features from a same pair are encouraged to be closer in a feature space and the image and the text features from a different pair are encouraged to be farther in the feature space.
11 . The computer program product of claim 9 , wherein accumulating the weights further comprises performing for multiple iterations using an image-text matching loss implemented as a binary classification task that encourages a higher score for matching image-text pairs from the NN based text and multimodal encoder than for non-matching image-text pairs.
12 . The computer program product of claim 9 , wherein accumulating the weights further comprises performing for multiple iterations using a masked image and masked text modeling loss that comprises masking some parts of at least one of the image and the text to obtain masked parts and providing the masked parts to corresponding ones of the NN based visual encoder and the NN based text and multimodal encoder for reconstruction and minimization of a difference between the some parts and reconstruction versions of the some parts.
13 . The computer program product of claim 9 , wherein accumulating the weights further comprises performing for multiple iterations using a concept alignment loss that encourages the NN based visual encoder to predict semantic concepts present in at least one of the image and the text.
14 . The computer program product of claim 9 , wherein the image-text pairs are noisy by at least one of missing one or more concepts, being abstract and being irrelevant.
15 . The computer program product of claim 9 , wherein accumulating the weights further comprises performing further comprises masking the tokens from one modality selected from the image and the text and using cross-modal information to reconstruct masked tokens to obtain a fine-grained alignment between the image and the text.
16 . The computer program product of claim 9 , further comprising:
computing a global alignment by capturing modality-invariant information; and
computing a loss value based on the modality-invariant information.
17 . A computer processing system, comprising:
a memory device for storing program code; and
a hardware processor operatively coupled to the memory device for running the program code for pretraining vision and language models including:
receive image-text pairs, each including an image and a text describing the image;
encode, by a neural network (NN) based visual encoder implemented by the hardware processor, an image into a set of feature vectors corresponding to input image patches and a Classification (CLS) token which represents a global feature of the image;
parse, by a text tokenizer implemented by the hardware processor, the text into a set of feature vectors as tokens for each word in the text;
encode the CLS token from the NN based visual encoder and the tokens from the text tokenizer into a set of features by a NN based text and multimodal encoder implemented by the hardware processor that shares weights for encoding both the CLS token and the tokens;
accumulate, by a NN based momentum encoder implemented by the hardware processor, the weights from multiple iterations as an exponential moving average of the weights during the pretraining until a predetermined error threshold is reduced to be under a threshold amount including minimizing a divergence between predicted pseudolabels and momentum pseudolabels that are concepts obtained by projecting the CLS token to a V-dimensional space with a fully-connected layer of the NN based momentum encoder to obtain a pre-trained NN-based visual and language model; and
control an autonomous vehicle to avoid collisions based on a trajectory generated with the pre-trained NN-based visual and language model that utilizes input images from a traffic scene.
18 . The computer processing system of claim 17 , wherein accumulating the weights further comprises performing for multiple iterations using an image-text contrastive loss to align features of the image and the text, such that the image and the text features from a same pair are encouraged to be closer in a feature space and the image and the text features from a different pair are encouraged to be farther in the feature space.
19 . The computer processing system of claim 17 , wherein accumulating the weights further comprises performing for multiple iterations using an image-text matching loss implemented as a binary classification task that encourages a higher score for matching image-text pairs from the NN based text and multimodal encoder than for non-matching image-text pairs.
20 . The computer processing system of claim 17 , wherein accumulating the weights further comprises performing for multiple iterations using a masked image and masked text modeling loss that comprises masking some parts of at least one of the image and the text to obtain masked parts and providing the masked parts to corresponding ones of the NN based visual encoder and the NN based text and multimodal encoder for reconstruction and minimization of a difference between the some parts and reconstruction versions of the some parts.