Visual question answering using model trained on unlabeled videos
An example system includes a processor to learn a shared embedding space on unlabeled videos using speech visual correspondence. The processor can learn a number of additional embeddings including a question plus video embedding and an answer embedding using the shared embedding space to generate a trained visual question answering model. The processor can execute a visual question answering based on the trained visual question answering model.
1 . A system, comprising a processor configured to:
pre-train a machine learning model to learn a shared embedding space on unlabeled videos using a speech visual correspondence, wherein an ImageNet pre-trained Resnet is used to extract 2D features from the unlabeled videos, and wherein a Kinetics pre-trained ResNeXt is used to extract 3D features from the unlabeled videos, and wherein the shared embedding space comprises a network layer with both speech and vision features in a single feature vector, and wherein training the shared embedding space with vision embeddings embeds similar speech and images within a threshold proximity;
train the machine learning model to learn a plurality of additional embeddings comprising a question plus video embedding and an answer embedding using the shared embedding space to generate a trained visual question answering model, wherein the plurality of additional embeddings comprise word representations learned using a pre-trained word2vec embedding model; and
executing the trained visual question answering model, wherein the executing comprises a nearest neighbor search over a set of predetermined answers in a shared question plus video and answer space.
2 . The system of claim 1 , wherein the shared embedding space comprises coordinated text-video representations.
3 . The system of claim 1 , wherein the processor is configured to train the visual question answering model using a max margin ranking loss function to embed an answer close to a corresponding question plus video in the shared embedding space.
4 . The system of claim 1 , wherein the question plus video embedding and the answer embedding each comprise a single feature vector.
5 . The system of claim 1 , wherein the visual question answering model comprises a neural network model.
6 . The system of claim 1 , wherein the additional embeddings comprise embedding layers of a neural network.
7 . A computer-implemented method, comprising:
pre-training a machine learning model to learn, via a processor, a shared embedding space on unlabeled videos using a speech visual correspondence, wherein an ImageNet pre-trained Resnet is used to extract 2D features from the unlabeled videos, and wherein a Kinetics pre-trained ResNeXt is used to extract 3D features from the unlabeled videos, and wherein the shared embedding space comprises a network layer with both speech and vision features in a single feature vector, and wherein training the shared embedding space with vision embeddings embeds similar speech and images within a threshold proximity;
train the machine learning model to learn, via the processor, a plurality of additional embeddings comprising a question plus video embedding and an answer embedding using the shared embedding space to generate a trained visual question answering model, wherein the plurality of additional embeddings comprise word representations learned using a pre-trained word2vec embedding model; and
executing the trained visual question answering model, wherein the executing comprises a nearest neighbor search over a set of predetermined answers in a shared question plus video and answer space.
8 . The computer-implemented method of claim 7 , wherein learning the question plus video embedding comprises training the visual question answering model on labeled data.
9 . The computer-implemented method of claim 7 , wherein training the visual question answering model comprises using a max margin ranking loss function to embed an answer with a reduced distance to a corresponding question plus video in the shared embedding space.
10 . The computer-implemented method of claim 7 , wherein learning the question plus video embedding comprises embedding question plus video embeddings with a reduced distance to corresponding answer embeddings.
11 . The computer-implemented method of claim 7 , wherein executing the visual question answering comprises executing an action recognition or object recognition using the trained visual question answering model.
12 . The computer-implemented method of claim 7 , wherein executing the visual question answering comprises executing an image captioning or video captioning using the trained visual question answering model.
13 . A computer program product for training visual question answering models, the computer program product comprising a computer-readable storage medium having program code embodied therewith, the program code executable by a processor to cause the processor to:
pre-train a machine learning model to learn a shared embedding space on unlabeled videos using a speech visual correspondence, wherein an ImageNet pre-trained Resnet is used to extract 2D features from the unlabeled videos, and wherein a Kinetics pre-trained ResNeXt is used to extract 3D features from the unlabeled videos, and wherein the shared embedding space comprises a network layer with both speech and vision features in a single feature vector, and wherein training the shared embedding space with vision embeddings embeds similar speech and images within a threshold proximity;
train the machine learning model to learn a plurality of additional embeddings comprising a question plus video embedding and an answer embedding using the shared embedding space to generate a trained visual question answering model, wherein the plurality of additional embeddings comprise word representations learned using a pre-trained word2vec embedding model; and
executing the trained visual question answering model, wherein the executing comprises a nearest neighbor search over a set of predetermined answers in a shared question plus video and answer space.
14 . The computer program product of claim 13 , further comprising program code executable by the processor to train the visual question answering model on labeled data.
15 . The computer program product of claim 13 , further comprising program code executable by the processor to use a max margin ranking loss function to embed an answer with a reduced distance to a corresponding question plus video in the shared embedding space.
16 . The computer program product of claim 13 , further comprising program code executable by the processor to embed question plus video embeddings with a reduced distance to corresponding answer embeddings.
17 . The computer program product of claim 13 , further comprising program code executable by the processor to execute a computer vision task based on the trained visual question answering model.