Cross-modal processing for vision and language

1 . A computer-implemented method comprising:

extracting a set of visual features of a training image and training text associated with the training image according to a visual feature extraction sub-model in a target model, wherein:

each visual feature corresponds to a pixel block in the training image; and

each visual feature comprises a visual feature vector in a common embedding space;

determining, based on a visual semantic dictionary, a set of visual semantic features corresponding to the set of visual features, wherein each visual semantic feature comprises a visual semantic feature vector in the common embedding space;

extracting a set of text features of a training text corresponding to the training image according to a text feature extraction sub-model in the target model, wherein:

each text feature corresponds to at least one word in the training text;

the training image and the training text comprise an image-text pair of a plurality of image-text pairs, each image-text pair including a label indicating whether the training image and training text are matched or not matched; and

each text feature comprises a text feature vector in the common embedding space having a same dimensionality as the visual feature vectors and the visual semantic feature vectors; and

training, based on the set of visual semantic features and the set of text features and the label, the target model to determine association information between an input text and an input image.

2 . The method of claim 1 , wherein determining the set of visual semantic features based on the visual semantic dictionary comprises:

determining, from the visual semantic dictionary, at least one candidate semantic feature to which the set of visual features is mapped;

updating the at least one candidate semantic feature based on the set of visual features; and

determining the set of visual semantic features based on the at least one updated candidate semantic feature.

3 . The method of claim 2 , wherein updating the at least one candidate semantic feature comprises:

for a specified candidate semantic feature of the at least one candidate semantic feature, determining, from the set of visual features, at least one visual feature mapped to the specified candidate semantic feature; and

updating the specified candidate semantic feature based on the at least one visual feature and a weight for the specified candidate semantic feature, wherein determining the set of visual semantic features based on the at least one updated candidate semantic feature comprises determining the updated specified candidate semantic feature as a visual semantic feature of the set of visual semantic features that is corresponding to the at least one visual feature.

4 . The method of claim 1 , wherein training the target model comprises:

generating a set of fused features for the image-text pair, wherein the generating is:

based on the set of visual semantic features and the set of text features; and

according to a fusion sub-model in the target model;

determining an objective function based on the set of fused features; and

training the target model by minimizing the objective function.

5 . The method of claim 4 , wherein generating the set of fused features comprises:

masking at least one visual semantic feature of the set of visual semantic features;

generating the set of fused features, wherein the generating is:

according to the fusion sub-model; and

based on the set of text features and the masked set of visual semantic features, wherein determining the objective function comprises:

generating a predicted feature corresponding to the at least one masked visual semantic feature based on the set of fused features; and

determining the objective function based on a difference between the predicted feature and the at least one visual semantic feature.

6 . The method of claim 5 , wherein the at least one visual semantic feature corresponds to a same candidate semantic feature in the visual semantic dictionary.

7 . The method of claim 4 , wherein determining the objective function comprises determining the objective function based on the set of fused features and a specific task to which the target model is to be applied.

8 . The method of claim 7 , wherein the specific task comprises at least one of:

image retrieval;

text retrieval;

visual question answering;

visual reasoning with natural language;

image generation; or

text generation.

9 . A computer-implemented method, comprising:

extracting a set of visual features of an input image according to a visual feature extraction sub-model in a trained target model, wherein;

each visual feature corresponds to a pixel block in the input image; and

each visual feature comprises a visual feature vector in a common embedding space;

determining a set of visual semantic features corresponding to the set of visual features based on a visual semantic dictionary, wherein each visual semantic feature comprises a visual semantic feature vector in the common embedding space;

extracting a set of text features of a training text corresponding to a training image according to a text feature extraction sub-model in the trained target model, wherein:

each text feature corresponds to at least one word in the training text;

the training image and the training text comprise an image-text pair of a plurality of image-text pairs, each image-text pair including a label indicating whether the training image and training text are matched or not matched; and

each text feature comprises a text feature vector in the common embedding space having a same dimensionality as the visual feature vectors and the visual semantic feature vectors; and

training, based on the set of visual semantic features and the set of text features and the label, the target model to determine association information between an input text and an input image.

10 . The method of claim 9 , wherein the association information is applied in at least one of:

image retrieval based on the input text;

text retrieval based on the input image;

visual question answering related to the input text and the input image;

visual reasoning with natural language related to the input text and the input image;

image generation based on the input text; or

text generation based on the input image.

11 . An electronic device, comprising:

a processing unit; and

a memory coupled to the processing unit and comprising instructions stored thereon which, when executed by the processing unit, cause the device to perform acts comprising:

extracting a set of visual features of an input image according to a visual feature extraction sub-model in a trained target model, wherein;

each visual feature corresponds to a pixel block in the input image; and

each visual feature comprises a visual feature vector in a common embedding space;

determining a set of visual semantic features corresponding to the set of visual features based on a visual semantic dictionary, wherein each visual semantic feature comprises a visual semantic feature vector in the common embedding space;

extracting a set of text features of a training text corresponding to a training image according to a text feature extraction sub-model in the trained target model, wherein:

each text feature corresponds to at least one word in the training text;

the training image and the training text comprise an image-text pair of a plurality of image-text pairs, each image-text pair including a label indicating whether the training image and training text are matched or not matched; and

each text feature comprises a text feature vector in the common embedding space having a same dimensionality as the visual feature vectors and the visual semantic feature vectors; and

training, based on the set of visual semantic features and the set of text features and the label, the target model to determine association information between an input text and an input image.

12 . The device of claim 11 , wherein determining the set of visual semantic features based on the visual semantic dictionary comprises:

determining, from the visual semantic dictionary, at least one candidate semantic feature to which the set of visual features is mapped;

updating the at least one candidate semantic feature based on the set of visual features; and

determining the set of visual semantic features based on the at least one updated candidate semantic feature.

13 . An electronic device, comprising:

a processing unit; and

a memory coupled to the processing unit and comprising instructions stored thereon which, when executed by the processing unit, cause the device to perform acts comprising:

extracting a set of visual features of an input image according to a visual feature extraction sub-model in a trained target model, wherein:

each visual feature corresponds to a pixel block in the input image; and

each visual feature comprises a visual feature vector in a common embedding space;

determining a set of visual semantic features corresponding to the set of visual features based on a visual semantic dictionary, wherein each visual semantic feature comprises a visual semantic feature vector in the common embedding space;

extracting a set of text features of a training text corresponding to a training image according to a text feature extraction sub-model in the trained target model, wherein:

each text feature corresponds to at least one word in the training text;

the training image and the training text comprise an image-text pair of a plurality of image-text pairs, each image-text pair including a label indicating whether the training image and training text are matched or not matched; and

each text feature comprises a text feature vector in the common embedding space having a same dimensionality as the visual feature vectors and the visual semantic feature vectors; and

training, based on the set of visual semantic features and the set of text features and the label, the target model to determine association information between an input text and an input image.

14 . A non-transitory computer storage medium, comprising machine-executable instructions which, when executed by a device, cause the device to perform acts comprising:

extracting a set of visual features of a training image and training text associated with the training image according to a visual feature extraction sub-model in a target model, wherein each visual feature is corresponding to a pixel block in the training image, wherein:

each visual feature comprises a visual feature vector in a common embedding space; and

the training image and the training text comprise an image-text pair of a plurality of image-text pairs, each image-text pair including a label indicating whether the training image and training text are matched or not matched;

determining, based on a visual semantic dictionary, a set of visual semantic features corresponding to the set of visual features, wherein each visual semantic feature comprises a visual semantic feature vector in the common embedding space;

extracting a set of text features of a training text corresponding to the training image according to a text feature extraction sub-model in the target model, wherein:

each text feature is corresponding to at least one word in the training text; and

each text feature comprises a text feature vector in the common embedding space having a same dimensionality as the visual feature vectors and the visual semantic feature vectors; and

training, based on the set of visual semantic features and the set of text features, the target model to determine association information between an input text and an input image.