Multimodal data processing
Embodiments of the present disclosure provide a solution for multimodal data processing. A method comprises: obtaining image data and text data; and extracting a target visual feature of image data and a target textual feature of text data using a feature extraction model. The feature extraction model comprises alternatively deployed cross-modal encoding parts and visual encoding parts. The extracting comprises: performing, using a first cross-modal encoding part of the feature extraction model, cross-modal feature encoding on a first intermediate visual feature of the image data and a first intermediate textual feature of the text data, to obtain a second intermediate visual feature and a second intermediate textual feature; performing, using a first visual encoding part of the feature extraction model, visual modal feature encoding on the second intermediate visual feature, to obtain a third intermediate visual feature.
1 . A method for multimodal data processing, comprising:
obtaining image data and text data; and
extracting, using a feature extraction model, a target visual feature of the image data and a target textual feature of the text data, the feature extraction model comprises alternately deployed cross-modal encoding parts and visual encoding parts, and the extracting comprises:
performing, using a first cross-modal encoding part of the feature extraction model, cross-modal feature encoding on a first intermediate visual feature of the image data and a first intermediate textual feature of the text data, to obtain a second intermediate visual feature and a second intermediate textual feature;
performing, using a first visual encoding part of the feature extraction model, visual modal feature encoding on the second intermediate visual feature, to obtain a third intermediate visual feature;
performing, using a second cross-modal encoding part of the feature extraction model, cross-modal feature encoding on the third intermediate visual feature and the second intermediate textual feature, to obtain a fourth intermediate visual feature and a third intermediate textual feature; and
determining the target visual feature and the target textual feature based on the fourth intermediate visual feature and the third intermediate textual feature.
2 . The method of claim 1 , further comprising:
determining a match degree between the image data and the text data based on the target visual feature and the target textual feature.
3 . The method of claim 1 , wherein determining the target visual feature and the target textual feature based on the fourth intermediate visual feature and the third intermediate textual feature comprises:
performing visual modal feature encoding on the fourth intermediate visual feature using a second visual encoding part of the feature extraction model, to obtain a fifth intermediate visual feature; and
determining the target visual feature and the target textual feature based on the fifth intermediate visual feature and the third intermediate textual feature.
4 . The method of claim 1 , wherein the feature extraction model comprises a plurality of pairs of cross-modal encoding parts and visual encoding parts that are alternately deployed, and wherein a visual encoding part deployed between two adjacent cross-modal encoding parts comprises a predetermined number of visual encoding layers.
5 . The method of claim 1 , wherein the feature extraction model comprises a plurality of pairs of alternating cross-modal encoding parts and visual encoding parts, and wherein a visual encoding part deployed between a first pair of adjacent cross-modal encoding parts comprises a first number of visual encoding layers, and a visual encoding part deployed between a second pair of adjacent cross-modal encoding parts comprises a second number of visual encoding layers, the first number being different from the second number.
6 . The method of claim 1 , wherein the image data and the text data are comprised in training data for the feature extraction model, and wherein the image data comprises a plurality of video frames in a video clip, wherein extracting the target visual feature and the textual feature comprises:
generating a plurality of masked video frames by masking at least one visual block of at least one of the plurality of video frames; and
extracting target visual features of the plurality of masked video frames and a target textual feature of the text data using the feature extraction model.
7 . The method of claim 6 , further comprising:
performing parameter update to the feature extraction model based on the target visual features and the target textual feature.
8 . The method of claim 6 , wherein generating the plurality of masked video frames comprises:
randomly masking at least one visual block of respective ones of the plurality of video frames, to obtain the plurality of masked video frames.
9 . The method of claim 6 , wherein generating the plurality of masked video frames comprises:
masking respective ones of the plurality of video frames using a predetermined mask map, to obtain the plurality of masked video frames, the predetermined mask map indicating at least one visual block to be masked at a predetermined position in a video frame.
10 . The method of claim 6 , wherein generating the plurality of masked video frames comprises:
selecting at least one visual block for masking from respective ones of the plurality of video frames, to obtain the plurality of masked video frames, positions of the masked visual blocks in the plurality of video frames being different from each other.
11 . An electronic device, comprising:
at least one processing unit; and
at least one memory coupled to the at least one processing unit and storing instructions executable by the at least one processing unit, the instructions, upon execution by the at least one processing unit, causing the device to perform:
obtaining image data and text data; and
extracting, using a feature extraction model, a target visual feature of the image data and a target textual feature of the text data, the feature extraction model comprises alternately deployed cross-modal encoding parts and visual encoding parts, and the extracting comprises:
performing, using a first cross-modal encoding part of the feature extraction model, cross-modal feature encoding on a first intermediate visual feature of the image data and a first intermediate textual feature of the text data, to obtain a second intermediate visual feature and a second intermediate textual feature;
performing, using a first visual encoding part of the feature extraction model, visual modal feature encoding on the second intermediate visual feature, to obtain a third intermediate visual feature;
performing, using a second cross-modal encoding part of the feature extraction model, cross-modal feature encoding on the third intermediate visual feature and the second intermediate textual feature, to obtain a fourth intermediate visual feature and a third intermediate textual feature; and
determining the target visual feature and the target textual feature based on the fourth intermediate visual feature and the third intermediate textual feature.
12 . The device of claim 11 , wherein determining the target visual feature and the target textual feature based on the fourth intermediate visual feature and the third intermediate textual feature comprises:
performing visual modal feature encoding on the fourth intermediate visual feature using a second visual encoding part of the feature extraction model, to obtain a fifth intermediate visual feature; and
determining the target visual feature and the target textual feature based on the fifth intermediate visual feature and the third intermediate textual feature.
13 . The device of claim 11 , wherein the feature extraction model comprises a plurality of pairs of cross-modal encoding parts and visual encoding parts that are alternately deployed, and wherein a visual encoding part deployed between two adjacent cross-modal encoding parts comprises a predetermined number of visual encoding layers.
14 . The device of claim 11 , wherein the feature extraction model comprises a plurality of pairs of alternating cross-modal encoding parts and visual encoding parts, and wherein a visual encoding part deployed between a first pair of adjacent cross-modal encoding parts comprises a first number of visual encoding layers, and a visual encoding part deployed between a second pair of adjacent cross-modal encoding parts comprises a second number of visual encoding layers, the first number being different from the second number.
15 . The device of claim 11 , wherein the image data and the text data are comprised in training data for the feature extraction model, and wherein the image data comprises a plurality of video frames in a video clip, wherein extracting the target visual feature and the textual feature comprises:
generating a plurality of masked video frames by masking at least one visual block of at least one of the plurality of video frames; and
extracting target visual features of the plurality of masked video frames and a target textual feature of the text data using the feature extraction model.
16 . The device of claim 15 , wherein the acts further comprise:
performing parameter update to the feature extraction model based on the target visual features and the target textual feature.
17 . The device of claim 15 , wherein generating the plurality of masked video frames comprises:
randomly masking at least one visual block of respective ones of the plurality of video frames, to obtain the plurality of masked video frames.
18 . The device of claim 15 , wherein generating the plurality of masked video frames comprises:
masking respective ones of the plurality of video frames using a predetermined mask map, to obtain the plurality of masked video frames, the predetermined mask map indicating at least one visual block to be masked at a predetermined position in a video frame.
19 . The device of claim 15 , wherein generating the plurality of masked video frames comprises:
selecting at least one visual block for masking from respective ones of the plurality of video frames, to obtain the plurality of masked video frames, positions of the masked visual blocks in the plurality of video frames being different from each other.
20 . A non-transitory computer-readable storage medium, having a computer program stored thereon which, upon execution by a processor, causes the following acts to be performed:
obtaining image data and text data; and
extracting, using a feature extraction model, a target visual feature of the image data and a target textual feature of the text data, the feature extraction model comprises alternately deployed cross-modal encoding parts and visual encoding parts, and the extracting comprises:
performing, using a first cross-modal encoding part of the feature extraction model, cross-modal feature encoding on a first intermediate visual feature of the image data and a first intermediate textual feature of the text data, to obtain a second intermediate visual feature and a second intermediate textual feature;
performing, using a first visual encoding part of the feature extraction model, visual modal feature encoding on the second intermediate visual feature, to obtain a third intermediate visual feature;
performing, using a second cross-modal encoding part of the feature extraction model, cross-modal feature encoding on the third intermediate visual feature and the second intermediate textual feature, to obtain a fourth intermediate visual feature and a third intermediate textual feature; and
determining the target visual feature and the target textual feature based on the fourth intermediate visual feature and the third intermediate textual feature.