Difference description statement generation method and apparatus, device and medium
A difference description statement generation method includes: performing feature concatenation on an image encoding feature and a text encoding feature; inputting a concatenated encoding feature to a preset image-text alignment unit constructed based on a preset self-attention mechanism to obtain a concatenated alignment feature; processing, by using a preset noise monitoring unit constructed based on the preset self-attention mechanism and a preset cross-attention mechanism, the text encoding feature, and an image alignment feature and a text alignment feature that are obtained by splitting the concatenated alignment feature respectively, to extract a difference signal; and generating a difference description statement based on the difference signal by using a preset difference description generation algorithm.
1. A difference description statement generation method, comprising:
encoding a target image and target text respectively, and performing feature concatenation on an image encoding feature and a text encoding feature that are obtained by encoding, to obtain a concatenated encoding feature;
inputting the concatenated encoding feature to a preset image-text alignment unit constructed based on a preset self-attention mechanism to perform image-text alignment processing, to obtain a concatenated alignment feature;
splitting the concatenated alignment feature to obtain an image alignment feature and a text alignment feature, and inputting the image alignment feature, the text encoding feature, and the text alignment feature to a preset noise monitoring unit constructed based on the preset self-attention mechanism and a preset cross-attention mechanism to perform processing, to extract a difference signal between the target image and the target text; and
generating a difference description statement based on the difference signal by using a preset difference description generation algorithm.
2. The difference description statement generation method according to claim 1 , wherein the encoding the target image and the target text respectively comprises:
encoding the target image and the target text by using an image encoder.
3. The difference description statement generation method according to claim 2 , wherein the image encoder for encoding the target image is a residual neural network (ResNet) or a deep neural network ResNeXt.
4. The difference description statement generation method according to claim 2 , wherein the image encoder for encoding the target text is robustly optimized bidirectional encoder representation from transformers (BERT) pretraining approach (RoBERTa) or BERT.
5. The difference description statement generation method according to claim 1 , wherein before the inputting the concatenated encoding feature to the preset image-text alignment unit constructed based on the preset self-attention mechanism to perform image-text alignment processing, the method further comprises:
constructing an image-text alignment subunit based on a self-attention operation unit, a layer normalization unit, and a feature addition unit; and
connecting a first preset quantity of image-text alignment subunits in series to construct the preset image-text alignment unit.
6. The difference description statement generation method according to claim 5 , wherein the inputting the concatenated encoding feature to the preset image-text alignment unit constructed based on the preset self-attention mechanism to perform image-text alignment processing comprises:
determining a first image-text alignment subunit in the preset image-text alignment unit as a current image-text alignment subunit, and determining the concatenated encoding feature as a feature to be processed;
inputting the feature to be processed to the current image-text alignment subunit;
performing a self-attention operation, a layer normalization operation, and a feature addition operation sequentially on the feature to be processed by using the current image-text alignment subunit, to obtain a corresponding current operation processing result; and
in response to the current image-text alignment subunit being not a last image-text alignment subunit, updating the current image-text alignment subunit to a next image-text alignment subunit, updating the feature to be processed to the current operation processing result, and performing the inputting the feature to be processed to the current image-text alignment subunit.
7. The difference description statement generation method according to claim 6 , further comprising:
in response to the current image-text alignment subunit being the last image-text alignment subunit, determining the current operation processing result as the concatenated alignment feature.
8. The difference description statement generation method according to claim 6 , wherein the performing the self-attention operation, the layer normalization operation, and the feature addition operation sequentially on the feature to be processed by using the current image-text alignment subunit comprises:
performing the self-attention operation on the feature to be processed by using a self-attention operation unit in the current image-text alignment subunit, to obtain a first operation feature;
performing layer normalization processing on the first operation feature by using a layer normalization unit in the current image-text alignment subunit, to obtain a second operation feature; and
performing the feature addition operation on the second operation feature and the feature to be processed by using a feature addition unit in the current image-text alignment subunit, to obtain the current operation processing result.
9. The difference description statement generation method according to claim 1 , wherein before the splitting the concatenated alignment feature to obtain the image alignment feature and the text alignment feature, and inputting to the preset noise monitoring unit constructed based on the preset self-attention mechanism and the preset cross-attention mechanism to perform processing, the method further comprises:
constructing a noise monitoring subunit based on a self-attention operation unit, a layer normalization unit, a feature addition unit, a cross-attention operation unit, and an attention superposition unit; and
connecting a second preset quantity of noise monitoring subunits in series to construct the preset noise monitoring unit.
10. The difference description statement generation method according to claim 9 , wherein the splitting the concatenated alignment feature to obtain the image alignment feature and the text alignment feature, and inputting the image alignment feature, the text encoding feature, and the text alignment feature to the preset noise monitoring unit constructed based on the preset self-attention mechanism and the preset cross-attention mechanism to perform processing, to extract the difference signal between the target image and the target text comprises:
determining a first noise monitoring subunit in the preset noise monitoring unit as a current noise monitoring subunit, and determining the image alignment feature, the text encoding feature, and the text alignment feature all as features to be processed;
inputting the features to be processed to the current noise monitoring subunit;
performing a self-attention operation, a layer normalization operation, and a feature addition operation sequentially on the features to be processed by using the current noise monitoring subunit, to obtain current first operation processing results respectively corresponding to the image alignment feature, the text encoding feature, and the text alignment feature; performing a cross-attention operation, a layer normalization operation, and a feature addition operation on the first operation processing results respectively corresponding to the image alignment feature, the text encoding feature, and the text alignment feature respectively, to obtain current second operation processing results respectively corresponding to the image alignment feature, the text encoding feature, and the text alignment feature; and
in response to the current noise monitoring subunit being not a last noise monitoring subunit, updating the current noise monitoring subunit to a next noise monitoring subunit, updating the features to be processed to the current second operation processing results respectively corresponding to the image alignment feature, the text encoding feature, and the text alignment feature, and performing the inputting the features to be processed to the current noise monitoring subunit.
11. The difference description statement generation method according to claim 10 , further comprising:
in response to the current noise monitoring subunit being the last noise monitoring subunit, determining the current second operation processing result as the difference signal between the target image and the target text.
12. The difference description statement generation method according to claim 10 , wherein the process of performing the cross-attention operation, the layer normalization operation, and the feature addition operation on the first operation processing result corresponding to the text encoding feature respectively comprises:
performing the cross-attention operation on the first operation processing result corresponding to the text encoding feature to obtain a corresponding cross-attention operation result; inputting the cross-attention operation result to an attention superposition unit constructed based on a preset formula to perform a corresponding attention superposition operation, to obtain a corresponding attention superposition operation result; and
performing the layer normalization operation and the feature addition operation sequentially on the attention superposition operation result to obtain the corresponding second operation processing result.
13. The difference description statement generation method according to claim 12 , wherein the cross-attention operation comprises a cross-attention operation between the first operation processing result corresponding to the text encoding feature, and each of the first operation processing result corresponding to the image alignment feature, and the first operation processing result corresponding to the text alignment feature.
14. The difference description statement generation method according to claim 12 , wherein the preset formula is:
f =attn cross ( f align ,g raw )+attn cross ( g align ,g raw ),
wherein
f represents the attention superposition operation result, f align represents the image alignment feature, g align represents the text encoding feature, g raw represents the text alignment feature, and attn cross represents the cross-attention operation.
15. The difference description statement generation method according to claim 1 , wherein the generating the difference description statement based on the difference signal by using the preset difference description generation algorithm comprises:
inputting the difference signal, a third preset quantity of preset reference description statements, and a predictor to a difference description generation unit constructed based on the preset difference description generation algorithm, and outputting, at a location corresponding to the predictor by using the difference description generation unit, a corresponding prediction result and a distribution probability corresponding to the prediction result;
inputting the distribution probability to a preset text generator to calculate a score of each prediction result, and adding the score of each prediction result and a score of the preset reference description statement to obtain a score of a current description statement;
sorting the score of the current description statement in descending order, and selecting, according to a sorting order, a third preset quantity of current description statements from a result of sorting in descending order as new preset reference description statements; and
in response to the prediction result being not a preset terminator, performing the inputting the difference signal, the third preset quantity of preset reference description statements, and the predictor to the difference description generation unit constructed based on the preset difference description generation algorithm.
16. The difference description statement generation method according to claim 15 , further comprising:
in response to the prediction result being the preset terminator, selecting a current description statement with a highest score as the difference description statement.
17. An electronic device, comprising:
a memory storing computer-readable instructions; and
one or more processors configure to execute the computer-readable instructions, wherein upon execution of the computer-readable instructions, the one or more processors is configured to:
encode a target image and target text respectively, and perform feature concatenation on an image encoding feature and a text encoding feature that are obtained by encoding, to obtain a concatenated encoding feature;
input the concatenated encoding feature to a preset image-text alignment unit constructed based on a preset self-attention mechanism to perform image-text alignment processing, to obtain a concatenated alignment feature;
split the concatenated alignment feature to obtain an image alignment feature and a text alignment feature, and input the image alignment feature, the text encoding feature, and the text alignment feature to a preset noise monitoring unit constructed based on the preset self-attention mechanism and a preset cross-attention mechanism to perform processing, to extract a difference signal between the target image and the target text; and
generate a difference description statement based on the difference signal by using a preset difference description generation algorithm.
18. One or more non-volatile computer-readable storage media, storing computer-readable instructions, wherein the computer-readable instructions, upon execution by one or more processors, are configured to cause the one or more processors to:
encode a target image and target text respectively, and perform feature concatenation on an image encoding feature and a text encoding feature that are obtained by encoding, to obtain a concatenated encoding feature;
input the concatenated encoding feature to a preset image-text alignment unit constructed based on a preset self-attention mechanism to perform image-text alignment processing, to obtain a concatenated alignment feature;
split the concatenated alignment feature to obtain an image alignment feature and a text alignment feature, and input the image alignment feature, the text encoding feature, and the text alignment feature to a preset noise monitoring unit constructed based on the preset self-attention mechanism and a preset cross-attention mechanism to perform processing, to extract a difference signal between the target image and the target text; and
generate a difference description statement based on the difference signal by using a preset difference description generation algorithm.
19. The difference description statement generation method according to claim 8 , wherein a matching relationship between the target image and a text is learned by using the preset image-text alignment unit, so as to locate parts in which image and text features cannot be aligned as error parts.
20. The difference description statement generation method according to claim 15 , wherein the preset reference description statement includes no content in response to prediction being performed for the first time by using the difference description generation unit, and the preset reference description statement is formed in response to prediction being cyclically performed over and over again.