Multi-modal model training method, apparatus and device, and storage medium
View Patent ↗Provided are a multi-modal model training method, apparatus and device, and a storage medium. The method includes the following steps: obtaining a training sample set, and training a multi-modal model for a plurality of rounds by successively using each of training sample pair in the training sample set: during use of any one of the training sample pairs for training, obtaining an image feature of a target visual sample firstly, and then determining whether back translation needs to be performed on a target original text; when back translation needs to be performed on the target original text, performing corresponding back translation to obtain a target back-translated text, and obtaining a text feature of the target back-translated text; and training the multi-modal model based on the image feature and the text feature.
1 . A multi-modal machine translation model training method, comprising:
obtaining a training sample set, the training sample set comprising a plurality of training sample pairs, each of the training sample pairs comprising a visual sample and a corresponding original text; and
training a multi-modal machine translation model for a plurality of rounds by successively using each of the training sample pairs in the training sample set until reaching a set end condition, wherein
for any one of the training sample pairs in the training sample set, each time the multi-modal machine translation model is trained by using the training sample by the following steps:
obtaining an image feature of a target visual sample comprised in the training sample pair;
determining whether back translation needs to be performed on a target original text comprised in the training sample pair;
when back translation needs to be performed on the target original text comprised in the training sample pair, the original text is input into a first translation model of a back translation module intermediate texts are selected from output texts of the first translation model, and the intermediate texts are input into a second translation model of the back translation module, a target back-translated text is randomly selected among back-translated texts output from the second translation model, and obtaining a text feature of the target back-translated text by performing language feature encoding on the target back-translated text; and
training the multi-modal machine translation model based on the image feature of the target visual sample and the text feature of the target back-translated text.
2 . The method according to claim 1 ,
wherein the back translation module comprises at least two translation components connected in sequence, an input text of a first translation component of the at least two translation components is the target original text, and an output text of a last translation component is in a same language as the target original text.
3 . The method according to claim 2 , wherein during the process of performing back translation on the target original text through a back translation module, an input text of each of the translation components comprised in the back translation module is determined by the following steps:
determining the target original text as the input text of the first translation component comprised in the back translation module; and
for any one of translation components other than the first translation component comprised in the back translation module, determining an input text of the translation component based on output texts of a previous translation component of the translation component.
4 . The method according to claim 3 , wherein the determining an input text of the translation component based on output texts of a previous translation component of the translation component comprises:
calculating a loss rate of each of the output texts of the previous translation component of the translation component; and
selecting part of the output texts of the previous translation component of the translation component based on the loss rates, and determining the selected output texts as the input text of the translation component.
5 . The method according to claim 4 , wherein the selecting part of the output texts of the previous translation component of the translation component based on the loss rates, and determining the selected output texts as the input text of the translation component comprises:
determining N output texts, with the smallest loss rate, of the previous translation component of the translation component as the input text of the translation component; or
determining an output text, with a loss rate less than a preset loss rate threshold, of the previous translation component of the translation component as the input text of the translation component.
6 . The method according to claim 2 , wherein each of the translation components comprised in the back translation module respectively comprises a plurality of translation models.
7 . The method according to claim 1 , wherein the determining whether back translation needs to be performed on a target original text comprised in the training sample pair comprises:
determining whether back translation needs to be performed on the target original text comprised in the training sample pair based on a preset back translation parameter.
8 . The method according to claim 1 , wherein when it is determined that back translation does not need to be performed on the target original text, the method further comprises:
obtaining a text feature of the target original text; and
training the multi-modal machine translation model based on the image feature of the target visual sample and the text feature of the target original text.
9 . A multi-modal machine translation model training device, comprising:
a memory, configured to store a computer program; and
a processor, configured to execute the computer program to:
obtain a training sample set, the training sample set comprising a plurality of training sample pairs, each of the training sample pairs comprising a visual sample and a corresponding original text; and
train a multi-modal machine translation model for a plurality of rounds by successively using each of the training sample pairs in the training sample set until reaching a set end condition, wherein
for any one of the training sample pairs in the training sample set, each time the multi-modal machine translation model is trained by using the training sample by the following steps:
obtaining an image feature of a target visual sample comprised in the training sample pair;
determining whether back translation needs to be performed on a target original text comprised in the training sample pair;
when back translation needs to be performed on the target original text comprised in the training sample pair, the original text is input into a first translation model of a back translation module, intermediate texts are selected from output texts of the first translation model, and the intermediate texts are input into a second translation model of the back translation module, a target back-translated text is randomly selected among bask-translated texts output from the second translation model, and obtaining a text feature of the target back-translated text by performing language feature encoding on the target back-translated text; and
training the multi-modal machine translation model based on the image feature of the target visual sample and the text feature of the target back-translated text.
10 . The multi-modal machine translation model training device according to claim 9 ,
wherein the back translation module comprises at least two translation components connected in sequence, an input text of a first translation component of the at least two translation components is the target original text, and an output text of a last translation component is in a same language as the target original text.
11 . The multi-modal machine translation model training device according to claim 10 , the processor is further configured to:
determine the target original text as the input text of the first translation component comprised in the back translation module; and
for any one of translation components other than the first translation component comprised in the back translation module, determining an input text of the translation component based on output texts of a previous translation component of the translation component.
12 . The multi-modal machine translation model training device according to claim 11 , the processor is further configured to:
calculate a loss rate of each of the output texts of the previous translation component of the translation component; and
select part of the output texts of the previous translation component of the translation component based on the loss rates, and determine the selected output texts as the input text of the translation component.
13 . The multi-modal machine translation model training device according to claim 12 , the processor is further configured to:
determine N output texts, with the smallest loss rate, of the previous translation component of the translation component as the input text of the translation component; or
determine an output text, with a loss rate less than a preset loss rate threshold, of the previous translation component of the translation component as the input text of the translation component.
14 . The multi-modal machine translation model training device according to claim 10 , wherein each of the translation components comprised in the back translation module respectively comprises a plurality of translation models.
15 . The multi-modal machine translation model training device according to claim 9 , the processor is further configured to:
determine whether back translation needs to be performed on the target original text comprised in the training sample pair based on a preset back translation parameter.
16 . The multi-modal machine translation model training device according to claim 9 , when it is determined that back translation does not need to be performed on the target original text, the processor is further configured to:
obtain a text feature of the target original text; and
train the multi-modal machine translation model based on the image feature of the target visual sample and the text feature of the target original text.
17 . A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is configured to, when executed by a processor, cause the processor to:
obtain a training sample set, the training sample set comprising a plurality of training sample pairs, each of the training sample pairs comprising a visual sample and a corresponding original text; and
train a multi-modal machine translation model for a plurality of rounds by successively using each of the training sample pairs in the training sample set until reaching a set end condition, wherein
for any one of the training sample pairs in the training sample set, each time the multi-modal machine translation model is trained by using the training sample by the following steps:
obtaining an image feature of a target visual sample comprised in the training sample pair;
determining whether back translation needs to be performed on a target original text comprised in the training sample pair;
when back translation needs to be performed on the target original text comprised in the training sample pair, the original text is input into a first translation model of a back translation module, intermediate texts are selected from output texts of the first translation model, and the intermediate texts are input into a second translation model of the back translation module, a target back-translated text is randomly selected among back-translated texts output from the second translation model, and obtaining a text feature of the target back-translated text by performing language feature encoding on the target back-translated text; and
training the multi-modal machine translation model based on the image feature of the target visual sample and the text feature of the target back-translated text.