Granted Patent B2

US 12,277,401 · App. 17/502,108 · Granted Apr 15, 2025

Inventors: Guocheng Niu (Beijing, CN); Wei Li (Beijing, CN); Can Gao (Beijing, CN); Xinyan Xiao (Beijing, CN); Hua Wu (Beijing, CN)

Assignee: BEIJING BAIDU NETCOM SCIENCE TECHNOLOGY CO., LTD.

G06F40/58G06F18/256G06F40/205G06F40/47G06N3/02

Abstract

Claims (46)

1. A method for training a pre-trained model, comprising:

acquiring training data, the training data comprising a single-modal language material comprising an image or a text and a multi-modal language material comprising an image-text pair which is semantically in pairs, and the multi-modal language material comprising a language material pair formed by a first-modal language material which is a text in a first language and a second-modal language material which is a text in a second language;

performing at least one of rewriting extension and retrieval extension on the multi-modal language material in the training data, and adding the extended multi-modal language material into the training data;

performing a multi-task training operation on a pre-trained model using the training data, the multi-task comprising at least one cross-modal contrastive learning task and at least one single-modal learning task;

wherein the cross-modal contrastive learning task which is trained utilizing the multi-modal language material comprises: determining similarity between the first-modal language material and the second-modal language material in the multi-modal language material utilizing a vector representation of the first-modal language material and a vector representation of the second-modal language material in the multi-modal language material by the pre-trained model, with a training target of maximizing the similarity between the first-modal language material and the second-modal language material in a positive multi-modal language material and minimizing the similarity between the first-modal language material and the second-modal language material in a negative multi-modal language material; and

the single-modal learning task which is trained utilizing the single-modal language material comprises: predicting a second part of content in the single-modal language material utilizing a vector representation of a first part of content in the single-modal language material by the pre-trained model, with a training target of minimizing a difference between the predicted second part of content and the second part of content in the single-modal language material,

wherein in the cross-modal contrastive learning task, the similarity between the first-modal language material and the second-modal language material in the multi-modal language material obtained by the retrieval extension is determined by: calculating similarity between a vector representation of the first-modal language material obtained by the pre-trained model and a vector representation of the second-modal language material obtained by the pre-trained model; and

the similarity between the first-modal language material and the second-modal language material in the multi-modal language material obtained by the rewriting extension is determined by: stitching the first-modal language material and the second-modal language material, and mapping a vector representation of the stitched language material obtained by the pre-trained model into a similarity value,

wherein parameters of the pre-trained model are updated using a constructed total loss function when the multi-task training operation is performed;

the total loss function is obtained by an arithmetic sum of the loss function of the at least one cross-modal contrastive learning task and the loss function of the at least one single-modal learning task, wherein the loss function of the at least one cross-modal contrastive learning task is constructed according to: a similarity value calculated by stitching the image and the text, and mapping a vector representation of the stitched language material obtained by the pre-trained model; or a similarity calculated by calculating cosine similarity between a vector representation of the image obtained by the pre-trained model and a vector representation of the text obtained by the pre-trained model, and the loss function of the at least one single-modal learning task comprises the loss function of a visual learning task and the loss function of a text learning task, and

adjusting finely the pre-trained model according to training data corresponding to a downstream task comprising a text classification task, an image classification task, a task of generating questions and answers for images, or a task of generating images for texts.

2. The method according to claim 1 , wherein the rewriting extension comprises:

translating the text in the first language of the positive multi-modal language material into a text in the second language and then translating the text in the second language back to a text in the first language using a translation model, and constructing a new positive multi-modal language material by the text in the first language obtained after translation and the second-modal language material in the positive multi-modal language material; or

parsing the first-modal language material in the positive multi-modal language material into a scenario graph, randomly replacing at least one of an entity, an attribute and a relationship in the scenario graph, converting the scenario graph obtained after replacement back into the first-modal language material, and constructing a new negative multi-modal language material by the first-modal language material obtained after conversion and the second-modal language material in the positive multi-modal language material.

3. The method according to claim 1 , wherein the retrieval extension comprises:

retrieving the first-modal language material of the positive multi-modal language material in a first-modal corpus, and constructing a new positive multi-modal language material using the retrieved language material and the second-modal language material in the positive multi-modal language material.

4. An electronic device, comprising:

at least one processor; and

a memory communicatively connected with the at least one processor;

wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform a method for training a pre-trained model, wherein the method comprises:

acquiring training data, the training data comprising a single-modal language material comprising an image or a text and a multi-modal language material comprising an image-text pair which is semantically in pairs, and the multi-modal language material comprising a language material pair formed by a first-modal language material which is a text in a first language and a second-modal language material which is a text in a second language;

performing at least one of rewriting extension and retrieval extension on the multi-modal language material in the training data, and adding the extended multi-modal language material into the training data;

performing a multi-task training operation on a pre-trained model using the training data, the multi-task comprising at least one cross-modal contrastive learning task and at least one single-modal learning task;

wherein the cross-modal contrastive learning task which is trained utilizing the multi-modal language material comprises: determining similarity between the first-modal language material and the second-modal language material in the multi-modal language material utilizing a vector representation of the first-modal language material and a vector representation of the second-modal language material in the multi-modal language material by the pre-trained model, with a training target of maximizing the similarity between the first-modal language material and the second-modal language material in a positive multi-modal language material and minimizing the similarity between the first-modal language material and the second-modal language material in a negative multi-modal language material; and

the single-modal learning task which is trained utilizing the single-modal language material comprises: predicting a second part of content in the single-modal language material utilizing a vector representation of a first part of content in the single-modal language material by the pre-trained model, with a training target of minimizing a difference between the predicted second part of content and the second part of content in the single-modal language material,

wherein in the cross-modal contrastive learning task, the similarity between the first-modal language material and the second-modal language material in the multi-modal language material obtained by the retrieval extension is determined by: calculating similarity between a vector representation of the first-modal language material obtained by the pre-trained model and a vector representation of the second-modal language material obtained by the pre-trained model; and

the similarity between the first-modal language material and the second-modal language material in the multi-modal language material obtained by the rewriting extension is determined by: stitching the first-modal language material and the second-modal language material, and mapping a vector representation of the stitched language material obtained by the pre-trained model into a similarity value,

wherein parameters of the pre-trained model are updated using a constructed total loss function when the multi-task training operation is performed;

the total loss function is obtained by an arithmetic sum of the loss function of the at least one cross-modal contrastive learning task and the loss function of the at least one single-modal learning task, wherein the loss function of the at least one cross-modal contrastive learning task is constructed according to: a similarity value calculated by stitching the image and the text, and mapping a vector representation of the stitched language material obtained by the pre-trained model; or a similarity calculated by calculating cosine similarity between a vector representation of the image obtained by the pre-trained model and a vector representation of the text obtained by the pre-trained model, and the loss function of the at least one single-modal learning task comprises the loss function of a visual learning task and the loss function of a text learning task, and

adjusting finely the pre-trained model according to training data corresponding to a downstream task comprising a text classification task, an image classification task, a task of generating questions and answers for images, or a task of generating images for texts.

5. The electronic device according to claim 4 , wherein the rewriting extension comprises:

translating the text in the first language of the positive multi-modal language material into a text in the second language and then translate the text in the second language back to a text in the first language using a translation model, and constructing a new positive multi-modal language material by the text in the first language obtained after translation and the second-modal language material in the positive multi-modal language material; or

parsing the first-modal language material in the positive multi-modal language material into a scenario graph, randomly replace at least one of an entity, an attribute and a relationship in the scenario graph, converting the scenario graph obtained after replacement back into the first-modal language material, and constructing a new negative multi-modal language material by the first-modal language material obtained after conversion and the second-modal language material in the positive multi-modal language material.

6. The electronic device according to claim 4 , wherein the retrieval extension comprises:

retrieving the first-modal language material of the positive multi-modal language material in a first-modal corpus, and constructing a new positive multi-modal language material using the retrieved language material and the second-modal language material in the positive multi-modal language material.

7. A non-transitory computer readable storage medium with computer instructions stored thereon, wherein the computer instructions are used for causing a computer to perform a method for training a pre-trained model, wherein the method comprises:

acquiring training data, the training data comprising a single-modal language material comprising an image or a text and a multi-modal language material comprising an image-text pair which is semantically in pairs, and the multi-modal language material comprising a language material pair formed by a first-modal language material which is a text in a first language and a second-modal language material which is a text in a second language;

performing at least one of rewriting extension and retrieval extension on the multi-modal language material in the training data, and adding the extended multi-modal language material into the training data;

performing a multi-task training operation on a pre-trained model using the training data, the multi-task comprising at least one cross-modal contrastive learning task and at least one single-modal learning task;

wherein the cross-modal contrastive learning task which is trained utilizing the multi-modal language material comprises: determining similarity between the first-modal language material and the second-modal language material in the multi-modal language material utilizing a vector representation of the first-modal language material and a vector representation of the second-modal language material in the multi-modal language material by the pre-trained model, with a training target of maximizing the similarity between the first-modal language material and the second-modal language material in a positive multi-modal language material and minimizing the similarity between the first-modal language material and the second-modal language material in a negative multi-modal language material; and

the single-modal learning task which is trained utilizing the single-modal language material comprises: predicting a second part of content in the single-modal language material utilizing a vector representation of a first part of content in the single-modal language material by the pre-trained model, with a training target of minimizing a difference between the predicted second part of content and the second part of content in the single-modal language material,

wherein in the cross-modal contrastive learning task, the similarity between the first-modal language material and the second-modal language material in the multi-modal language material obtained by the retrieval extension is determined by: calculating similarity between a vector representation of the first-modal language material obtained by the pre-trained model and a vector representation of the second-modal language material obtained by the pre-trained model; and

the similarity between the first-modal language material and the second-modal language material in the multi-modal language material obtained by the rewriting extension is determined by: stitching the first-modal language material and the second-modal language material, and mapping a vector representation of the stitched language material obtained by the pre-trained model into a similarity value,

wherein parameters of the pre-trained model are updated using a constructed total loss function when the multi-task training operation is performed;

the total loss function is obtained by an arithmetic sum of the loss function of the at least one cross-modal contrastive learning task and the loss function of the at least one single-modal learning task, wherein the loss function of the at least one cross-modal contrastive learning task is constructed according to: a similarity value calculated by stitching the image and the text, and mapping a vector representation of the stitched language material obtained by the pre-trained model; or a similarity calculated by calculating cosine similarity between a vector representation of the image obtained by the pre-trained model and a vector representation of the text obtained by the pre-trained model, and the loss function of the at least one single-modal learning task comprises the loss function of a visual learning task and the loss function of a text learning task, and

adjusting finely the pre-trained model according to training data corresponding to a downstream task comprising a text classification task, an image classification task, a task of generating questions and answers for images, or a task of generating images for texts.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (33)