Granted Patent B2

US 12,670,236 · App. 17/502,385 · Granted Jun 30, 2026

Inventors: Feng He (Beijing, CN); Qi Wang (Beijing, CN); Zhifan Feng (Beijing, CN); Hu Yang (Beijing, CN); Chunguang Chai (Beijing, CN)

Assignee: BEIJING BAIDU NETCOM SCIENCE TECHNOLOGY CO., LTD.

G06F18/256G06F18/2178G06F18/22

Abstract

Claims (64)

1 . A computer-implemented method for training a cross-modal retrieval model which is used by a cross-modal retrieval system, wherein a cross-modal retrieval is used for a retrieval of data of one modal using data of another modal, the method comprising:

determining a similarity of a cross-modal sample pair according to the cross-modal sample pair, the cross-modal sample pair comprising a sample of a first modal and a sample of a second modal, and the first modal being different from the second modal, wherein the first modal is a text, and the second modal is a video, and the cross-modal sample pair comprises a positive sample pair and a negative sample pair, the positive sample pair comprises an anchor sample and a positive sample, the negative sample pair comprises the anchor sample and a negative sample, the anchor sample has the first modal, and the positive sample and the negative sample have the second modal, and wherein the anchor sample is a text in a sample set, the positive sample is a video related to the text in the sample set, and the negative sample is a randomly selected video which is related or not related to the text in the sample set;

determining a soft margin based on the similarity, and determining a soft margin loss function based on the soft margin, wherein the soft margin is a non-fixed value; and

determining a total loss function based on the soft margin loss function, and

training a cross-modal retrieval model according to the total loss function,

using the trained cross-modal retrieval model to receive a text input by a user, determine a video matched with the text using the cross-modal retrieval model, and feed the matched video back to the user,

wherein the determining a soft margin based on the similarity comprises:

calculating a distance between similarity of the positive sample pair and similarity of the negative sample pair to obtain a similarity distance; and

normalizing the similarity distance to obtain a normalized similarity distance, and determining the normalized similarity distance as the soft margin, and

wherein the similarity distance comprises a similarity distance in the first modal and a similarity distance in the second modal, the similarity distance in the first modal being obtained by processing the sample pair in the first modal using a semantic representation model in the first modal and the similarity distance in the second modal being obtained by processing the sample pair in the second modal using a semantic representation model in the second modal;

the determining a soft margin based on the similarity and determining a soft margin loss function based on the soft margin comprises:

determining a soft margin in the first modal based on the similarity distance in the first modal, and calculating a contrastive loss function in the first modal based on the soft margin in the first modal;

determining a soft margin in the second modal based on the similarity distance in the second modal, and calculating a contrastive loss function in the second modal based on the soft margin in the second modal; and

calculating the soft margin loss function according to the contrastive loss function in the first modal and the contrastive loss function in the second modal.

2 . The method according to claim 1 , wherein the cross-modal sample pair corresponds to at least one contrastive sample set, and the determining a total loss function based on the soft margin loss function comprises:

calculating a loss function of the corresponding contrastive sample set based on the soft margin loss function; and

calculating the total loss function based on the loss function of each corresponding sample set of the at least one contrastive sample set.

3 . The method according to claim 2 , wherein the soft margin loss function comprises a soft margin loss function for at least one state, and the calculating a loss function of the corresponding contrastive sample set based on the soft margin loss function comprises:

performing weighted summation on the soft margin loss function for the at least one state to obtain a weighted summation function; and

adding the weighted summation function and a hard margin loss function, and calculating the loss function of the corresponding contrastive sample set based on the added function.

4 . The method according to claim 1 , wherein the similarity distance comprises a static similarity distance and a dynamic similarity distance, the soft margin loss function comprises a static soft margin loss function and a dynamic soft margin loss function, the static soft margin loss function is calculated based on the static similarity distance, the dynamic soft margin loss function is calculated based on the dynamic similarity distance, and the calculating a distance between similarity of the positive sample pair and similarity of the negative sample pair to obtain a similarity distance comprises:

calculating the distance between the similarity of the positive sample pair and the similarity of the negative sample pair using a pre-training model, so as to obtain the static similarity distance; and/or

calculating the distance between the similarity of the positive sample pair and the similarity of the negative sample pair using current parameters of the cross-modal retrieval model to obtain the dynamic similarity distance.

5 . The method according to claim 1 , wherein the cross-modal sample pair corresponds to at least one contrastive sample set, and the determining a total loss function based on the soft margin loss function comprises:

calculating a loss function of the corresponding contrastive sample set based on the soft margin loss function; and

calculating the total loss function based on the loss function of each corresponding sample set of the at least one contrastive sample set.

6 . An electronic device, comprising:

at least one processor; and

a memory communicatively connected with the at least one processor;

wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform a method for training a cross-modal retrieval model which is used by a cross-modal retrieval system, wherein a cross-modal retrieval is used for a retrieval of data of one modal using data of another modal, wherein the method comprises:

determining a similarity of a cross-modal sample pair according to the cross-modal sample pair, the cross-modal sample pair comprising a sample of a first modal and a sample of a second modal, and the first modal being different from the second modal, wherein the first modal is a text, and the second modal is a video, and the cross-modal sample pair comprises a positive sample pair and a negative sample pair, the positive sample pair comprises an anchor sample and a positive sample, the negative sample pair comprises the anchor sample and a negative sample, the anchor sample has the first modal, and the positive sample and the negative sample have the second modal, and wherein the anchor sample is a text in a sample set, the positive sample is a video related to the text in the sample set, and the negative sample is a randomly selected video which is related or not related to the text in the sample set;

determining a soft margin based on the similarity, and determining a soft margin loss function based on the soft margin, wherein the soft margin is a non-fixed value; and

determining a total loss function based on the soft margin loss function, and training a cross-modal retrieval model according to the total loss function,

using the trained cross-modal retrieval model to receive a text input by a user, determine a video matched with the text using the cross-modal retrieval model, and feed the matched video back to the user,

wherein the determining a soft margin based on the similarity comprises:

calculating a distance between similarity of the positive sample pair and similarity of the negative sample pair to obtain a similarity distance; and

normalizing the similarity distance to obtain a normalized similarity distance, and determining the normalized similarity distance as the soft margin, and

wherein the similarity distance comprises a similarity distance in the first modal and a similarity distance in the second modal, the similarity distance in the first modal being obtained by processing the sample pair in the first modal using a semantic representation model in the first modal and the similarity distance in the second modal being obtained by processing the sample pair in the second modal using a semantic representation model in the second modal;

the determining a soft margin based on the similarity and determining a soft margin loss function based on the soft margin comprises:

determining a soft margin in the first modal based on the similarity distance in the first modal, and calculating a contrastive loss function in the first modal based on the soft margin in the first modal;

determining a soft margin in the second modal based on the similarity distance in the second modal, and calculating a contrastive loss function in the second modal based on the soft margin in the second modal; and

calculating the soft margin loss function according to the contrastive loss function in the first modal and the contrastive loss function in the second modal.

7 . The electronic device according to claim 6 , wherein the cross-modal sample pair corresponds to at least one contrastive sample set, and the determining a total loss function based on the soft margin loss function comprises:

calculating a loss function of the corresponding contrastive sample set based on the soft margin loss function; and

calculating the total loss function based on the loss function of each corresponding sample set of the at least one contrastive sample set.

8 . The electronic device according to claim 7 , wherein the soft margin loss function comprises a soft margin loss function for at least one state, and the calculating a loss function of the corresponding contrastive sample set based on the soft margin loss function comprises:

performing weighted summation on the soft margin loss function for the at least one state to obtain a weighted summation function; and

adding the weighted summation function and a hard margin loss function, and calculating the loss function of the corresponding contrastive sample set based on the added function.

9 . The electronic device according to claim 6 , wherein the similarity distance comprises a static similarity distance and a dynamic similarity distance, the soft margin loss function comprises a static soft margin loss function and a dynamic soft margin loss function, the static soft margin loss function is calculated based on the static similarity distance, the dynamic soft margin loss function is calculated based on the dynamic similarity distance, and the calculating a distance between similarity of the positive sample pair and similarity of the negative sample pair to obtain a similarity distance comprises:

calculating the distance between the similarity of the positive sample pair and the similarity of the negative sample pair using a pre-training model, so as to obtain the static similarity distance; and/or

calculating the distance between the similarity of the positive sample pair and the similarity of the negative sample pair using current parameters of the cross-modal retrieval model to obtain the dynamic similarity distance.

10 . A non-transitory computer readable storage medium with computer instructions stored thereon, wherein the computer instructions are used for causing a computer to perform a method for training a cross-modal retrieval model which is used by a cross-modal retrieval system, wherein a cross-modal retrieval is used for a retrieval of data of one modal using data of another modal, wherein the method comprises:

determining a similarity of a cross-modal sample pair according to the cross-modal sample pair, the cross-modal sample pair comprising a sample of a first modal and a sample of a second modal, and the first modal being different from the second modal, wherein the first modal is a text, and the second modal is a video, and the cross-modal sample pair comprises a positive sample pair and a negative sample pair, the positive sample pair comprises an anchor sample and a positive sample, the negative sample pair comprises the anchor sample and a negative sample, the anchor sample has the first modal, and the positive sample and the negative sample have the second modal, and wherein the anchor sample is a text in a sample set, the positive sample is a video related to the text in the sample set, and the negative sample is a randomly selected video which is related or not related to the text in the sample set;

determining a soft margin based on the similarity, and determining a soft margin loss function based on the soft margin, wherein the soft margin is a non-fixed value; and

determining a total loss function based on the soft margin loss function, and training a cross-modal retrieval model according to the total loss function,

using the trained cross-modal retrieval model to receive a text input by a user, determine a video matched with the text using the cross-modal retrieval model, and feed the matched video back to the user,

wherein the determining a soft margin based on the similarity comprises:

calculating a distance between similarity of the positive sample pair and similarity of the negative sample pair to obtain a similarity distance; and

normalizing the similarity distance to obtain a normalized similarity distance, and determining the normalized similarity distance as the soft margin, and

wherein the similarity distance comprises a similarity distance in the first modal and a similarity distance in the second modal, the similarity distance in the first modal being obtained by processing the sample pair in the first modal using a semantic representation model in the first modal and the similarity distance in the second modal being obtained by processing the sample pair in the second modal using a semantic representation model in the second modal;

the determining a soft margin based on the similarity and determining a soft margin loss function based on the soft margin comprises:

determining a soft margin in the first modal based on the similarity distance in the first modal, and calculating a contrastive loss function in the first modal based on the soft margin in the first modal;

determining a soft margin in the second modal based on the similarity distance in the second modal, and calculating a contrastive loss function in the second modal based on the soft margin in the second modal; and

calculating the soft margin loss function according to the contrastive loss function in the first modal and the contrastive loss function in the second modal.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (29)