Model training method and apparatus, computer device, and storage medium

1 . A video content retrieval method performed by a computer device, the method comprising:

obtaining a query text from a user, wherein the query text is associated with a plurality of candidate videos;

performing feature extraction processing on the query text through a text encoder in a video content retrieval model, to obtain a plurality of text content features at different feature granularities, wherein the plurality of text content features at the different feature granularities include a first text content feature at a coarse granularity and a second text content feature at a fine granularity;

transcribing the plurality of candidate video based on optical character recognition (OCR) and automatic speech recognition (ASR) technology to obtain a plurality of speech features and text features from the plurality of candidate videos;

performing multi-modal feature extraction processing on the plurality of candidate videos and their corresponding plurality of speech features and text features through a video encoder in the video content retrieval model, to obtain a plurality of multi-modal video content features at different feature granularities, wherein the plurality of multi-modal video content features at the different feature granularities include a first multi-modal video content feature at a coarse granularity and a second multi-modal video content feature at a fine granularity;

calculating, based on the text content feature of each feature granularity, a similarity corresponding to the query text and a candidate video at the corresponding feature granularity, wherein the candidate video a first multi-modal video content feature at the coarse granularity and a second multi-modal video content feature at the fine granularity, further including:

performing cross-modal alignment on the coarse granularity between the first text content feature and the first multi-modal video content feature to obtain a similarity between the query text and the candidate video corresponding to the coarse granularity; and

performing cross-modal alignment on the fine granularity between the second text content feature and the second multi-modal video content feature to obtain a similarity between the query text and the candidate video content retrieval result corresponding to the fine granularity; and

determining, based on the similarities between the query text and the plurality of candidate videos at different feature granularities including the coarse granularity and the fine granularity, one of the plurality of candidate videos having a maximum similarity as a video content retrieval result corresponding to the query text.

2 . The content retrieval method according to claim 1 , wherein the calculating, based on the text content feature of each feature granularity, a similarity corresponding to the query and a candidate video at the corresponding feature granularity comprises:

determining a quantified retrieval result feature corresponding to each feature granularity of the candidate video; and

calculating a similarity between the text content feature and the quantified retrieval result feature according to the feature granularity, to obtain the similarity at the corresponding feature granularity.

3 . The method according claim 1 , wherein the video content retrieval model is trained by:

obtaining sample query text and sample video content retrieval result that matches the sample query text, wherein the sample video content retrieval result includes a plurality of sample videos;

performing, through the text encoder and the video encoder in the video content retrieval model, feature extraction processing on the sample query text and the sample video content retrieval result, respectively, to obtain feature information of a plurality of feature granularities, the feature information comprising a query text content feature corresponding to the sample query text and a plurality of multi-modal sample video content features corresponding to the sample video content retrieval result;

performing, through the video content retrieval model, feature quantification processing on the query text content feature and the plurality of multi-modal sample video content features corresponding to the sample video content retrieval result, to obtain quantified feature information of each feature granularity;

calculating, based on the feature information and the quantified feature information, a retrieval semantic loss corresponding to each feature granularity; and

performing model training on the video content retrieval model based on the retrieval semantic loss corresponding to each feature granularity.

4 . The method according to claim 3 , wherein the calculating, based on the feature information and the quantified feature information, a retrieval semantic loss corresponding to each feature granularity comprises:

calculating, based on the feature information and the quantified feature information, a first retrieval semantic loss of each feature granularity in a first semantic retrieval direction, and a second retrieval semantic loss of each feature granularity in a second semantic retrieval direction;

performing loss aggregation processing on the first retrieval semantic loss and the second retrieval semantic loss; and

determining, according to a processing result of the loss aggregation processing, the retrieval semantic loss corresponding to each feature granularity.

5 . The method according to claim 3 , wherein the content retrieval model comprises a feature quantification module corresponding to each feature granularity, and each feature quantification module is configured to perform feature quantification processing on feature information of the feature granularity corresponding to each feature quantification module.

6 . The method according to claim 3 , wherein content retrieval model comprises a first coarse-grained feature extraction module for the sample video content retrieval result; and

the performing, through a video content retrieval model, feature extraction processing on the sample video content retrieval result comprises:

obtaining a modal content feature corresponding to at least one content mode of the sample video content retrieval result;

separately performing, through the first coarse-grained feature extraction module, feature encoding processing on the modal content feature based on a self-attention mechanism, to obtain an encoded feature corresponding to each content mode; and

performing feature aggregation processing on the encoded feature corresponding to each content mode, to obtain a coarse-grained video content retrieval result content feature of the sample video content retrieval result.

7 . The method according to claim 3 , wherein the content retrieval model comprises a fine-grained feature extraction module shared by the sample query text and the sample video content retrieval result; and

the performing, through a video content retrieval model, feature extraction processing on the sample query text and the sample video content retrieval result, to obtain feature information of a plurality of feature granularities comprises:

obtaining a plurality of content features of the sample query text and the sample video content retrieval result;

performing feature clustering processing on the content features through the fine-grained feature extraction module; and

determining feature information of a plurality of fine granularities based on a clustering result of the feature clustering processing, wherein the feature information comprises the query text content feature and the video content retrieval result content feature.

8 . The method according to claim 3 , wherein the performing model training on the video content retrieval model based on the retrieval semantic loss corresponding to each feature granularity comprises:

performing, according to a granularity type of the feature granularity, loss aggregation processing on the retrieval semantic loss corresponding to each feature granularity; and

performing model training on the content retrieval model based on an aggregated retrieval semantic loss.

9 . A computer device, comprising a memory and a processor, the memory storing a plurality of instructions, and the processor being configured to execute the plurality of instructions in the memory and cause the computer device to perform a video content retrieval method including:

obtaining a query text from a user, wherein the query text is associated with a plurality of candidate videos;

performing feature extraction processing on the query text through a text encoder in a video content retrieval model, to obtain a plurality of text content features at different feature granularities, wherein the plurality of text content features at the different feature granularities include a first text content feature at a coarse granularity and a second text content feature at a fine granularity;

transcribing the plurality of candidate video based on optical character recognition (OCR) and automatic speech recognition (ASR) technology to obtain a plurality of speech features and text features from the plurality of candidate videos;

performing multi-modal feature extraction processing on the plurality of candidate videos and their corresponding plurality of speech features and text features through a video encoder in the video content retrieval model, to obtain a plurality of multi-modal video content features at different feature granularities, wherein the plurality of multi-modal video content features at the different feature granularities include a first multi-modal video content feature at a coarse granularity and a second multi-modal video content feature at a fine granularity;

calculating, based on the text content feature of each feature granularity, a similarity corresponding to the query text and a candidate video at the corresponding feature granularity, wherein the candidate video a first multi-modal video content feature at the coarse granularity and a second multi-modal video content feature at the fine granularity, further including:

performing cross-modal alignment on the coarse granularity between the first text content feature and the first multi-modal video content feature to obtain a similarity between the query text and the candidate video corresponding to the coarse granularity; and

performing cross-modal alignment on the fine granularity between the second text content feature and the second multi-modal video content feature to obtain a similarity between the query text and the candidate video content retrieval result corresponding to the fine granularity; and

determining, based on the similarities between the query text and the plurality of candidate videos at different feature granularities including the coarse granularity and the fine granularity, one of the plurality of candidate videos having a maximum similarity as a video content retrieval result corresponding to the query text.

10 . The computer device according to claim 9 , wherein the calculating, based on the text content feature of each feature granularity, a similarity corresponding to the query and a candidate video at the corresponding feature granularity comprises:

determining a quantified retrieval result feature corresponding to each feature granularity of the candidate video; and

calculating a similarity between the text content feature and the quantified retrieval result feature according to the feature granularity, to obtain the similarity at the corresponding feature granularity.

11 . The computer device according to claim 9 , wherein the video content retrieval model is trained by:

obtaining sample query text and sample video content retrieval result that matches the sample query text, wherein the sample video content retrieval result includes a plurality of sample videos;

performing, through the text encoder and the video encoder in the video content retrieval model, feature extraction processing on the sample query text and the sample video content retrieval result, respectively, to obtain feature information of a plurality of feature granularities, the feature information comprising a query text content feature corresponding to the sample query text and a plurality of multi-modal sample video content features corresponding to the sample video content retrieval result;

performing, through the video content retrieval model, feature quantification processing on the query text content feature and the plurality of multi-modal sample video content features corresponding to the sample video content retrieval result, to obtain quantified feature information of each feature granularity;

calculating, based on the feature information and the quantified feature information, a retrieval semantic loss corresponding to each feature granularity; and

performing model training on the video content retrieval model based on the retrieval semantic loss corresponding to each feature granularity.

12 . The computer device according to claim 11 , wherein the calculating, based on the feature information and the quantified feature information, a retrieval semantic loss corresponding to each feature granularity comprises:

calculating, based on the feature information and the quantified feature information, a first retrieval semantic loss of each feature granularity in a first semantic retrieval direction, and a second retrieval semantic loss of each feature granularity in a second semantic retrieval direction;

performing loss aggregation processing on the first retrieval semantic loss and the second retrieval semantic loss; and

determining, according to a processing result of the loss aggregation processing, the retrieval semantic loss corresponding to each feature granularity.

13 . The computer device according to claim 11 , wherein the content retrieval model comprises a feature quantification module corresponding to each feature granularity, and each feature quantification module is configured to perform feature quantification processing on feature information of the feature granularity corresponding to each feature quantification module.

14 . The computer device according to claim 11 , wherein content retrieval model comprises a first coarse-grained feature extraction module for the sample video content retrieval result; and

the performing, through a video content retrieval model, feature extraction processing on the sample video content retrieval result comprises:

obtaining a modal content feature corresponding to at least one content mode of the sample video content retrieval result;

separately performing, through the first coarse-grained feature extraction module, feature encoding processing on the modal content feature based on a self-attention mechanism, to obtain an encoded feature corresponding to each content mode; and

performing feature aggregation processing on the encoded feature corresponding to each content mode, to obtain a coarse-grained video content retrieval result content feature of the sample video content retrieval result.

15 . The computer device according to claim 11 , wherein the content retrieval model comprises a fine-grained feature extraction module shared by the sample query text and the sample video content retrieval result; and

the performing, through a video content retrieval model, feature extraction processing on the sample query text and the sample video content retrieval result, to obtain feature information of a plurality of feature granularities comprises:

obtaining a plurality of content features of the sample query text and the sample video content retrieval result;

performing feature clustering processing on the content features through the fine-grained feature extraction module; and

determining feature information of a plurality of fine granularities based on a clustering result of the feature clustering processing, wherein the feature information comprises the query text content feature and the video content retrieval result content feature.

16 . The computer device according to claim 11 , wherein the performing model training on the video content retrieval model based on the retrieval semantic loss corresponding to each feature granularity comprises:

performing, according to a granularity type of the feature granularity, loss aggregation processing on the retrieval semantic loss corresponding to each feature granularity; and

performing model training on the content retrieval model based on an aggregated retrieval semantic loss.

17 . A non-transitory computer-readable storage medium, storing a plurality of instructions, and the instructions being configured to be loaded by a processor of a computer device, and causing the computer device to perform a video content retrieval method including:

obtaining a query text from a user, wherein the query text is associated with a plurality of candidate videos;

performing feature extraction processing on the query text through a text encoder in a video content retrieval model, to obtain a plurality of text content features at different feature granularities, wherein the plurality of text content features at the different feature granularities include a first text content feature at a coarse granularity and a second text content feature at a fine granularity;

transcribing the plurality of candidate video based on optical character recognition (OCR) and automatic speech recognition (ASR) technology to obtain a plurality of speech features and text features from the plurality of candidate videos;

performing multi-modal feature extraction processing on the plurality of candidate videos and their corresponding plurality of speech features and text features through a video encoder in the video content retrieval model, to obtain a plurality of multi-modal video content features at different feature granularities, wherein the plurality of multi-modal video content features at the different feature granularities include a first multi-modal video content feature at a coarse granularity and a second multi-modal video content feature at a fine granularity;

calculating, based on the text content feature of each feature granularity, a similarity corresponding to the query text and a candidate video at the corresponding feature granularity, wherein the candidate video a first multi-modal video content feature at the coarse granularity and a second multi-modal video content feature at the fine granularity, further including:

performing cross-modal alignment on the coarse granularity between the first text content feature and the first multi-modal video content feature to obtain a similarity between the query text and the candidate video corresponding to the coarse granularity; and

performing cross-modal alignment on the fine granularity between the second text content feature and the second multi-modal video content feature to obtain a similarity between the query text and the candidate video content retrieval result corresponding to the fine granularity; and

determining, based on the similarities between the query text and the plurality of candidate videos at different feature granularities including the coarse granularity and the fine granularity, one of the plurality of candidate videos having a maximum similarity as a video content retrieval result corresponding to the query text.

18 . The non-transitory computer-readable storage medium according to claim 17 , wherein the calculating, based on the text content feature of each feature granularity, a similarity corresponding to the query and a candidate video at the corresponding feature granularity comprises:

determining a quantified retrieval result feature corresponding to each feature granularity of the candidate video; and

calculating a similarity between the text content feature and the quantified retrieval result feature according to the feature granularity, to obtain the similarity at the corresponding feature granularity.

19 . The non-transitory computer-readable storage medium according to claim 17 , wherein the video content retrieval model is trained by:

obtaining sample query text and sample video content retrieval result that matches the sample query text, wherein the sample video content retrieval result includes a plurality of sample videos;

performing, through the text encoder and the video encoder in the video content retrieval model, feature extraction processing on the sample query text and the sample video content retrieval result, respectively, to obtain feature information of a plurality of feature granularities, the feature information comprising a query text content feature corresponding to the sample query text and a plurality of multi-modal sample video content features corresponding to the sample video content retrieval result;

performing, through the video content retrieval model, feature quantification processing on the query text content feature and the plurality of multi-modal sample video content features corresponding to the sample video content retrieval result, to obtain quantified feature information of each feature granularity;

calculating, based on the feature information and the quantified feature information, a retrieval semantic loss corresponding to each feature granularity; and

performing model training on the video content retrieval model based on the retrieval semantic loss corresponding to each feature granularity.

20 . The non-transitory computer-readable storage medium according to claim 19 , wherein the performing model training on the video content retrieval model based on the retrieval semantic loss corresponding to each feature granularity comprises:

performing, according to a granularity type of the feature granularity, loss aggregation processing on the retrieval semantic loss corresponding to each feature granularity; and

performing model training on the content retrieval model based on an aggregated retrieval semantic loss.