Content recognition method and apparatus, computer device, and storage medium

1 . A method for content recognition, comprising:

acquiring, from a content for recognition, a text piece and a media piece associated with the text piece, the media piece including image data or audio data of the content;

performing a first feature extraction on the text piece to obtain text features;

performing a second feature extraction on the media piece associated with the text piece to obtain media features;

determining feature association measures between the media features and the text features, a feature association measure for a first feature in the media features and a second feature in the text features indicating an association degree between the first feature and the second feature;

obtaining feature attention intensities corresponding to the text features based on the feature association measures, each value of the feature association measures being positively correlated with a respective value of the feature attention intensities, and the feature attention intensities including image attention intensities based on the image data or audio attention intensities based on the audio data;

adjusting the text features based on the feature association measures to obtain adjusted text features that include first adjusted text features obtained by adjusting the text features based on the image attention intensities or second adjusted text features obtained by adjusting the text features based on the audio attention intensities; and

performing a recognition based on the adjusted text features to obtain a content recognition result of the content.

2 . The method according to claim 1 , wherein the performing the recognition comprises:

fusing the adjusted text features and the text features to obtain fused text features; and

performing the recognition based on the fused text features to obtain the content recognition result of the content.

3 . The method according to claim 2 , wherein the fusing the adjusted text features and the text features comprises:

encoding the text features to obtain first encoded features;

encoding the adjusted text features to obtain second encoded features;

fusing the first encoded features and the second encoded features to obtain fused encoded features;

obtaining adjusted feature weights respectively associated with the adjusted text features based on the fused encoded features; and

fusing the adjusted text features and the text features based on the adjusted feature weights to obtain the fused text features.

4 . The method according to claim 3 , wherein

the first encoded features are obtained by encoding through a first encoder in a neural network with a trained content recognition model,

the second encoded features are obtained by encoding through a second encoder in the neural network with the trained content recognition model, and

the obtaining the adjusted feature weights respectively associated with the adjusted text features based on the fused encoded features comprises:

inputting the fused encoded features into an activation layer in the neural network that performs an activation processing to obtain activation values; and

providing the activation values as the adjusted feature weights respectively associated with the adjusted text features, the activation layer being a shared activation layer of the first encoder and the second encoder.

5 . The method according to claim 3 , wherein the fusing the adjusted text features and the text features comprises:

obtaining text feature weights respectively associated with the text features based on the adjusted feature weights;

performing product operations on the adjusted text features with the adjusted feature weights that are respectively associated with the adjusted text features to weight the adjusted text features and obtain weighted adjusted text features;

performing product operations on the text features with the text feature weights that are respectively associated with the text features to weight the text features and obtain weighted text features; and

adding the weighted adjusted text features and the weighted text features to obtain the fused text features.

6 . The method according to claim 1 , wherein

the content is a video, and

the acquiring, from the content, the text piece and the media piece associated with the text piece comprises:

acquiring the text piece corresponding to a target time in the video; and

acquiring the image data or the audio data corresponding to the target time in the video.

7 . The method according to claim 6 , wherein

the image data corresponds to a video frame,

the adjusted text features include the first adjusted text features obtained by a first adjustment according to the image attention intensities of the video frame, and

the performing the recognition includes:

fusing the first adjusted text features and the text features to obtain fused text features; and

performing the recognition based on the fused text features to obtain the content recognition result of the content.

8 . The method according to claim 7 , wherein

the audio data corresponds to an audio frame,

the adjusted text features include the second adjusted text features obtained by a second adjustment according to the audio attention intensities of the audio frame, and

the performing the recognition includes:

fusing the first adjusted text features, the second adjusted text features, and the text features to obtain the fused text features; and

performing the recognition based on the fused text features to obtain the content recognition result of the content.

9 . The method according to claim 1 , wherein the adjusting the text features comprises:

multiplying the feature attention intensities respectively with corresponding feature values of the text features to obtain feature value products;

arranging the feature value products into a sequence of the feature value products according to positions of the corresponding feature values in the text features; and

providing the sequence of the feature value products as the adjusted text features.

10 . The method according to claim 1 , wherein

the text features correspond to segmented words in the text piece,

the adjusted text features form a feature sequence according to an order of the segmented words in the text piece, and

the performing the recognition comprises:

obtaining, based on the feature sequence, corresponding positional relationships of the segmented words;

acquiring a named entity from the text piece based on the positional relationships; and

providing the named entity as the content recognition result of the content.

11 . The method according to claim 10 , wherein the acquiring the named entity comprises:

acquiring a first segmented word, which has a first corresponding positional relationship indicative of a beginning position, to be a beginning word of the named entity;

extracting one or more second segmented words following the beginning word to be one or more constituent words, the one or more second segmented words having a second corresponding positional relationship indicative of inside positions; and

combining the beginning word and the one or more constituent words to obtain the named entity.

12 . The method according to claim 11 , wherein

the obtaining the corresponding positional relationships of the segmented words comprises:

obtaining, based on the feature sequence, the corresponding positional relationships of the segmented words and entity types respectively associated with the segmented words, and

the extracting the one or more second segmented words comprises:

extracting the one or more second segmented words following the beginning word to be one or more constituent words, the one or more second segmented words having the second corresponding positional relationship indicative of the inside positions, and having a same entity type as the beginning word.

13 . The method according to claim 1 , wherein the determining the feature association measures between the media features and the text features comprises:

performing respective product operations on media feature values in the media features and text feature values in the text features at corresponding positions to obtain product operation values; and

collecting statistics on the product operation values to obtain the feature association measures between the media features and the text features.

14 . An apparatus for content recognition, comprising:

processing circuitry configured to:

acquire, from a content for recognition, a text piece and a media piece associated with the text piece, the media piece including image data or audio data of the content;

perform a first feature extraction on the text piece to obtain text features;

perform a second feature extraction on the media piece associated with the text piece to obtain media features;

determine feature association measures between the media features and the text features, a feature association measure for a first feature in the media features and a second feature in the text features indicating an association degree between the first feature and the second feature;

obtain feature attention intensities corresponding to the text features based on the feature association measures, each value of the feature association measures being positively correlated with a respective value of the feature attention intensities, and the feature attention intensities including image attention intensities based on the image data or audio attention intensities based on the audio data;

adjust the text features based on the feature association measures to obtain adjusted text features that include first adjusted text features obtained by adjusting the text features based on the image attention intensities or second adjusted text features obtained by adjusting the text features based on the audio attention intensities; and

perform a recognition based on the adjusted text features to obtain a content recognition result of the content.

15 . The apparatus according to claim 14 , wherein the processing circuitry is configured to:

fuse the adjusted text features and the text features to obtain fused text features; and

perform the recognition based on the fused text features to obtain the content recognition result of the content.

16 . The apparatus according to claim 15 , wherein the processing circuitry is configured to:

encode the text features to obtain first encoded features;

encode the adjusted text features to obtain second encoded features;

fuse the first encoded features and the second encoded features to obtain fused encoded features;

obtain adjusted feature weights respectively associated with the adjusted text features based on the fused encoded features; and

fuse the adjusted text features and the text features based on the adjusted feature weights to obtain the fused text features.

17 . The apparatus according to claim 16 , wherein

the first encoded features are obtained by encoding through a first encoder in a neural network with a trained content recognition model,

the second encoded features are obtained by encoding through a second encoder in the neural network with the trained content recognition model, and

the processing circuitry is configured to:

input the fused encoded features into an activation layer in the neural network for an activation processing to obtain activation values; and

provide the activation values as the adjusted feature weights respectively associated with the adjusted text features, the activation layer being a shared activation layer of the first encoder and the second encoder.

18 . The apparatus according to claim 16 , wherein the processing circuitry is configured to:

obtain text feature weights respectively associated with the text features based on the adjusted feature weights;

perform product operations on the adjusted text features with the adjusted feature weights that are respectively associated with the adjusted text features to weight the adjusted text features and obtain weighted adjusted text features;

perform product operations on the text features with the text feature weights that are respectively associated with the text features to weight the text features and obtain weighted text features; and

add the weighted adjusted text features and the weighted text features to obtain the fused text features.

19 . A non-transitory computer-readable storage medium storing instructions which, when executed by at least one processor, cause the at least one processor to perform a method for content recognition comprising:

acquiring, from a content for recognition, a text piece and a media piece associated with the text piece, the media piece including image data or audio data of the content;

performing a first feature extraction on the text piece to obtain text features;

performing a second feature extraction on the media piece associated with the text piece to obtain media features;

determining feature association measures between the media features and the text features, a feature association measure for a first feature in the media features and a second feature in the text features indicating an association degree between the first feature and the second feature;

obtaining feature attention intensities corresponding to the text features based on the feature association measures, each value of the feature association measures being positively correlated with a respective value of the feature attention intensities, and the feature attention intensities including image attention intensities based on the image data or audio attention intensities based on the audio data;

adjusting the text features based on the feature association measures to obtain adjusted text features that include first adjusted text features obtained by adjusting the text features based on the image attention intensities or second adjusted text features obtained by adjusting the text features based on the audio attention intensities; and

performing a recognition based on the adjusted text features to obtain a content recognition result of the content.

20 . The apparatus according to claim 14 , wherein the processing circuitry is configured to:

multiply the feature attention intensities respectively with corresponding feature values of the text features to obtain feature value products;

arrange the feature value products into a sequence of the feature value products according to positions of the corresponding feature values in the text features; and

provide the sequence of the feature value products as the adjusted text features.