Granted Patent B2

US 12,738,095 · App. 18/563,992 · Granted Sep 15, 2026

Inventors: Wei Zhang (Hangzhou, CN); Feng Qiu (Hangzhou, CN); Yu Ding (Hangzhou, CN); Hao Zeng (Hangzhou, CN); Suzhen Wang (Hangzhou, CN); Zhimeng Zhang (Hangzhou, CN); Bowen Ma (Hangzhou, CN); Rudong An (Hangzhou, CN); Tangjie Lv (Hangzhou, CN); Changjie Fan (Hangzhou, CN); Zhipeng Hu (Hangzhou, CN)

Assignee: NETEASE (HANGZHOU) NETWORK CO., LTD.

G06V40/174G06V10/809G06V10/82G06V2201/10

Abstract

Claims (66)

1 . An object emotion analysis method, comprising:

acquiring multimedia data associated with a target object, and extracting a static facial feature and a dynamic feature of the target object from the multimedia data, wherein the dynamic feature comprises at least one of an expression change feature, a sound feature or a language content feature of the target object; and

inputting the static facial feature and the dynamic feature into an object emotion analysis model, wherein the object emotion analysis model is pre-trained, fusing the static facial feature and the dynamic feature by the object emotion analysis model to obtain a fusion feature, and outputting an emotion analysis result of the target object based on the fusion feature, wherein the object emotion analysis model comprises a dynamic feature processing module, and the dynamic feature processing module is configured to input the dynamic feature of a corresponding type; and wherein fusing the static facial feature and the dynamic feature by the object emotion analysis model to obtain the fusion feature comprises:

inputting, for the dynamic feature processing module, the dynamic feature corresponding to the dynamic feature processing module and the static facial feature of the target object into the dynamic feature processing module; and

fusing the static facial feature and the dynamic feature by an attention network in the dynamic feature processing module to obtain the fusion feature, wherein the fusion feature comprises a common key feature for characterizing an emotion, between the static facial feature and the dynamic feature,

wherein the method further comprises:

inputting the dynamic feature into a time sequence model, and outputting a context feature of the dynamic feature, wherein the context feature is configured for characterizing a variation trend of the dynamic feature with time; and

adding time sequence information into the context feature, and determining the context feature in which the time sequence information is added, as the dynamic feature to be input into the dynamic feature processing module, wherein the time sequence information is configured for indicating time sequence positions of features in the context feature.

2 . The method according to claim 1 , wherein extracting the dynamic feature of the target object from the multimedia data comprises:

acquiring a specified video frame sequence from video data in the multimedia data in response to determining that the dynamic feature comprises the expression change feature, wherein an expression of the target object dynamically changes in the video frame sequence; and

extracting the expression change feature of the target object from the video frame sequence by a pre-trained expression feature extraction model.

3 . The method according to claim 1 , wherein extracting the dynamic feature of the target object from the multimedia data comprises:

acquiring a specified audio sequence from audio data in the multimedia data in response to determining that the dynamic feature comprises the sound feature, wherein the audio sequence comprises a sound signal emitted by the target object; and

extracting the sound feature of the target object from the audio sequence by a pre-trained sound feature extraction model, wherein the sound feature comprises at least one of a frequency feature, a tone feature, a pitch feature, or an energy feature.

4 . The method according to claim 1 , wherein extracting the dynamic feature of the target object from the multimedia data comprises:

acquiring a language content text of the target object from subtitle data or audio data in the multimedia data in response to determining that the dynamic feature comprises the language content feature; and

extracting a language content feature of the target object from the language content text by a pre-trained language content feature extraction model, wherein the language content feature is configured to characterize a linguistic meaning of a language uttered by the target object.

5 . The method according to claim 1 , wherein the attention network in the dynamic feature processing module comprises a self-attention network and a cross-attention network; and

wherein fusing the static facial feature and the dynamic feature by the attention network in the dynamic feature processing module to obtain the fusion feature comprises:

transforming the dynamic feature to obtain a first input parameter of the self-attention network, inputting the first input parameter into the self-attention network, and outputting an intermediate feature of the dynamic feature, wherein the intermediate feature is configured for characterizing autocorrelation of the dynamic feature; and

determining a second input parameter of the cross-attention network based on the intermediate feature, determining a third input parameter of the cross-attention network based on the static facial feature, inputting the second input parameter and the third input parameter to the cross-attention network to obtain an output result, and determining the fusion feature based on the output result.

6 . The method according to claim 5 , wherein determining the second input parameter of the cross-attention network based on the intermediate feature comprises:

performing first fusion processing on the intermediate feature and the dynamic feature to obtain a first fusion result; and transforming the first fusion result to obtain the second input parameter of the cross-attention network.

7 . The method according to claim 6 , wherein performing first fusion processing on the intermediate feature and the dynamic feature to obtain the first fusion result comprises:

performing feature addition on the intermediate feature and the dynamic feature to obtain an addition result; and

performing normalization processing on the addition result to obtain the first fusion result.

8 . The method according to claim 7 , wherein the feature addition comprises: feature splicing, or addition of feature data located at same position points.

9 . The method according to claim 5 , wherein determining the third input parameter of the cross-attention network based on the static facial feature comprises:

transforming the static facial feature to obtain the third input parameter of the cross-attention network.

10 . The method according to claim 5 , wherein determining the fusion feature based on the output result comprises:

performing second fusion processing on the output result and the first fusion result corresponding to the second input parameter to obtain a second fusion result, wherein the second input parameter is obtained by transforming the first fusion result;

inputting the second fusion result into a preset first multilayer perceptron, and performing mapping on the second fusion result by the first multilayer perceptron to obtain a mapping result; and

performing third fusion processing on the mapping result and the second fusion result to obtain the fusion feature.

11 . The method according to claim 1 , wherein the object emotion analysis model comprises a plurality of dynamic feature processing modules, and each dynamic feature processing module is configured to input the dynamic feature of a corresponding type and output a fusion feature corresponding to the dynamic feature; and

wherein outputting the emotion analysis result of the target object based on the fusion feature comprises:

splicing the fusion features output by the dynamic feature processing module and a static face feature to obtain spliced features, and inputting the spliced features into a preset second multilayer perceptron to output the emotion analysis result of the target object.

12 . The method according to claim 11 , wherein the second multilayer perceptron comprises a plurality of branch networks; and

wherein inputting the spliced features into the preset second multilayer perceptron to output the emotion analysis result of the target object comprises:

inputting the spliced features into the plurality of branch networks of the second multilayer perceptron respectively, wherein each of the branch networks are preset with a feature mapping mode corresponding to the branch network, and the feature mapping mode comprises: performing linear combination mapping based on a preset facial action unit, performing linear combination mapping based on a plurality of preset basic emotion types, and performing linear characterization mapping based on a positive-negative degree and an intense degree of an emotion; and

performing mapping on the spliced features by the branch networks according to the feature mapping modes corresponding to the branch networks, so as to obtain the emotion analysis results output by the branch networks.

13 . The method according to claim 1 , wherein the multimedia data is video data with video pictures containing the target object; and wherein extracting the static facial feature and the dynamic feature of the target object from the multimedia data comprises:

determining each video frame one by one in the video data as a current video frame;

acquiring the static facial feature from the current video frame;

acquiring the expression change feature from a video frame sequence containing the current video frame, acquiring the sound feature from audio data corresponding to the video frame sequence, and acquiring the language content feature from audio data or subtitle data corresponding to the video frame sequence.

14 . The method according to claim 13 , further comprising:

acquiring an emotion analysis result corresponding to the current video frame in the video data;

determining that there is an emotion sudden-change video frame in the video data, based on the emotion analysis result corresponding to the current video frame, wherein an emotion analysis result of the emotion sudden-change video frame is a first analysis result, emotion analysis results of video frames adjacent to, before and after the emotion sudden-change video frame are second analysis results, and the first analysis result is different from the second analysis results; and

smoothing the emotion analysis result of the emotion sudden-change video frame based on the second analysis results to obtain a smoothed emotion analysis result.

15 . The method according to claim 1 , wherein the object emotion analysis model is restricted by using a plurality of loss functions during training.

16 . The method according to claim 1 , wherein the static facial feature is configured to characterize an appearance feature, action and posture features and an expression feature of the target object.

17 . An electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions executable by the processor, and the processor is configured to execute the machine executable instructions to implement an object emotion analysis method, the object emotion analysis method comprising:

acquiring multimedia data associated with a target object, and extracting a static facial feature and a dynamic feature of the target object from the multimedia data, wherein the dynamic feature comprises at least one of an expression change feature, a sound feature or a language content feature of the target object; and

inputting the static facial feature and the dynamic feature into an object emotion analysis model, wherein the object emotion analysis model is pre-trained, fusing the static facial feature and the dynamic feature by the object emotion analysis model to obtain a fusion feature, and outputting an emotion analysis result of the target object based on the fusion feature, wherein the object emotion analysis model comprises a dynamic feature processing module, and the dynamic feature processing module is configured to input the dynamic feature of a corresponding type; and

wherein fusing the static facial feature and the dynamic feature by the object emotion analysis model to obtain the fusion feature comprises:

inputting, for the dynamic feature processing module, the dynamic feature corresponding to the dynamic feature processing module and the static facial feature of the target object into the dynamic feature processing module; and

fusing the static facial feature and the dynamic feature by an attention network in the dynamic feature processing module to obtain the fusion feature, wherein the fusion feature comprises a common key feature for characterizing an emotion, between the static facial feature and the dynamic feature, wherein the method further comprises:

inputting the dynamic feature into a time sequence model, and outputting a context feature of the dynamic feature, wherein the context feature is configured for characterizing a variation trend of the dynamic feature with time; and

adding time sequence information into the context feature, and determining the context feature in which the time sequence information is added, as the dynamic feature to be input into the dynamic feature processing module, wherein the time sequence information is configured for indicating time sequence positions of features in the context feature.

18 . A non-transitory machine-readable storage medium, wherein the non-transitory machine-readable storage medium stores machine executable instructions, and when invoked and executed by a processor, the machine executable instructions cause the processor to implement an object emotion analysis method, the object emotion analysis method comprising:

acquiring multimedia data associated with a target object, and extracting a static facial feature and a dynamic feature of the target object from the multimedia data, wherein the dynamic feature comprises at least one of an expression change feature, a sound feature or a language content feature of the target object; and

inputting the static facial feature and the dynamic feature into an object emotion analysis model, wherein the object emotion analysis model is pre-trained, fusing the static facial feature and the dynamic feature by the object emotion analysis model to obtain a fusion feature, and outputting an emotion analysis result of the target object based on the fusion feature, wherein the object emotion analysis model comprises a dynamic feature processing module, and the dynamic feature processing module is configured to input the dynamic feature of a corresponding type; and

wherein fusing the static facial feature and the dynamic feature by the object emotion analysis model to obtain the fusion feature comprises:

inputting, for the dynamic feature processing module, the dynamic feature corresponding to the dynamic feature processing module and the static facial feature of the target object into the dynamic feature processing module; and

fusing the static facial feature and the dynamic feature by an attention network in the dynamic feature processing module to obtain the fusion feature, wherein the fusion feature comprises a common key feature for characterizing an emotion, between the static facial feature and the dynamic feature, wherein the method further comprises:

inputting the dynamic feature into a time sequence model, and outputting a context feature of the dynamic feature, wherein the context feature is configured for characterizing a variation trend of the dynamic feature with time; and

adding time sequence information into the context feature, and determining the context feature in which the time sequence information is added, as the dynamic feature to be input into the dynamic feature processing module, wherein the time sequence information is configured for indicating time sequence positions of features in the context feature.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (20)