Joint expression coding system and method based on static and dynamic expression images

1 . A joint expression coding method based on static and dynamic expression images, comprising following specific steps S1 to S4:

step S1: collecting expression video images of a person by a camera, performing image preprocessing on the expression video images to obtain a preprocessed expression video, the image preprocessing comprising downsampling, image cropping, and image alignment;

step S2: generating a dynamic expression image according to the preprocessed expression video, wherein calculating and generating the dynamic expression image according to two adjacent frames of images comprises following specific sub-steps S21 to S22:

step S21: calculating a motion vector based on following formula:

u

=

arg

min

u

′

∑

x

[

I

t

+

1

(

x

+

u

′

)

-

P

⁡

(

x

)

]

2

wherein P(x) is a regional block of an image I t at time t, and a sum of squares of a difference between P(x) and a corresponding regional block in an image I t+1 at time t+1 is minimized at a center point x=(x,y) to obtain a motion vector u=(u,v), u, v representing a motion direction and a motion velocity, namely an optical flow estimation of the regional block between two images, and u′ being a first derivative of the motion vector;

step S22: calculating the dynamic expression image based on following formula:

I

d

=

1

∑

i

⁢

Z

i

⁢

∑

N

s

i

(

Z

i

·

u

i

)

wherein I d is an optical flow field between two adjacent frames of images I t+1 and I t at time t and t+1, namely the dynamic expression image, N s is a number of regional blocks in an image, and u i is an optical flow estimation of a motion vector of a corresponding i th regional block in two adjacent frames of images;

Z

i

=

λ

i

/

max

⁡

(

1

,



d

i

(

x

)



2

)

d

i

(

x

)

=

I

t

+

1

(

x

+

u

i

)

-

P

⁡

(

x

)

wherein λ i is an indicator variable, which equals 1 only when corresponding regional blocks in two adjacent images overlap;

step S3: generating a dynamic weight image according to the dynamic expression image based on following specific sub-steps S31 to S32:

step S31: normalizing the dynamic expression image based on following formula:

I

d

′

=

I

d

255

;

and

step S32: calculating and generating the dynamic weight image I d″ according to the normalized dynamic expression image I d′ based on following formula:

I

d

″

=

w

⁡

(

x

)

=

1

1

+

e

-

x

*

I

d

′

;

step S4: generating a joint expression coding image according to the dynamic weight image and a static image, and calculating each pixel value of the static image I s according to the dynamic weight image I d″ to obtain the joint expression coding image based on following formula:

I

c

=

I

d

″

⊗

I

s

wherein I c is the joint expression coding image, and ⊗ represents element multiplication;

and performing an emotion recognition on the person according to the joint expression coding image.

2 . A joint expression coding system adopting the joint expression coding method based on static and dynamic expression images according to claim 1 , comprising an image preprocessing module, a dynamic expression image generation module, a dynamic weight image generation module, and a joint expression coding image generation module, wherein

the image preprocessing module is connected to an expression video input end and is configured to perform image preprocessing on input expression video images of a person collected by a camera to obtain a preprocessed image;

the dynamic expression image generation module is connected to the image preprocessing module and is configured to generate a dynamic expression image from the preprocessed image;

the dynamic weight image generation module is connected to the dynamic expression image generation module and is configured to generate a dynamic weight image according to the dynamic expression image; and

the joint expression coding image generation module is connected to the dynamic weight image generation module and is configured to generate a joint expression coding image according to the dynamic weight image and a static image;

and an emotion recognition is performed on the person according to the joint expression coding image.