Method for detecting facial expression and repairing smile face of portrait photo

1. A method for detecting a facial expression and repairing a smile face of a portrait photo, which is applied to an electronic device, comprising:

detecting a location and a range of a mouth region in an inputted portrait photo;

capturing a patch in the mouth region and a predetermined peripheral range thereof, wherein the patch is able to fully show various facial expressions in the mouth region and the predetermined peripheral range;

executing a comparison process to a smile state or a stiff state of the mouth region in the patch by a mouth state classifier, wherein the mouth state classifier is trained according to a predetermined number of mouth photo samples in a smile state or a stiff state, for determining if the mouth region in the patch is in the smile state or the stiff state;

executing a calculation process to a repaired region of the mouth region when the mouth region is determined to be in the stiff state, wherein the calculation process comprises the steps of:

selecting a plurality of outermost feature points of the mouth region to define a to-be-repaired region, and

expanding the to-be-repaired region with a predetermined range to a repaired region which has an expanded range symmetrically extended outward along each perpendicular direction of each sides of the to-be-repaired region;

calculating a location of a plurality of feature points in the repaired region of the mouth region; and

executing an image warping process to the location of the feature points and adjacent pixels thereof, for generating a repaired portrait photo showing a repaired mouth with smile state, wherein the image warping process comprises the steps of:

setting a mouth model of a smile face according to a portrait size and the location of the feature points in the portrait photo, wherein the mouth model is set according to a point based warping principle:

{ L i 1 }={( x i ,y i )|0 <=i<=M− 1} and { L i 0 }={( u i ,v i )|0 <=i<=M− 1},

{L i 1 } represents each of the feature points of an original photo, (x i ,y i ) represents coordinates of each feature point of the original photo on the portrait photo, {L i 0 } represents each of feature points of a warped photo, and (u i ,v i ) represents coordinates of each feature point 71 of the warped photo on the portrait photo;

generating a warping function based on the mouth model of the smile face and according to a mapping relationship, wherein the warping function is generated according to the mapping relationship, and the warping function is:

T

⁡

(

P

)

=

P

+

∑

i

=

0

M

-

1

⁢

w

i

⁡

(

P

)

⁢

(

L

i

0

-

L

i

1

)

∑

i

=

0

M

-

1

⁢

w

i

⁡

(

P

)

,

w

i

=

(

a

+

d

i

)

-

b

wherein, T(P) is the warping function; P is equal to (x,v); (L i 0 −L i 1 ) is the distance between the location of the original feature point and that of the same feature point after being warped; w i is a weighted value of the feature point; d i is the distance between P and {L i 0 }; and a, b are constants; and

executing the image warping process to the mouth region of the to-be-repaired region according to the warping function, so as to form a mouth photo of a smile face in the repaired region.

2. A method for detecting a facial expression and repairing a smile face of a portrait photo, which is applied to an electronic device, comprising:

detecting a location and a range of a mouth region in an inputted portrait photo;

capturing a patch in the mouth region and a predetermined peripheral range thereof, wherein the patch is able to fully show various facial expressions in the mouth region and the predetermined peripheral range;

executing a comparison process to a smile state or a stiff state of the mouth region in the patch by a mouth state classifier; and executing a calculation process to a repaired region of the mouth region when the mouth region is determined to be in the stiff state, wherein the calculation process comprises the steps of:

selecting a plurality of outermost feature points of the mouth region to define a to-be-repaired region, and

expanding the to-be-repaired region with a predetermined range to a repaired region which has an expanded range symmetrically extended outward along each perpendicular direction of each sides of the to-be-repaired region;

calculating a location of a plurality of feature points in the repaired region of the mouth region; and

executing an image warping process to the location of the feature points and adjacent pixels thereof, for generating a repaired portrait photo showing a repaired mouth with smile state, wherein the image warping process comprises the steps of:

setting a mouth model of a smile face according to a portrait size and the location of the feature points in the portrait photo, wherein the mouth model is set according to a point based warping principle:

{ L i 1 }={( x i ,y i )|0 <=i<=M− 1} and { L i 0 }={( u i ,v i )|0 <=i<=M− 1},

{L i 1 } represents each of the feature points of an original photo, (x i ,y i ) represents coordinates of each feature point of the original photo on the portrait photo, {L i 0 } represents each of feature points of a warped photo, and (u i ,v i ) represents coordinates of each feature point 71 of the warped photo on the portrait photo;

generating a warping function based on the mouth model of the smile face and according to a mapping relationship, wherein the warping function is generated according to the mapping relationship, and the warping function is:

T

⁡

(

P

)

=

P

+

∑

i

=

0

M

-

1

⁢

w

i

⁡

(

P

)

⁢

(

L

i

0

-

L

i

1

)

∑

i

=

0

M

-

1

⁢

w

i

⁡

(

P

)

,

w

i

=

(

a

+

d

i

)

-

b

wherein, T(P) is the warping function; P is equal to (x,y); (L i 0 −L i 1 ) is the distance between the location of the original feature point and that of the same feature point after being warped; w i is a weighted value of the feature point; d i is the distance between P and {L i 0 }; and a,b are constants; and

executing the image warping process to the mouth region of the to-be-repaired region according to the warping function, so as to form a mouth photo of a smile face in the repaired region.