Identifying facial expressions in acquired digital images
A face is detected and identified within an acquired digital image. One or more features of the face is/are extracted from the digital image, including two independent eyes or subsets of features of each of the two eyes, or lips or partial lips or one or more other mouth features and one or both eyes, or both. A model including multiple shape parameters is applied to the two independent eyes or subsets of features of each of the two eyes, and/or to the lips or partial lips or one or more other mouth features and one or both eyes. One or more similarities between the one or more features of the face and a library of reference feature sets is/are determined. A probable facial expression is identified based on the determining of the one or more similarities.
1. A method for mirroring a user facial expression in an in-game avatar within a game display during a game, the method comprising:
capturing a user facial expression in a digital image of a user's face as the user participates in the game;
based on content of the image, determining the user facial expression by extracting one or more features of the user's face from the digital image, including one or two eyes or a subset of features of the one or two eyes, or one or two lips by:
(i) extracting the one or more features of the user's face using a first method;
(ii) extracting the one of more features of the user's face using a second method; and
(ii) comparing a first calculated error of the first method and a second calculated error of the second method and using the extracted one or more features produced by the first method if the first calculated error is lower than the second calculated error;
applying a model, which includes multiple shape parameters to determine one or more similarities between the one or more facial features and one or more reference features, using a Nearest Neighbor classification method or a Support Vector Machines classification method;
based on the one or more similarities determined with the model, identifying a probable expression of the user's face; and
causing the in-game avatar to mimic the probable expression of the user's face in the game display.
2. The method of claim 1 , wherein the one or more features of the user's face extracted from the digital image include one or more skin texture appearance changes comprising one or more furrows, bulges, expression wrinkles, or illumination variations or a blushing; or multiple pose variations and multiple directional illumination conditions.
3. The method of claim 1 , further comprising categorizing the user facial expression as indicating one or more of: surprise, fear, happiness, anger, neutral, sadness, disgust.
4. The method of claim 1 , wherein the first method includes a component Active Appearance Model.
5. The method of claim 1 , wherein some of the features are extracted using at least a first eye sub-model and a second eye sub-model.
6. The method of claim 5 , wherein a common eye sub-model is fitted into at least one of the first eye sub-model and the second eye sub-model.
7. The method of claim 1 , wherein the model includes an 3D face model.
8. The method of claim 7 , wherein the in-game avatar includes a 3D face.
9. The method of claim 1 , wherein a position and a shape of each of the two eyes are independently modeled.
10. A non-transitory computer-readable storage medium, storing one or more computer instructions which, when executed by one or more processors, cause the processors to perform:
generating an in-game avatar and displaying the in-game avatar within a game display during a game;
capturing a user facial expression in a digital image of a user's face as the user participates in the game;
based on content of the image, determining the user facial expression by extracting one or more features of the user's face from the digital image, including one or two eyes or a subset of features of the one or two eyes, or one or two lips by:
(i) extracting the one or more features of the user's face using a first method;
(ii) extracting the one of more features of the user's face using a second method; and
(ii) comparing a first calculated error of the first method and a second calculated error of the second method and using the extracted one or more features produced by the first method if the first calculated error is lower than the second calculated error;
applying a model which includes multiple shape parameters to determine one or more similarities between the one or more facial features and one or more reference features, using a Nearest Neighbor classification method or a Support Vector Machines classification method;
based on the one or more similarities determined with the model, identifying a probable expression of the user's face; and
causing the in-game avatar to mimic the probable expression of the user's face in the game display.
11. The non-transitory computer-readable storage medium of claim 10 , wherein the one or more features depicted in the image of the user's face extracted from the digital image include one or more skin texture appearance changes comprising one or more furrows, bulges, expression wrinkles, or illumination variations or a blushing; or multiple pose variations and multiple directional illumination conditions.
12. The non-transitory computer-readable storage medium of claim 10 , further comprising instructions for categorizing the user facial expression as indicating one or more of:
surprise, fear, happiness, anger, neutral, sadness, disgust.
13. An apparatus for mirroring a user facial expression in an in-game avatar, the apparatus comprising:
a user facial expression detection unit configured to:
capture a user facial expression in a digital image of a user's face as the user participates in a game;
based on content of the image, determine a user facial expression by extracting one or more features of the user's face from the digital image, including one or two eyes or a subset of features of the one or two eyes, or one or two lips by:
(i) extracting the one or more features of the user's face using a first method;
(ii) extracting the one of more features of the user's face using a second method; and
(iii) comparing a first calculated error of the first method and a second calculated error of the second method and using the extracted one or more features produced by the first method if the first calculated error is lower than the second calculated error;
apply a model which includes multiple shape parameters to determine one or more similarities between the one or more facial features and one or more reference features, using a Nearest Neighbor classification method or a Support Vector Machines classification method;
based on the one or more similarities determined with the model, identify a probable expression of the user's face; and
cause the in-game avatar to mimic the probable expression of the user's face in a game display.
14. The apparatus for mirroring a user facial expression of claim 13 , wherein the one or more features of the user's face extracted from the digital image include one or more skin texture appearance changes comprising one or more furrows, bulges, expression wrinkles, or illumination variations or a blushing.
15. The apparatus for mirroring a user facial expression of claim 13 , wherein the determining of the user facial expression is based, at least in part, on one or more of:
multiple pose variations, or multiple directional illumination conditions.
16. The apparatus for mirroring a user facial expression of claim 13 , wherein the user facial expression detection unit is further configured to:
determine one or more facial expressions taken from the group consisting of an appearance of surprise, fear, happiness, anger, neutral disposition, sadness, and disgust.
17. The apparatus for mirroring a user facial expression of claim 13 , wherein the user facial expression detection unit is further configured to: categorize the user facial expression as indicating one or more of: surprise, fear, happiness, anger, neutral, sadness, disgust.
18. The apparatus for mirroring a user facial expression of claim 13 , wherein the user facial expression detection unit is further configured to: adapt a difficulty of the game based on the user facial expression.