Hybrid deep learning method for recognizing facial expressions
A computer implemented method for recognizing facial expressions by applying feature learning and feature engineering to face images. The method includes conducting feature learning on a face image comprising feeding the face image into a first convolution neural network to obtain a first decision, conducting feature engineering on a face image, comprising the steps of automatically detecting facial landmarks in the face image, transforming the facial features into a two-dimensional matrix, and feeding the two-dimensional matrix into a second convolution neural network to obtain a second decision, computing a hybrid decision based on the first decision and the second decision, and recognizing a facial expression in the face image in accordance to the hybrid decision.
1. A computer-implemented method for recognizing facial expressions, comprising:
feeding a face image into a first convolution neural network to obtain a first decision by one or more computer processors;
automatically detecting facial landmarks in the face image;
describing each of the facial landmarks by a set of facial features;
transforming the facial features that describe the facial landmarks into a two-dimensional matrix comprising coordinates of the facial features by the one or more computer processors;
feeding the two-dimensional matrix into a second convolution neural network to obtain a second decision;
computing a hybrid decision based on the first decision and the second decision; and
recognizing a facial expression in the face image in accordance to the hybrid decision.
2. The computer-implemented method of claim 1 , further comprising: multiplying the first decision by a first weight to produce a first weighted decision; and multiplying the second decision by a second weight to produce a second weighted decision, wherein the hybrid decision is computed based on the first weighted decision and the second weighted decision.
3. The computer-implemented method of claim 2 , further comprising:
updating the first weight and the second weight in the hybrid decision by backpropagation.
4. The computer-implemented method of claim 1 , wherein the hybrid decision is an average, a sum, or a root-mean square function of the first decision and the second decision.
5. The computer-implemented method of claim 1 , wherein the two-dimensional matrix comprises multiple pairs of coordinates each associated with one of the facial features.
6. The computer-implemented method of claim 1 , further comprising:
automatically detecting a face in a digital image by the one or more computer processors;
extracting a face portion surrounding the face from the digital image by the one or more computer processors; and
normalizing the face portion to obtain the face image.
7. The computer-implemented method of claim 1 , wherein the facial landmarks are respectively associated with at least a portion of an eye, an eyebrow, a mouth, a chin, an edge of a face, or a nose in the face image.
8. The computer-implemented method of claim 1 , wherein the facial landmarks include a plurality of groups each of which describes at least a portion of a facial feature.
9. The computer-implemented method of claim 8 , wherein the facial landmarks include an eye, an eyebrow, a mouth, a chin, an edge of a face, or a nose in the face image.
10. The computer-implemented method of claim 1 , wherein each of the facial features is described by a pair of coordinates.
11. The computer-implemented method of claim 1 , wherein transforming the facial features into a two-dimensional matrix comprises:
describing each of the facial features by a pair of coordinates; and
forming a first matrix using the facial features and their respective coordinates.
12. The computer-implemented method of claim 11 , further comprising:
concatenating copied of the first matrix one or more times to produce the two-dimensional matrix.
13. The computer-implemented method of claim 1 , wherein the two-dimensional matrix is a square matrix.