Hybrid deep learning method for recognizing facial expressions

1. A computer-implemented method for recognizing facial expressions by applying a hybrid of feature learning and feature engineering to face images, comprising:

conducting feature learning on a face image by one or more computer processors, comprising:

feeding the face image into a first convolution neural network to obtain a first decision;

conducting feature engineering on a face image by the one or more computer processors, comprising:

automatically detecting facial landmarks in the face image;

describing each of the facial landmarks by a set of facial features;

transforming the facial features into a two-dimensional matrix comprising coordinates of the facial features; and

feeding the two-dimensional matrix into a second convolution neural network to obtain a second decision;

computing a hybrid decision based on the first decision and the second decision; and

recognizing a facial expression in the face image in accordance to the hybrid decision.

2. The computer-implemented method of claim 1 , further comprising:

multiplying the first decision by a first weight to produce a first weighted decision; and

multiplying the second decision by a second weight to produce a second weighted decision,

wherein the hybrid decision is computed based on the first weighted decision and the second weighted decision.

3. The computer-implemented method of claim 2 , wherein the hybrid decision is an average, a sum, or a root-mean square function of the first weighted decision and the second weighted decision.

4. The computer-implemented method of claim 2 , updating the first weight and the second weight in the hybrid decision by backpropagation.

5. The computer-implemented method of claim 1 , wherein the two-dimensional matrix comprises multiple pairs of coordinates each associated with one of the facial features.

6. The computer-implemented method of claim 1 , further comprising:

automatically detecting a face in a digital image by the one or more computer processors;

extracting a face portion surrounding the face from the digital image by the one or more computer processors; and

normalizing the face portion to obtain the face image.

7. The computer-implemented method of claim 1 , wherein the facial landmarks respectively describe at least a portion of an eye, an eye brow, a mouth, a chin, an edge of a face, or a nose in the face image.

8. The computer-implemented method of claim 1 , wherein the facial landmarks include a plurality of groups each of which describes at least a portion of a facial feature.

9. The computer-implemented method of claim 8 , wherein the facial landmarks include an eye, an eye brow, a mouth, a chin, an edge of a face, or a nose in the face image.

10. The computer-implemented method of claim 1 , wherein each of the facial features is described by a pair of coordinates.

11. The computer-implemented method of claim 1 , wherein the step of transforming the facial features into a two-dimensional matrix comprises:

expressing each of the facial features by a pair of coordinates; and

forming a first matrix using the facial features and their respective coordinates.

12. The computer-implemented method of claim 11 , further comprising:

concatenating copied of the first matrix one or more times to produce the two-dimensional matrix.

13. The computer-implemented method of claim 1 , wherein the two-dimensional matrix is a square matrix.