Method and apparatus for face recognition robust to alignment status of the face

1. A method for recognizing a face using a deep neural network (DNN), the method comprising:

estimating prior information of a facial shape from an input image cropped from an image including at least one face by using a first DNN of a face recognition apparatus;

extracting feature information of facial appearance from the input image by using a second DNN of the face recognition apparatus;

training, by using a face image decoder of the face recognition apparatus based on the prior information and the feature information, the face recognition apparatus so that a face image close to a optimally aligned face image is decoded; and

extracting, from a test image, facial shape-aware features in an inference step by using the second DNN included in the trained face recognition apparatus,

wherein:

the training the face recognition apparatus comprises

performing trainiig by updating parameters of the second DNN and the face image decoder,

wherein:

the performing of the training comprises

performing the training based on at least one loss function regarding training parameters of the first DNN, the second DNN and the face image decoder,

wherein:

the at least one loss function includes a cross-entropy loss function about the feature information extracted by the second DNN, a loss function in a feature alignment process, and a loss function in a pixel alignment process,

wherein:

the cross-entropy loss function is used when classifying the feature information of the facial appearance as one of classes for embedded features,

the loss function in a feature alignment process is used when training the facial shape-aware features, and

the loss function in a pixel alignment process is used when training the face image decoder.

2. The method of claim 1 , wherein:

the estimating prior information of a facial shape comprises

estimating a facial key point coordinate or a parsing map as the prior information to identify contour of elements of the face.

3. The method of claim 1 , wherein:

the performing of the training comprises

obtaining the parameters through mini batch gradient descent scheme.

4. The method of claim 1 , wherein

the feature alignment process is a process of comparing a global average pooling (GAP) result for combination information of the prior information and the feature information and a GAP result of a feature map extracted from the second DNN.

5. The method of claim 1 , wherein:

the pixel alignment process is a comparing process between the face image decoded by the face image decoder and the optimally aligned face image.

6. The method of claim 1 , wherein:

the extracting, from a test image, facial shape-aware features in an inference step comprises:

outputting a feature map from the feature information extracted by the second DNN by using an additional learner;

outputting a GAP result from the feature map by using a global average pooling layer; and

extracting the facial shape-aware features by using a fully connected layer from the GAP result.

7. An apparatus for recognizing a face using a deep neural network (DNN), the apparatus comprising:

processor and memory;

wherein the processor executes a program stored in the memory to perform:

estimating prior information of a facial shape from an input image cropped from an image including at least one face by using a first DNN;

extracting feature information of the facial appearance from the input image by using a second DNN;

training, by using a face image decoder based on the prior information and the feature information, the apparatus so that a face image close to an aligned face image is decoded; and

extracting, from a test image, facial shape-aware features by using the second DNN included in the trained apparatus,

wherein when the processor performs the training the apparatus the processor performs

performing training by updating parameters of the second DNN and the face image decoder,

wherein when the processor performs the performing of the training, the processor performs

performing the training based on at least one loss function regarding training parameters of the first DNN, the second DNN, and the face image decoder,

wherein the at least one loss function includes at least one of a cross-entropy loss function about the feature information extracted by the second DNN, a loss function in a feature alignment process, and a loss function in a pixel alignment process,

wherein:

the cross-entropy loss function is used when classifying the feature information of the facial appearance as one of classes for embedded features,

the loss function in a feature alignment process is used when training the facial shape-aware features, and

the loss function in a pixel alignment process is used when training the face image decoder.

8. The apparatus of claim 7 ,

wherein when the processor performs the estimating prior information of a facial shape, the processor performs

estimating a facial key point coordinate or a parsing map as the prior information to identify contour of elements of the face.

9. The apparatus of claim 7 ,

wherein when the processor performs the performing of the training, the processor performs

obtaining the parameters through mini batch gradient descent scheme.

10. The apparatus of claim 7 , wherein

the feature alignment process is a process of comparing a global average pooling (GAP) result for combination information of the prior information and the feature information and a GAP result of a feature map extracted from the second DNN.

11. The apparatus of claim 7 , wherein

the pixel alignment process is a comparing process between the face image decoded by the face image decoder and the optimally aligned face image.

12. The apparatus of claim 7 ,

wherein when the processor performs the extracting, from a test image, facial shape-aware features, the processor performs:

outputting a feature map from the feature information extracted by the second DNN by using an additional learner;

outputting a GAP result from the feature map by using a global average pooling layer; and

extracting the facial shape-aware features by using a fully connected layer from the GAP result.