Method for recognizing face using multiple patch combination based on deep neural network with fault tolerance and fluctuation robustness in extreme situation

1. A method for face recognition by using a multiple patch combination based on a deep neural network, comprising:

(a) if a face image with a 1-st size is acquired, a face-recognizing device performing a process of inputting the face image into a feature extraction network, to thereby allow the feature extraction network to generate a feature map by applying at least one convolution operation to the face image with the 1-st size, and thus to generate multiple features by applying sliding-pooling operation to the feature map, wherein the feature extraction network has been learned to extract at least one feature using a face image for training having a 2-nd size and wherein the 2-nd size is smaller than the 1-st size; and

(b) the face-recognizing device performing a process of inputting the multiple features into a learned neural aggregation network, to thereby allow the neural aggregation network to aggregate the multiple features and thus to output at least one optimal feature for the face recognition,

wherein, at (a), the face-recognizing device performs a process of inputting the face image with the 1-st size into at least one convolutional layer of the feature extraction network, to thereby allow the at least one convolutional layer to generate the feature map with a (1_1)-st size by applying at least one convolution operation to the face image with the 1-st size, a process of inputting the feature map with the (1_1)-st size into a pooling layer, to thereby allow the pooling layer to generate the multiple features with a (2_1)-st size by applying sliding-pooling operation to the feature map with the (1_1)-st size, wherein the (2_1)-st size is a size of a feature map for training, corresponding to a face image for training with the 2-nd size, generated by the feature extraction network,

wherein, the feature extraction network has been learned by a 1-st learning device repeating more than once (i) a process of inputting the face image for training with the 2-nd size into the at least one convolutional layer, to thereby allow the at least one convolutional layer to apply at least one convolution operation using at least one previously learned convolution parameter of the at least one convolutional layer to the face image for training with the 2-nd size and thus to generate the feature map for training with the (2_1)-st size, and (ii) a process of updating the at least one previously learned convolution parameter of the at least one convolutional layer by referring to characteristic information for training and its corresponding ground truth wherein the characteristic information for training corresponds to the feature map for training with the (2_1)-st size, such that one or more 1-st losses are minimized which are created by a 1-st loss layer.

2. The method of claim 1 , wherein the face-recognizing device instructs the pooling layer to apply sliding-pooling operation to the feature map with the (1_1)-st size using a preset stride.

3. The method of claim 1 , wherein, at (b), the face-recognizing device performs a process of inputting the multiple features into at least two attention blocks of the neural aggregation network, to thereby allow the at least two attention blocks to aggregate the multiple features and thus to output the optimal feature.

4. The method of claim 3 , wherein the face-recognizing device performs a process of allowing the at least two attention blocks to aggregate the multiple features and thus to generate each of quality scores corresponding to each of the multiple features, and a process of outputting the optimal feature by weighted summation of the multiple features using the quality scores.

5. The method of claim 3 , wherein the neural aggregation network has been learned by a 2-nd learning device repeating more than once (i) a process of inputting multiple face features for training corresponding to an image set of a single face or a video of the single face into the at least two attention blocks, to thereby allow the at least two attention blocks to generate each of quality scores for training corresponding to each of the face features for training by aggregation of the face features for training using previously learned attention parameters of the at least two attention blocks, (ii) a process of outputting at least one optimal feature for training by weighted summation of the face features for training using the quality scores for training, and (iii) a process of updating the previously learned attention parameters of the at least two attention blocks such that one or more 2-nd losses are minimized which are created by a 2-nd loss layer referring to the optimal feature for training and its corresponding ground truth.

6. The method of claim 1 , further comprising:

(c) the face-recognizing device performing a process of retrieving a reference feature from a face information database by referring to the optimal feature, to thereby recognize a face on the face image.

7. A face-recognizing device for face recognition by using a multiple patch combination based on a deep neural network, comprising:

at least one memory that stores instructions; and

at least one processor configured to execute the instructions to perform or support another device to perform: (I) if a face image with a 1-st size is acquired, a process of inputting the face image into a feature extraction network, to thereby allow the feature extraction network to generate a feature map by applying at least one convolution operation to the face image with the 1-st size, and thus to generate multiple features by applying sliding-pooling operation to the feature map, wherein the feature extraction network has been learned to extract at least one feature using a face image for training having a 2-nd size and wherein the 2-nd size is smaller than the 1-st size and (II) a process of inputting the multiple features into a learned neural aggregation network, to thereby allow the neural aggregation network to aggregate the multiple features and thus to output at least one optimal feature for the face recognition,

wherein, at the process of (I), the processor performs a process of inputting the face image with the 1-st size into at least one convolutional layer of the feature extraction network, to thereby allow the at least one convolutional layer to generate the feature map with a (1_1)-st size by applying at least one convolution operation to the face image with the 1-st size, a process of inputting the feature map with the (1_1)-st size into a pooling layer, to thereby allow the pooling layer to generate the multiple features with a (2_1)-st size by applying sliding-pooling operation to the feature map with the (1_1)-st size, wherein the (2_1)-st size is a size of a feature map for training, corresponding to a face image for training with the 2-nd size, generated by the feature extraction network,

wherein, the feature extraction network has been learned by a 1-st learning device repeating more than once (i) a process of inputting the face image for training with the 2-nd size into the at least one convolutional layer, to thereby allow the at least one convolutional layer to apply at least one convolution operation using at least one previously learned convolution parameter of the at least one convolutional layer to the face image for training with the 2-nd size and thus to generate the feature map for training with the (2_1)-st size, and (ii) a process of updating the at least one previously learned convolution parameter of the at least one convolutional layer by referring to characteristic information for training and its corresponding ground truth wherein the characteristic information for training corresponds to the feature map for training with the (2_1)-st size, such that one or more 1-st losses are minimized which are created by a 1-st loss layer.

8. The face-recognizing device of claim 7 , wherein the processor instructs the pooling layer to apply sliding-pooling operation to the feature map with the (1_1)-st size using a preset stride.

9. The face-recognizing device of claim 7 , wherein, at the process of (II), the processor performs a process of inputting the multiple features into at least two attention blocks of the neural aggregation network, to thereby allow the at least two attention blocks to aggregate the multiple features and thus to output the optimal feature.

10. The face-recognizing device of claim 9 , wherein the processor performs a process of allowing the at least two attention blocks to aggregate the multiple features and thus to generate each of quality scores corresponding to each of the multiple features, and a process of outputting the optimal feature by weighted summation of the multiple features using the quality scores.

11. The face-recognizing device of claim 9 , wherein the neural aggregation network has been learned by a 2-nd learning device repeating more than once (i) a process of inputting multiple face features for training corresponding to an image set of a single face or a video of the single face into the at least two attention blocks, to thereby allow the at least two attention blocks to generate each of quality scores for training corresponding to each of the face features for training by aggregation of the face features for training using previously learned attention parameters of the at least two attention blocks, (ii) a process of outputting at least one optimal feature for training by weighted summation of the face features for training using the quality scores for training, and (iii) a process of updating the previously learned attention parameters of the at least two attention blocks such that one or more 2-nd losses are minimized which are created by a 2-nd loss layer referring to the optimal feature for training and its corresponding ground truth.

12. The face-recognizing device of claim 7 , wherein the processor further performs:

(III) a process of retrieving a reference feature from a face information database by referring to the optimal feature, to thereby recognize a face on the face image.

13. A method for face recognition by using a multiple patch combination based on a deep neural network, comprising:

(a) if a face image with a 1-st size is acquired, a face-recognizing device performing a process of inputting the face image into a feature extraction network, to thereby allow the feature extraction network to generate a feature map by applying at least one convolution operation to the face image with the 1-st size, and thus to generate multiple features by applying sliding-pooling operation to the feature map, wherein the feature extraction network has been learned to extract at least one feature using a face image for training having a 2-nd size and wherein the 2-nd size is smaller than the 1-st size; and

(b) the face-recognizing device performing a process of inputting the multiple features into a learned neural aggregation network, to thereby allow the neural aggregation network to aggregate the multiple features and thus to output at least one optimal feature for the face recognition,

wherein, at (a), the face-recognizing device performs a process of inputting the face image with the 1-st size into at least one convolutional layer of the feature extraction network, to thereby allow the at least one convolutional layer to generate the feature map with a (1_1)-st size by applying at least one convolution operation to the face image with the 1-st size, a process of inputting the feature map with the (1_1)-st size into a pooling layer, to thereby allow the pooling layer to generate the multiple features with a (2_1)-st size by applying sliding-pooling operation to the feature map with the (1_1)-st size, wherein the (2_1)-st size is a size of a feature map for training, corresponding to a face image for training with the 2-nd size, generated by the feature extraction network,

wherein the face-recognizing device instructs the pooling layer to apply sliding-pooling operation to the feature map with (1_1)-st size using a preset stride.

14. A method for face recognition by using a multiple patch combination based on a deep neural network, comprising:

(a) if a face image with a 1-st size is acquired, a face-recognizing device performing a process of inputting the face image into a feature extraction network, to thereby allow the feature extraction network to generate a feature map by applying at least one convolution operation to the face image with the 1-st size, and thus to generate multiple features by applying sliding-pooling operation to the feature map, wherein the feature extraction network has been learned to extract at least one feature using a face image for training having a 2-nd size and wherein the 2-nd size is smaller than the 1-st size; and

(b) the face-recognizing device performing a process of inputting the multiple features into a learned neural aggregation network, to thereby allow the neural aggregation network to aggregate the multiple features and thus to output at least one optimal feature for the face recognition,

wherein, at (b), the face-recognizing device performs a process of inputting the multiple features into at least two attention blocks of the neural aggregation network, to thereby allow the at least two attention blocks to aggregate the multiple features and thus to output the optimal feature,

wherein the neural aggregation network has been learned by a 2-nd learning device repeating more than once (i) a process of inputting multiple face features for training corresponding to an image set of a single face or a video of the single face into the at least two attention blocks, to thereby allow the at least two attention blocks to generate each of quality scores for training corresponding to each of the face features for training by aggregation of the face features for training using previously learned attention parameters of the at least two attention blocks, (ii) a process of outputting at least one optimal feature for training by weighted summation of the face features for training using the quality scores for training, and (iii) a process of updating the previously learned attention parameters of the at least two attention blocks such that one or more 2-nd losses are minimized which are created by a 2-nd loss layer referring to the optimal feature for training and its corresponding ground truth.

15. A face-recognizing device for face recognition by using a multiple patch combination based on a deep neural network, comprising:

at least one memory that stores instructions; and

at least one processor configured to execute the instructions to perform or support another device to perform: (I) if a face image with a 1-st size is acquired, a process of inputting the face image into a feature extraction network, to thereby allow the feature extraction network to generate a feature map by applying at least one convolution operation to the face image with the 1-st size, and thus to generate multiple features by applying sliding-pooling operation to the feature map, wherein the feature extraction network has been learned to extract at least one feature using a face image for training having a 2-nd size and wherein the 2-nd size is smaller than the 1-st size and (II) a process of inputting the multiple features into a learned neural aggregation network, to thereby allow the neural aggregation network to aggregate the multiple features and thus to output at least one optimal feature for the face recognition,

wherein, at the process of (I), the processor performs a process of inputting the face image with the 1-st size into at least one convolutional layer of the feature extraction network, to thereby allow the at least one convolutional layer to generate the feature map with a (1_1)-st size by applying at least one convolution operation to the face image with the 1-st size, a process of inputting the feature map with the (1_1)-st size into a pooling layer, to thereby allow the pooling layer to generate the multiple features with a (2_1)-st size by applying sliding-pooling operation to the feature map with the (1_1)-st size, wherein the (2_1)-st size is a size of a feature map for training, corresponding to a face image for training with the 2-nd size, generated by the feature extraction network,

wherein the processor instructs the pooling layer to apply sliding-pooling operation to the feature map with the (1_1)-st size using a preset stride.

16. A face-recognizing device for face recognition by using a multiple patch combination based on a deep neural network, comprising:

at least one memory that stores instructions; and

at least one processor configured to execute the instructions to perform or support another device to perform: (I) if a face image with a 1-st size is acquired, a process of inputting the face image into a feature extraction network, to thereby allow the feature extraction network to generate a feature map by applying at least one convolution operation to the face image with the 1-st size, and thus to generate multiple features by applying sliding-pooling operation to the feature map, wherein the feature extraction network has been learned to extract at least one feature using a face image for training having a 2-nd size and wherein the 2-nd size is smaller than the 1-st size and (II) a process of inputting the multiple features into a learned neural aggregation network, to thereby allow the neural aggregation network to aggregate the multiple features and thus to output at least one optimal feature for the face recognition,

wherein, at the process of (II), the processor performs a process of inputting the multiple features into at least two attention blocks of the neural aggregation network, to thereby allow the at least two attention blocks to aggregate the multiple features and thus to output the optimal feature,

wherein the neural aggregation network has been learned by a 2-nd learning device repeating more than once (i) a process of inputting multiple face features for training corresponding to an image set of a single face or a video of the single face into the at least two attention blocks, to thereby allow the at least two attention blocks to generate each of quality scores for training corresponding to each of the face features for training by aggregation of the face features for training using previously learned attention parameters of the at least two attention blocks, (ii) a process of outputting at least one optimal feature for training by weighted summation of the face features for training using the quality scores for training, and (iii) a process of updating the previously learned attention parameters of the at least two attention blocks such that one or more 2-nd losses are minimized which are created by a 2-nd loss layer referring to the optimal feature for training and its corresponding ground truth.