Systems and methods for facial representation

1. A computer-implemented method, comprising:

determining, by a computing system, a set of fiducial points in a 2D image based on a first model trained to predict fiducial point configurations;

generating, by the computing system, a 2D face image based on the set of fiducial points determined in the 2D image;

generating, by the computing system, a 3D-aligned face image from the 2D face image, the generating further comprising:

determining, by the computing system, a localized set of fiducial points in the 2D face image based on a second model trained to predict fiducial point configurations and

transforming, by the computing system, the 2D face image to the 3D-aligned face image based at least in part on a piece-wise affine warping of the localized set of fiducial points in the 2D face image to the 3D-aligned face image; and

classifying, by the computing system, a facial identity corresponding to the 2D face image based at least in part on the 3D-aligned face image, wherein the classifying further comprises:

providing, by the computing system, the 3D-aligned face image as input to a deep neural network, wherein the deep neural network comprises at least a convolutional pooling layer and a set of locally-connected layers, wherein the convolutional pooling layer is configured to extract a first set of features from the 3D-aligned face image and at least one locally-connected layer is configured to extract a second set of features from the 3D-aligned face image.

2. The computer-implemented method of claim 1 , wherein transforming the 2D face image to the 3D-aligned face image further comprises:

generating, by the computing system, a set of anchor points that are placed on a 3D shape of a human face, wherein an anchor point corresponds to a fiducial point in the localized set of fiducial points determined for the 2D face image; and

transforming, by the computing system, the 2D face image to the 3D-aligned face image based on the localized set of fiducial points and the set of anchor points.

3. The computer-implemented method of claim 2 , wherein the 3D shape of the human face is generated based on an average of a plurality of 3D scans of human faces.

4. The computer-implemented method of claim 1 , wherein

the deep neural network is trained to classify the facial identity corresponding to the 2D face image based on a feature vector representation of the 3D-aligned face image.

5. The computer-implemented method of claim 4 , wherein the deep neural network includes a set of fully connected layers, wherein the 3D-aligned face image is provided as input to the convolutional pooling layer, and wherein the set of fully connected layers output the feature vector representation of the 3D-aligned face image.

6. The computer-implemented method of claim 5 , wherein the convolutional pooling layer includes at least a first convolutional layer, a max pooling layer, and a second convolutional layer, wherein at least the first convolutional layer includes a set of filters that each generate a feature map.

7. The computer-implemented method of claim 6 , wherein feature maps generated by the first convolutional layer are provided as inputs to the max pooling layer, and wherein outputs from the max pooling layer are provided as inputs to the second convolutional layer.

8. The computer-implemented method of claim 1 , wherein determining the localized set of fiducial points in the 2D face image further comprises:

determining, by the computing system, a face portion in the 2D face image.

9. The computer-implemented method of claim 1 , wherein the face portion in the 2D face image is determined based at least in part on a first machine learning regression model, and wherein the set of fiducial points in the 2D face image are localized based at least in part on a second machine learning regression model that is different from the first machine learning regression model.

10. A system comprising:

at least one processor; and

a memory storing instructions that, when executed by the at least one processor, cause the system to perform:

determining a set of fiducial points in a 2D image based on a first model trained to predict fiducial point configurations;

generating a 2D face image based on the set of fiducial points determined in the 2D image;

generating a 3D-aligned face image from the 2D face image, the generating further comprising:

determining a localized set of fiducial points in the 2D face image based on a second model trained to predict fiducial point configurations and

transforming the 2D face image to the 3D-aligned face image based at least in part on a piece-wise affine warping of the localized set of fiducial points in the 2D face image to the 3D-aligned face image; and

classifying a facial identity corresponding to the 2D face image based at least in part on the 3D-aligned face image, wherein the classifying further comprises:

providing the 3D-aligned face image as input to a deep neural network, wherein the deep neural network comprises at least a convolutional pooling layer and a set of locally-connected layers, wherein the convolutional pooling layer is configured to extract a first set of features from the 3D-aligned face image and at least one locally-connected layer is configured to extract a second set of features from the 3D-aligned face image.

11. The system of claim 10 , wherein transforming the 2D face image to the 3D-aligned face image further causes the system to perform:

generating a set of anchor points that are placed on a 3D shape of a human face, wherein an anchor point corresponds to a fiducial point in the localized set of fiducial points determined for the 2D face image; and

transforming the 2D face image to the 3D-aligned face image based on the localized set of fiducial points and the set of anchor points.

12. The system of claim 11 , wherein the 3D shape of the human face is generated based on an average of a plurality of 3D scans of human faces.

13. The system of claim 10 , wherein

the deep neural network is trained to classify the facial identity corresponding to the 2D face image based on a feature vector representation of the 3D-aligned face image.

14. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform:

determining a set of fiducial points in a 2D image based on a first model trained to predict fiducial point configurations;

generating a 2D face image based on the set of fiducial points determined in the 2D image;

generating a 3D-aligned face image from the 2D face image, the generating further comprising:

determining a localized set of fiducial points in the 2D face image based on a second model trained to predict fiducial point configurations and

transforming the 2D face image to the 3D-aligned face image based at least in part on a piece-wise affine warping of the localized set of fiducial points in the 2D face image to the 3D-aligned face image; and

classifying a facial identity corresponding to the 2D face image based at least in part on the 3D-aligned face image, wherein the classifying further comprises:

providing the 3D-aligned face image as input to a deep neural network, wherein the deep neural network comprises at least a convolutional pooling layer and a set of locally-connected layers, wherein the convolutional pooling layer is configured to extract a first set of features from the 3D-aligned face image and at least one locally-connected layer is configured to extract a second set of features from the 3D-aligned face image.

15. The non-transitory computer-readable storage medium of claim 14 , wherein transforming the 2D face image to the 3D-aligned face image further causes the computing system to perform:

generating a set of anchor points that are placed on a 3D shape of a human face, wherein an anchor point corresponds to a fiducial point in the localized set of fiducial points determined for the 2D face image; and

transforming the 2D face image to the 3D-aligned face image based on the localized set of fiducial points and the set of anchor points.

16. The non-transitory computer-readable storage medium of claim 15 , wherein the 3D shape of the human face is generated based on an average of a plurality of 3D scans of human faces.

17. The non-transitory computer-readable storage medium of claim 14 , wherein

the deep neural network is trained to classify the facial identity corresponding to the 2D face image based on a feature vector representation of the 3D-aligned face image.