Normalization of facial images using deep neural networks

1. A deep learning-based framework system for generating a normalization of a single two-dimensional image of an unconstrained human face, the system comprising a processor and memory, the processor executing instructions which cause the processor to:

receive the single two-dimensional image of the unconstrained human face;

use a perspective undistortion network to remove perspective distortion from the unconstrained human face to generate an undistorted face based on the unconstrained human face, wherein removing perspective distortion includes performing three-dimensional face fitting to the undistorted face to obtain a fitted three-dimensional mesh, spherical harmonics shading coefficients, and UV texture;

use a lighting translation network to generate an evenly lit face based on the undistorted face by normalizing lighting of the undistorted face; and

use an expression neutralization network to generate a frontalized and neutralized expression face based on the evenly lit face.

2. The system of claim 1 , wherein the perspective undistortion network:

predicts a dense two-dimensional flow based on the image and two-dimensional landmarks detected in the image to generate the undistorted face.

3. The system of claim 1 , wherein the lighting translation network uses the spherical harmonics shading coefficients to generate the evenly lit face.

4. The system of claim 1 , wherein the expression neutralization network uses the UV texture to generate the frontalized and neutralized expression face.

5. The system of claim 1 , wherein the expression neutralization network generates a frontalized and neutralized face by: using a geometry neutralization network to neutralize geometry of the evenly lit face to generate a geometry neutralized face; and using a facial texture neutralization network to neutralize facial texture of the geometry neutralized face to generate the frontalized and neutralized expression face.

6. The system of claim 1 , wherein the perspective undistortion network is a generative adversarial network trained via dense flow fields by fitting three-dimensional face models to input images with known focal length and camera-to-subject distance; wherein the lighting translation network is a conditional generative adversarial network trained via synthetic illumination data based on portrait relighting; and wherein the expression neutralization network is a generative adversarial network trained based on expression-to-neutral and neutral-to-expression datasets.

7. A method of generating a normalization of a single two-dimensional image of an unconstrained human face comprising:

receiving the single two-dimensional image of the unconstrained human face;

using a perspective undistortion network to remove perspective distortion from the unconstrained human face to generate an undistorted face based on the unconstrained human face, wherein removing perspective distortion includes performing three-dimensional face fitting to the undistorted face to obtain a fitted three-dimensional mesh, spherical harmonics shading coefficients, and UV texture;

using a lighting translation network to generate an evenly lit face based on the undistorted face by normalizing lighting of the undistorted face; and

using an expression neutralization network to generate a frontalized and neutralized expression face based on the evenly lit face.

8. The method of claim 7 , wherein the perspective undistortion network:

predicts a dense two-dimensional flow based on the image and two-dimensional landmarks detected in the image to generate the undistorted face.

9. The method of claim 7 , wherein the lighting translation network uses the spherical harmonics to generate the evenly lit face.

10. The method of claim 7 , wherein the expression neutralization network uses the UV texture to generate the frontalized and neutralized expression face.

11. The method of claim 7 , wherein the expression neutralization network generates a frontalized and neutralized face by: using a geometry neutralization network to neutralize geometry of the evenly lit face to generate a geometry neutralized face; and using a facial texture neutralization network to neutralize facial texture of the geometry neutralized face to generate the frontalized and neutralized face.

12. The method of claim 7 , wherein the perspective undistortion network is a generative adversarial network trained via dense flow fields by fitting three-dimensional face models to input images with known focal length and camera-to-subject distance; wherein the lighting translation network is a conditional generative adversarial network trained via synthetic illumination data based on portrait relighting; and wherein the expression neutralization network is a generative adversarial network trained based on expression-to-neutral and neutral-to-expression datasets.

13. An apparatus comprising a storage medium storing a program having instructions which when executed by a processor will cause the processor to:

receive the single two-dimensional image of an unconstrained human face;

use a perspective undistortion network to remove perspective distortion from the unconstrained human face to generate an undistorted face based on the unconstrained human face, wherein removing perspective distortion includes performing three-dimensional face fitting to the undistorted face to obtain a fitted three-dimensional mesh, spherical harmonics shading coefficients, and UV texture;

use a lighting translation network to generate an evenly lit face based on the undistorted face by normalizing lighting of the undistorted face; and

use an expression neutralization network to generate a frontalized and neutralized expression face based on the evenly lit face.

14. The apparatus of claim 13 , wherein the perspective undistortion network:

predicts a dense two-dimensional flow based on the image and two-dimensional landmarks detected in the image to generate the undistorted face.

15. The apparatus of claim 13 , wherein the lighting translation network uses the spherical harmonics shading coefficients to generate the evenly lit face.

16. The apparatus of claim 13 , wherein the expression neutralization network generates a frontalized and neutralized face by: using a geometry neutralization network to neutralize geometry of the evenly lit face to generate a geometry neutralized face; and using a facial texture neutralization network to neutralize facial texture of the geometry neutralized face to generate the frontalized and neutralized face.

17. The apparatus of claim 13 , wherein the perspective undistortion network is a generative adversarial network trained via dense flow fields by fitting three-dimensional face models to input images with known focal length and camera-to-subject distance; wherein the lighting translation network is a conditional generative adversarial network trained via synthetic illumination data based on portrait relighting; and wherein the expression neutralization network is a generative adversarial network trained based on expression-to-neutral and neutral-to-expression datasets.