Normalized three-dimensional avatar synthesis and perceptual refinement

1. A deep learning-based framework system for generating a normalized three-dimensional model of a human face from an unconstrained two-dimensional image of the human face, the system comprising a processor and memory, the processor executing instructions which cause the processor to:

receive the single unconstrained two-dimensional image of the human face;

use an inference network to determine an inferred normalized three-dimensional model of the human face based on the single unconstrained two-dimensional image of the human face; and

use a refinement network to iteratively determine the normalized three-dimensional model of the human face with a neutral expression and unshaded albedo textures under diffuse lighting conditions based on the inferred normalized three-dimensional model of the human face,

wherein the refinement network iteratively optimizes the latent vector by minimizing a perceptual difference between the single unconstrained two-dimensional image of the human face and the inferred normalized three-dimensional model of the human face to determine the normalized three-dimensional model of the human face with a neutral expression and unshaded albedo textures using a loss function that includes:

L inv =L pix +λ 1 L LPIPS +λ 2 L adv

where L pix is an L1 pixel loss of a synthesized position map and texture map, L LPIPS is a L PIPS distance as a perceptual loss, and L adv is an adversarial loss favoring realistic reconstruction results using three trained discriminators comprising a first discriminator for an evenly lit albedo map, a second discriminator for a neutral facial map, and a third discriminator for the combination of the evenly lit albedo map and the neutral facial map, and wherein the three discriminators are used to determine whether the minimizing a perceptual difference is real or fake compared to a real training dataset.

2. The system of claim 1 , wherein the inference network:

uses a pre-trained face recognition network to determine a facial embedding identity feature from the single unconstrained two-dimensional image of the human face; and

uses a synthesis network to determine an expression neutral face geometry and a normalized albedo texture for the inferred normalized three-dimensional model of the human face based on the facial embedding identity feature mapped to a latent vector.

3. The system of claim 2 , wherein the synthesis network is initially trained using a generative adversarial network based on a scan dataset of photometry scans and subsequently trained by the generative adversarial network based on a normalized face dataset of photometry scans and synthetic data.

4. The system of claim 2 , wherein a differential renderer generates a two-dimensional face image based on the inferred normalized three-dimensional model of the human face for comparison to the single unconstrained two-dimensional image of the human face to determine a perceptual difference.

5. The system of claim 4 , wherein the differential renderer generates the two-dimensional face image based on the inferred normalized three-dimensional model of the human face based on camera translation, camera rotation, and focal length of the single unconstrained two-dimensional image of the human face determined by a regression network.

6. The system of claim 5 , wherein the refinement network iteratively optimizes the latent vector by minimizing the perceptual difference between the single unconstrained two-dimensional image of the human face and the two-dimensional face image to determine the normalized three-dimensional model of the human face with a neutral expression and unshaded albedo textures using the loss function.

7. The system of claim 1 , wherein the normalized three-dimensional model of the human face comprises a position map and an albedo map.

8. A method of generating a normalized three-dimensional model of a human face from an unconstrained two-dimensional image of the human face, the method comprising:

receiving the single unconstrained two-dimensional image of the human face;

using an inference network to determine an inferred normalized three-dimensional model of the human face based on the single unconstrained two-dimensional image of the human face; and

using a refinement network to iteratively determine the normalized three-dimensional model of the human face with a neutral expression and unshaded albedo textures under diffuse lighting conditions based on the inferred normalized three-dimensional model of the human face,

wherein the refinement network iteratively optimizes the latent vector by minimizing a perceptual difference between the single unconstrained two-dimensional image of the human face and the inferred normalized three-dimensional model of the human face to determine the normalized three-dimensional model of the human face with a neutral expression and unshaded albedo textures using a loss function that includes:

L inv =L pix +λ 1 L LPIPS +λ 2 L adv

where L pix is an L1 pixel loss of a synthesized position map and texture map, L LPIPS is a L PIPS distance as a perceptual loss, and L adv is an adversarial loss favoring realistic reconstruction results using three trained discriminators comprising a first discriminator for an evenly lit albedo map, a second discriminator for a neutral facial map, and a third discriminator for the combination of the evenly lit albedo map and the neutral facial map, and wherein the three discriminators are used to determine whether the minimizing a perceptual difference is real or fake compared to a real training dataset.

9. The method of claim 8 , wherein the inference network:

uses a pre-trained face recognition network to determine a facial embedding identity feature from the single unconstrained two-dimensional image of the human face; and

uses a synthesis network to determine an expression neutral face geometry and a normalized albedo texture for the inferred normalized three-dimensional model of the human face based on the facial embedding identity feature mapped to a latent vector.

10. The method of claim 9 , wherein the synthesis network is initially trained using a generative adversarial network based on a scan dataset of photometry scans and subsequently trained by the generative adversarial network based on a normalized face dataset of photometry scans and synthetic data.

11. The method of claim 9 , wherein a differential renderer generates a two-dimensional face image based on the inferred normalized three-dimensional model of the human face for comparison to the single unconstrained two-dimensional image of the human face to determine a perceptual difference.

12. The method of claim 11 , wherein the differential renderer generates the two-dimensional face image based on the inferred normalized three-dimensional model of the human face based on camera translation, camera rotation, and focal length of the single unconstrained two-dimensional image of the human face determined by a regression network.

13. The method of claim 12 , wherein the refinement network iteratively optimizes the latent vector by minimizing the perceptual difference between the single unconstrained two-dimensional image of the human face and the two-dimensional face image to determine the normalized three-dimensional model of the human face with a neutral expression and unshaded albedo textures using the loss function.

14. The method of claim 8 , wherein the normalized three-dimensional model of the human face comprises a position map and an albedo map.

15. An apparatus comprising a storage medium storing a program having instructions which when executed by a processor will cause the processor to:

receive a single unconstrained two-dimensional image of a human face;

use an inference network to determine an inferred normalized three-dimensional model of the human face based on the single unconstrained two-dimensional image of the human face; and

use a refinement network to iteratively determine a normalized three-dimensional model of the human face with a neutral expression and unshaded albedo textures under diffuse lighting conditions based on the inferred normalized three-dimensional model of the human face,

wherein the refinement network iteratively optimizes the latent vector by minimizing a perceptual difference between the single unconstrained two-dimensional image of the human face and the inferred normalized three-dimensional model of the human face to determine the normalized three-dimensional model of the human face with a neutral expression and unshaded albedo textures using a loss function that includes:

L inv =L pix +λ 1 L LPIPS +λ 2 L adv

where L pix is an L1 pixel loss of a synthesized position map and texture map, L LPIPS is a L PIPS distance as a perceptual loss, and L adv is an adversarial loss favoring realistic reconstruction results using three trained discriminators comprising a first discriminator for an evenly lit albedo map, a second discriminator for a neutral facial map, and a third discriminator for the combination of the evenly lit albedo map and the neutral facial map, and wherein the three discriminators are used to determine whether the minimizing a perceptual difference is real or fake compared to a real training dataset.

16. The apparatus of claim 15 , wherein the inference network:

uses a pre-trained face recognition network to determine a facial embedding identity feature from the single unconstrained two-dimensional image of the human face; and

uses a synthesis network to determine an expression neutral face geometry and a normalized albedo texture for the inferred normalized three-dimensional model of the human face based on the facial embedding identity feature mapped to a latent vector.

17. The apparatus of claim 16 , wherein the synthesis network is initially trained using a generative adversarial network based on a scan dataset of photometry scans and subsequently trained by the generative adversarial network based on a normalized face dataset of photometry scans and synthetic data.

18. The apparatus of claim 16 , wherein a differential renderer generates a two-dimensional face image based on the inferred normalized three-dimensional model of the human face for comparison to the single unconstrained two-dimensional image of the human face to determine a perceptual difference.

19. The apparatus of claim 18 , wherein the differential renderer generates the two-dimensional face image based on the inferred normalized three-dimensional model of the human face based on camera translation, camera rotation, and focal length of the single unconstrained two-dimensional image of the human face determined by a regression network.

20. The apparatus of claim 19 , wherein the refinement network iteratively optimizes the latent vector by minimizing the perceptual difference between the single unconstrained two-dimensional image of the human face and the two-dimensional face image to determine the normalized three-dimensional model of the human face with a neutral expression and unshaded albedo textures using the loss function.

21. The apparatus of claim 15 , wherein the normalized three-dimensional model of the human face comprises a position map and an albedo map.