Image reenactment with illumination disentanglement

1. Apparatus for inserting identity information from a source image into a destination image while mimicking illumination of the destination image, said apparatus comprising:

an identity encoder configured to encode just identity information of the source image and to produce an identity vector, where the identity encoder does not encode any pose information or illumination information of the source image;

a pose encoder configured to encode pose information from the totality of the destination image and to produce a pose vector;

separate from and independent of the pose encoder, an illumination encoder configured to encode illumination information from the totality of the destination image and to produce an illumination vector; and

a neural network generator coupled to the identity encoder, pose encoder, and illumination encoder, and having three inputs: the identity vector, the pose vector, and the illumination vector; wherein

the generator is configured to generate, in response to the three inputs, a composite image comprising identity information of the source image inserted into the destination image, wherein the composite image has substantially the same illumination as the destination image.

2. The apparatus of claim 1 wherein the neural network generator has an associated loss function comprising identity, pose, and illumination components.

3. The apparatus of claim 2 wherein the neural network generator has an associated training phase whose objective is to minimize the loss function.

4. The apparatus of claim 1 wherein the source image and the destination image are static images.

5. The apparatus of claim 1 wherein the source image and the destination image are video images.

6. The apparatus of claim 5 wherein the encoded identity information of the source image comprises the gait of the source.

7. The apparatus of claim 1 wherein the encoded identity information of the source image comprises facial features of the source.

8. The apparatus of claim 1 wherein the encoded identity information of the source image comprises hair features of the source.

9. The apparatus of claim 1 wherein the encoded identity information of the source image comprises torso features of the source.

10. The apparatus of claim 1 wherein the identity encoder, pose encoder, and illumination encoder are neural networks.

11. The apparatus of claim 1 wherein:

the neural network generator is a generative adversarial network; and

a multi-layered perceptron neural network is positioned between the three encoders and the neural network generator.

12. Apparatus for inserting identity information from a source image into a destination image while mimicking illumination of the destination image, said apparatus comprising:

an identity encoder configured to encode identity information of the source image and to produce an identity vector, where the identity encoder does not encode any pose information, and the identity information includes uncompensated illumination information of the source image;

a pose encoder configured to encode just pose information from the totality of the destination image and to produce a pose vector, where the pose encoder does not encode any illumination information of the destination image;

separate from and independent of the pose encoder, an illumination encoder configured to encode just illumination information from the totality of the destination image and to produce an illumination vector;

a primary neural network generator having two inputs, the identity vector and the pose vector, and an uncompensated composite image output; and

an illumination generator having two inputs, the uncompensated composite image output from the primary generator and the illumination vector; wherein

the illumination generator is a neural network optimized to produce an output composite image comprising identity information of the source image inserted into the destination image, wherein the composite image has substantially the same illumination as the destination image.

13. The apparatus of claim 12 wherein the illumination vector is represented as a set of spherical harmonics.

14. The apparatus of claim 12 wherein:

the primary neural network generator is a generative adversarial network; and

the identity vector and the pose vector are fed to the primary neural network generator via a multi-layered perceptron neural network.

15. A method for inserting identity information from a source image into a destination image while mimicking illumination of the destination image, said method comprising the steps of:

encoding identity information of the source image to produce an identity vector, where the identity information encoding does not encode any pose information or illumination information of the source image;

encoding pose information from the totality of the destination image to produce a pose vector;

separately from and independently of the pose information encoding step, encoding illumination information from the totality of the destination image to produce an illumination vector; and

activating a neural network generator to accept as inputs the identity vector, the pose vector, and the illumination vector; and to generate in response thereto a composite image comprising identity information of the source image inserted into the destination image, wherein the composite image has substantially the same illumination as the destination image.

16. The method of claim 15 further comprising the step of training the neural network generator to minimize a loss function.

17. The method of claim 16 wherein the loss function takes into account identity information of the source image, and pose and illumination information of the destination image.

18. A method for inserting identity information from a source image into a destination image while mimicking illumination of the destination image, said method comprising the steps of:

encoding identity information of the source image to produce an identity vector, where the identity information encoding does not encode any pose information, and the identity information encoding does encode unwanted illumination of the source image;

encoding pose information from the totality of the destination image to produce a pose vector;

separately from and independently of the pose information encoding step, encoding illumination information from the totality of the destination image to produce an illumination vector;

activating a neural network generator to accept as inputs the identity vector and the pose vector, and to generate in response thereto a composite image comprising identity information of the source image inserted into the destination image, wherein the composite image contains uncompensated illumination information of the source image; and

activating an illumination generator to accept as inputs the uncompensated composite image and the illumination vector, and to generate in response thereto a compensated composite image comprising identity information of the source image inserted into the destination image, wherein the compensated composite image has substantially the same illumination as the destination image.

19. The method of claim 18 wherein the illumination vector is represented as a set of spherical harmonics.

20. The method of claim 18 wherein:

the neural network generator is a generative adversarial network; and

a multi-layered perceptron neural network is positioned between the identity vector and the pose vector on the one hand, and the neural network generator on the other hand.

21. At least one computer-readable medium for executing computer-readable instructions configured to insert identity information from a source image into a destination image while mimicking illumination of the destination image, said instructions performing the steps of:

encoding identity information of the source image to produce an identity vector, where the identity information encoding does not encode any pose information or illumination information of the source image;

encoding pose information from the totality of the destination image to produce a pose vector;

separately from and independently of the pose information encoding step, encoding illumination information from the totality of the destination image to produce an illumination vector; and

activating a neural network generator to accept as inputs the identity vector, the pose vector, and the illumination vector; and to generate in response thereto a composite image comprising identity information of the source image inserted into the destination image, wherein the composite image has substantially the same illumination as the destination image.

22. At least one computer-readable medium for executing computer-readable instructions configured to insert identity information from a source image into a destination image while mimicking illumination of the destination image, said instructions performing the steps of:

encoding identity information of the source image to produce an identity vector, where the identity information encoding does not encode any pose information, and the identity information encoding does encode unwanted illumination of the source image;

encoding pose information from the totality of the destination image to produce a pose vector;

separately from and independently of the pose information encoding step, encoding illumination information from the totality of the destination image to produce an illumination vector;

activating a neural network generator to accept as inputs the identity vector and the pose vector, and to generate in response thereto a composite image comprising identity information of the source image inserted into the destination image, wherein the composite image contains uncompensated illumination information of the source image; and

activating an illumination generator to accept as inputs the uncompensated composite image and the illumination vector, and to generate in response thereto a compensated composite image comprising identity information of the source image inserted into the destination image, wherein the compensated composite image has substantially the same illumination as the destination image.