Video reenactment with hair shape and motion transfer

1. An apparatus for inserting face and hair information from a source image into a destination video while mimicking pose, illumination, and hair motion of the destination video, said apparatus comprising:

an identity encoder configured to encode face and hair information of the source image and to produce as an output an identity vector, where said face and hair information does not comprise any motion information;

a pose encoder configured to encode pose information of the destination video and to produce as an output a pose vector;

an illumination encoder configured to encode head and hair illumination of the destination video, and to produce as an output an illumination vector;

a hair motion encoder configured to encode hair motion information of the destination video and to produce as an output a hair motion vector, where said hair motion information varies over time; and

a neural network generator having four inputs: the identify vector, the pose vector, the illumination vector, and the hair motion vector; wherein

the neural network generator is configured to generate, in response to the four inputs, a composite video comprising face and hair information from the source image inserted into the destination video.

2. The apparatus of claim 1 wherein:

the identity encoder, pose encoder, and hair motion encoder are neural networks; and

the illumination encoder is not a neural network.

3. The apparatus of claim 1 wherein the neural network generator is a generative adversarial network.

4. The apparatus of claim 1 wherein a code aggregator is positioned between each of the four encoders and the neural network generator.

5. The apparatus of claim 4 wherein each code aggregator comprises at least one of a recurrent neural network, an attention neural network, and a transformer neural network.

6. The apparatus of claim 1 wherein:

the neural network generator is a generative adversarial network; and

a multi-layered perceptron neural network mapper is inserted between each encoder and the neural network generator.

7. The apparatus of claim 1 wherein:

the illumination vector is not fed to the neural network generator, but rather is fed to an illumination generator neural network coupled to the composite video;

whereby illumination compensation is performed after the composite video has been generated.

8. A method for inserting face and hair information from a source image into a destination video while mimicking pose, illumination, and hair motion of the destination video, said method comprising steps of:

encoding face and hair identity information of the source image to produce an identity vector, where said face and hair identity information does not vary over time;

encoding pose and illumination information of the destination video to produce a pose and illumination vector;

encoding hair motion information of the destination video to produce a hair motion vector, where said hair motion information varies over time; and

feeding the identify vector, pose and illumination vector, and hair motion vector as three inputs to a neural network generator, said neural network generator generating in response to the three inputs a composite video comprising face and hair information from the source image inserted into the destination video.

9. The method of claim 8 wherein the identity encoder, pose and illumination encoder, hair motion encoder, and neural network generator are all neural networks.

10. The method of claim 9 wherein all of said neural networks are trained together in a training phase before performing the steps of claim 8 .

11. An apparatus for inserting face and hair information from a source image into a destination video while mimicking pose, illumination, and hair motion of the destination video, said apparatus comprising:

an identity encoder configured to encode face and hair information of the source image and to produce as an output an identity vector, where said face and hair information does not comprise any motion information;

a pose encoder configured to encode pose information of the destination video and to produce as an output a pose vector;

an illumination encoder configured to encode head and hair illumination of the destination video, and to produce as an output an illumination vector; and

a hair motion encoder configured to encode hair motion information of the destination video, and to produce as an output a hair motion vector, where said hair motion information varies over time; wherein

the identity vector, pose vector, illumination vector, and hair motion vector are fed as four inputs to a neural network generator plus sequencer, wherein the neural network generator plus sequencer, in response to the four inputs, produces a composite video comprising face and hair information of the source image inserted into the destination video.

12. The apparatus of claim 11 wherein:

the identity encoder, pose encoder, and hair motion encoder are neural networks; and

the illumination encoder is not a neural network.

13. The apparatus of claim 11 wherein the generator plus sequencer comprises at least one of a generative adversarial network, a recurrent neural network, an attention neural network, and a transformer neural network.

14. The apparatus of claim 11 wherein an identity code aggregator is inserted between the identity encoder and the generator plus sequencer.

15. The apparatus of claim 14 wherein the identity code aggregator is a neural network.

16. The apparatus of claim 11 wherein a multi-layered perceptron neural network mapper is positioned between the identity encoder and the generator plus sequencer.

17. The apparatus of claim 11 wherein:

the output of the illumination encoder is not fed to the generator plus sequencer, but rather is fed to an illumination generator neural network coupled to the composite video; whereby

illumination compensation is performed after the composite video has been generated.

18. A method for inserting face and hair information from a source image into a destination video while mimicking pose, illumination, and hair motion of the destination video, said method comprising steps of:

encoding face and hair identity information of the source image to produce an identity vector, where said face and hair identity information does not vary over time;

activating a pose and illumination encoder to encode pose and illumination information of the destination video;

activating a hair motion encoder to encode hair motion information of the destination video, where said hair motion information varies over time; and

feeding the identity vector, an output of the pose and illumination vector, and an output of the hair motion encoder as three inputs to a neural network generator plus sequencer, wherein the neural network generator plus sequencer, in response to the three inputs, produces a composite video comprising face and hair information of the source image inserted into the destination video.

19. The method of claim 18 wherein the identity encoder, pose and illumination encoder, hair motion encoder, and generator plus sequencer are all neural networks, and are all trained together in a training phase before performing the steps of claim 18 .

20. The method of claim 18 wherein:

the pose and illumination encoder produces two outputs: a pose vector and an illumination vector;

the pose vector is fed as an input to the generator plus sequencer; and

the illumination vector is fed as an input to an illumination generator neural network coupled to the composite video; whereby

illumination compensation is performed after the composite video has been generated.