Video reenactment taking into account temporal information

1. An apparatus for inserting identity information from a source image of a first subject into a destination video of a second subject different than the first subject while mimicking motion in the destination video, said apparatus comprising:

an identity encoder configured to encode identity information from the source image and to produce an identity vector;

a driver encoder comprising a pose encoder configured to encode pose information from the destination video and to produce a pose vector, and a separate and independent motion encoder configured to encode motion information from the destination video and to produce a motion vector; and

a neural network generator having three inputs: the identity vector, the pose vector, and the motion vector; wherein

the neural network generator is configured to generate, in response to the three inputs, a composite video comprising identity information from the source video inserted into the destination video, where the composite video has substantially the same temporal information as in the destination video.

2. The apparatus of claim 1 where the driver encoder further comprises an illumination encoder.

3. The apparatus of claim 1 where the source image is a multi-frame static image or video, and a neural network identity code aggregator is positioned at an output of the identity encoder.

4. The apparatus of claim 3 where the identity encoder and pose encoder are each neural networks, and the motion encoder is not a neural network.

5. The apparatus of claim 1 where the motion encoder uses a computer vision method to represent input motion vectors obtained from the destination video.

6. The apparatus of claim 5 where the computer vision method is performed in real time.

7. The apparatus of claim 5 where the computer vision method is a method from the group of object tracking methods consisting of meanshift, camshift, and optical flow.

8. The apparatus of claim 1 wherein:

the neural network generator is a generative adversarial network; and

a multi-layered perceptron neural network is positioned between the identity vector, pose vector, and motion vector; and the neural network generator.

9. A method for inserting identity information from a source image of a first subject into a destination video of a second subject different than the first subject while mimicking motion in the destination video, said method comprising the steps of:

encoding identity information from the source image to produce an identity vector;

encoding pose information from the destination video to produce a pose vector;

separate from and independent of the pose information encoding step, encoding motion information from the destination video to produce a motion vector; and

activating a neural network generator to accept as inputs the identity vector, the pose vector, and the motion vector; and to generate in response thereto a composite video comprising identity information from the source video inserted into the destination video, where the composite video has substantially the same temporal information as in the destination video.

10. The method of claim 9 further comprising the step of training the neural network generator to minimize a loss function, where the loss function takes into account identity information from the source image, and pose and motion information from the destination video.

11. The method of claim 9 further comprising the steps of:

encoding illumination of the destination video to produce an illumination vector; and

feeding the illumination vector as an input to the neural network generator.

12. The method of claim 9 further comprising the steps of:

encoding illumination of the destination image to produce an illumination vector;

feeding an output of the neural network generator and the illumination vector into an illumination generator; and

activating the illumination generator to produce a composite video that takes into account illumination and temporal information of the destination video.

13. An apparatus for inserting identity information from a source image of a first subject into a destination video of a second subject different than the first subject while mimicking motion in the destination video, said apparatus comprising:

an identity encoder configured to encode identity information from the source image and to produce an identity vector;

a pose encoder configured to encode pose information from the destination video, and to produce a pose vector; and

a neural network generator and sequencer having two inputs: the identity vector and the pose vector; wherein

the neural network generator and sequencer is configured to encode motion in the destination video and to generate a composite video comprising identity information from the source image inserted into the destination video, where the composite video has substantially the same temporal information as in the destination video.

14. The apparatus of claim 13 where the generator and sequencer is a generative adversarial network in combination with a many-to-one or many-to-many neural network.

15. The apparatus of claim 13 where the generator and sequencer comprises at least one of a recurrent neural network, an attention neural network, and a transformer neural network.

16. The apparatus of claim 13 further comprising:

an illumination encoder configured to encode illumination information of the destination video; wherein:

the illumination encoder produces an illumination vector fed as an input to the generator and sequencer; and

the composite video produced by the generator and sequencer takes into account illumination and temporal information of the destination video.

17. The apparatus of claim 13 further comprising:

an illumination encoder coupled to the destination video, said illumination encoder configured to encode illumination information of the destination video and produce an illumination vector; and

an illumination generator having as inputs the illumination vector and an output of the generator and sequencer; wherein

the illumination generator produces a composite video that takes into account illumination and temporal information of the destination video.

18. A method for inserting identity information from a source image of a first subject into a destination video of a second subject different than the first subject while mimicking motion in the destination video, said method comprising the steps of:

encoding identity information from the source image;

using the encoded identity information from the source image to produce an identity vector;

encoding pose information from the destination video to produce a pose vector;

activating a neural network generator and sequencer to accept as inputs the identity vector and the pose vector, to encode motion in the destination video, and to generate a composite video comprising identify information from the source image inserted into the destination video, where the composite video has substantially the same temporal information as in the destination video.

19. The method of claim 18 further comprising encoding illumination information of the destination video to produce an illumination vector, and feeding the illumination vector as an input to the generator and sequencer.

20. The method of claim 18 wherein:

the neural network generator and sequencer is a generative adversarial network; and

a multi-layered perceptron neural network is positioned between the identity vector and the pose vector; and the neural network generator and sequencer.

21. The method of claim 18 further comprising the steps of:

encoding illumination information of the destination video to produce an illumination vector;

feeding an output of the generator and sequencer and the illumination vector to an illumination generator; and

activating the illumination generator to produce a composite video that takes into account illumination and temporal information of the destination video.