Disentangling latent representations for image reenactment

1. A method for image processing, comprising:

encoding features of a source image to obtain a source appearance encoding that represents inherent attributes of a face in the source image;

encoding features of a target image to obtain a target non-appearance encoding that represents contextual attributes of the target image;

combining the source appearance encoding and the target non-appearance encoding using a reconstruction network to obtain combined image features, wherein the combined image features comprise a reconstructed vector in a latent space of a generator network; and

generating, using the generator network, a modified target image based on the combined image features, wherein the modified target image includes the inherent attributes of the face in the source image together with the contextual attributes of the target image.

2. The method of claim 1 , further comprising:

extracting a plurality of target frames from a target video, wherein each of the plurality of target frames includes the face of the target image;

encoding each of the plurality of target frames to obtain a target frame appearance encoding and a target frame non-appearance encoding;

generating a modified target frame for each of the plurality of target frames based on corresponding combined image features; and

generating a modified video depicting the face in motion based on the modified target frame for each of the plurality of target frames, wherein the modified video includes the inherent attributes of the face in the source image together with contextual attributes of the target video.

3. The method of claim 1 , further comprising:

encoding the features of the source image to obtain a source non-appearance encoding, wherein the source non-appearance encoding represents contextual attributes of the source image; and

encoding the features of the target image to obtain a target appearance encoding, wherein the target appearance encoding represents inherent attributes of a face in the target image.

4. The method of claim 1 , wherein:

the inherent attributes include an identity attribute, a hair attribute, an eyewear attribute, an age attribute, a gender expression attribute, or any combination thereof.

5. The method of claim 1 , wherein:

the contextual attributes include a pose attribute, an expression attribute, a gaze direction attribute, a lighting attribute, or any combination thereof.

6. The method of claim 1 , wherein:

the features of the source image and the features of the target image comprise vectors in the latent space of the generator network.

7. The method of claim 1 , wherein:

the source appearance encoding and the target non-appearance encoding are generated by a disentangling network.

8. An apparatus comprising:

at least one processor; and

at least one memory including instructions executable by the at least one processor to:

encode features of a source image to obtain a source appearance encoding that represents inherent attributes of a face in the source image;

encode features of a target image to obtain a target non-appearance encoding that represents contextual attributes of the target image;

combine the source appearance encoding and the target non-appearance encoding using a reconstruction network to obtain combined image features, wherein the combined image features comprise a reconstructed vector in a latent space of a generator network; and

generate, using the generator network, a modified target image based on the combined image features, wherein the modified target image includes the inherent attributes of the face in the source image together with the contextual attributes of the target image.

9. The apparatus of claim 8 , further comprising instructions executable by the at least one processor to:

extract a plurality of target frames from a target video, wherein each of the plurality of target frames includes the face of the target image;

encode each of the plurality of target frames to obtain a target frame appearance encoding and a target frame non-appearance encoding;

generate a modified target frame for each of the plurality of target frames based on corresponding combined image features; and

generate a modified video depicting the face in motion based on the modified target frame for each of the plurality of target frames, wherein the modified video includes the inherent attributes of the face in the source image together with contextual attributes of the target video.

10. The apparatus of claim 8 , further comprising instructions executable by the at least one processor to:

encode the features of the source image to obtain a source non-appearance encoding, wherein the source non-appearance encoding represents contextual attributes of the source image; and

encode the features of the target image to obtain a target appearance encoding, wherein the target appearance encoding represents inherent attributes of a face in the target image.

11. The apparatus of claim 8 , wherein:

the inherent attributes include an identity attribute, a hair attribute, an eyewear attribute, an age attribute, a gender expression attribute, or any combination thereof.

12. The apparatus of claim 8 , wherein:

the contextual attributes include a pose attribute, an expression attribute, a gaze direction attribute, a lighting attribute, or any combination thereof.

13. A non-transitory computer readable medium storing code for image processing, the code comprising instructions executable by at least one processor to:

encode features of a source image to obtain a source appearance encoding that represents inherent attributes of a face in the source image;

encode features of a target image to obtain a target non-appearance encoding that represents contextual attributes of the target image;

combine the source appearance encoding and the target non-appearance encoding using a reconstruction network to obtain combined image features, wherein the combined image features comprise a reconstructed vector in a latent space of a generator network; and

generate, using the generator network, a modified target image based on the combined image features, wherein the modified target image includes the inherent attributes of the face in the source image together with the contextual attributes of the target image.

14. The non-transitory computer readable medium of claim 13 , the code further comprising instructions executable by the at least one processor to:

extract a plurality of target frames from a target video, wherein each of the plurality of target frames includes the face of the target image;

encode each of the plurality of target frames to obtain a target frame appearance encoding and a target frame non-appearance encoding;

generate a modified target frame for each of the plurality of target frames based on corresponding combined image features; and

generate a modified video depicting the face in motion based on the modified target frame for each of the plurality of target frames, wherein the modified video includes the inherent attributes of the face in the source image together with contextual attributes of the target video.

15. The non-transitory computer readable medium of claim 13 , the code further comprising instructions executable by the at least one processor to:

encode the features of the source image to obtain a source non-appearance encoding, wherein the source non-appearance encoding represents contextual attributes of the source image; and

encode the features of the target image to obtain a target appearance encoding, wherein the target appearance encoding represents inherent attributes of a face in the target image.

16. The non-transitory computer readable medium of claim 13 , wherein:

the inherent attributes include an identity attribute, a hair attribute, an eyewear attribute, an age attribute, a gender expression attribute, or any combination thereof.

17. The non-transitory computer readable medium of claim 13 , wherein:

the contextual attributes include a pose attribute, an expression attribute, a gaze direction attribute, a lighting attribute, or any combination thereof.