Generative adversarial neural network assisted reconstruction
A latent code defined in an input space is processed by the mapping neural network to produce an intermediate latent code defined in an intermediate latent space. The intermediate latent code may be used as appearance vector that is processed by the synthesis neural network to generate an image. The appearance vector is a compressed encoding of data, such as video frames including a person's face, audio, and other data. Captured images may be converted into appearance vectors at a local device and transmitted to a remote device using much less bandwidth compared with transmitting the captured images. A synthesis neural network at the remote device reconstructs the images for display.
1. A computer-implemented method, comprising:
obtaining learned data for replicating a style specific to a subject;
configuring a neural network to apply the learned data to images processed by the neural network to modify at least one attribute according to the style specific to the subject;
receiving, through a network, a vector generated by a remote encoder that represents attributes of an image; and
processing, by the neural network, the vector to reconstruct the image with at least one of the attributes that is modified based on the style to produce a reconstructed image.
2. The computer-implemented method of claim 1 , wherein the image includes an object and the processing reconstructs the image including the object having at least one attribute modified based on the style.
3. The computer-implemented method of claim 1 , wherein the image includes an object that is blurry and the processing reconstructs the image with the object clearly defined by using the vector to control coarse scale styles and the learned data to control fine scale styles.
4. The computer-implemented method of claim 1 , wherein the subject is a real or synthetic human subject, avatar, or character.
5. The computer-implemented method of claim 1 , wherein the vector is a compressed encoding of the image.
6. The computer-implemented method of claim 1 , wherein the learned data is determined by training a generator neural network to produce predicted images of the subject that are compared with captured images of the subject.
7. The computer-implemented method of claim 1 , further comprising displaying the reconstructed image of the object in a viewing environment, wherein the neural network produces the reconstructed image according to lighting in the viewing environment instead of different lighting associated with the image and that is encoded in the vector.
8. The computer-implemented method of claim 1 , wherein the vector is computed by transforming landmark points that delineate positions of key points on the object according to a learned or optimized matrix.
9. The computer-implemented method of claim 1 , further comprising interpolating a first vector and a second vector corresponding to two frames in a video to produce the vector, wherein the image is between the two frames.
10. The computer-implemented method of claim 1 , further comprising receiving audio data, wherein the audio data is used to produce the reconstructed image.
11. The computer-implemented method of claim 1 , wherein the steps of obtaining, receiving, and processing are performed on a virtual machine comprising a portion of a graphics processing unit.
12. The computer-implemented method of claim 1 , wherein the image is used for training, testing, or certifying a neural network employed in a machine, robot, or autonomous vehicle.
13. A system, comprising a processor configured to:
obtain learned data for replicating a style specific to a subject; and
implement a neural network that is configured to apply the learned data to images to modify at least one attribute according to the style specific to the subject, wherein the neural network:
receives, through a network, a vector generated by a remote encoder that represents attributes of an image; and
processes the vector to reconstruct the image with at least one of the attributes that is modified based on the style to produce a reconstructed image.
14. The system of claim 13 , wherein the image includes an object and processing the vector reconstructs the image including the object having at least one attribute modified based on the style.
15. The system of claim 13 , wherein the image includes an object that is blurry and processing the vector reconstructs the image with the object clearly defined by using the vector to control coarse scale styles and the learned data to control fine scale styles.
16. The system of claim 13 , wherein the subject is a real or synthetic human subject, avatar, or character.
17. The system of claim 13 , wherein the vector is a compressed encoding of the image.
18. A non-transitory, computer-readable storage medium storing instructions that, when executed by a processing unit, cause the processing unit to:
obtain learned data for replicating a style specific to a subject; and
implement a neural network that is configured to apply the learned data to images to modify at least one attribute according to the style specific to the subject, wherein the neural network:
receives, through a network, a vector generated by a remote encoder that represents attributes of an image; and
processes the vector to reconstruct the image with at least one of the attributes that is modified based on the style to produce a reconstructed image.
19. The non-transitory, computer-readable storage medium of claim 18 , wherein the image includes an object and processing the vector reconstructs the image including the object having at least one attribute modified based on the style.
20. The non-transitory, computer-readable storage medium of claim 18 , wherein the image includes an object that is blurry and processing the vector reconstructs the image with the object clearly defined by using the vector to control coarse styles and the data to control fine styles.