Generative adversarial neural network assisted compression and broadcast
A latent code defined in an input space is processed by the mapping neural network to produce an intermediate latent code defined in an intermediate latent space. The intermediate latent code may be used as appearance vector that is processed by the synthesis neural network to generate an image. The appearance vector is a compressed encoding of data, such as video frames including a person's face, audio, and other data. Captured images may be converted into appearance vectors at a local device and transmitted to a remote device using much less bandwidth compared with transmitting the captured images. A synthesis neural network at the remote device reconstructs the images for display.
1. A computer-implemented method, comprising:
transmitting learned data for configuring a remote neural network to replicate a style specific to a subject by transferring the learned data to an object;
processing, by a generator neural network, an image of the object to generate a vector encoding attributes of the object; and
transmitting the vector to the remote neural network, wherein the remote neural network constructs output images by applying the learned data to vectors, wherein the remote neural network is configured to apply the learned data to the vector to construct an output image of the object with at least one of the attributes of the object modified based on the style specific to the subject.
2. The computer-implemented method of claim 1 , wherein the vector is a compressed encoding of the object.
3. The computer-implemented method of claim 1 , further comprising training the generator neural network to produce a predicted image of the subject that is compared with a captured image of the subject to produce the learned data.
4. The computer-implemented method of claim 1 , wherein the attributes comprise head pose and facial expression.
5. The computer-implemented method of claim 1 , wherein the vector encodes at least one additional attribute associated with clothing, hairstyle, or lighting.
6. The computer-implemented method of claim 1 , wherein the vector is transmitted to the remote neural network during a videoconferencing session.
7. The computer-implemented method of claim 1 , wherein the step of processing is performed on a virtual machine comprising a portion of a graphics processing unit.
8. The computer-implemented method of claim 1 , wherein the step of processing the image is performed to generate an output image used for training, testing, or certifying a neural network employed in a machine, robot, or autonomous vehicle.
9. A computer-implemented method, comprising:
transmitting learned data for configuring a remote neural network to replicate a style specific to a subject by transferring the learned data to an object;
processing, by a generator neural network, an image of the object to generate a vector encoding attributes of the object, wherein the image is a frame of a video and the generator neural network is configured to generate vector adjustment values for each additional frame of the video corresponding to additional images; and
transmitting the vector to the remote neural network, wherein the remote neural network constructs output images by applying the learned data to vectors.
10. The computer-implemented method of claim 9 , further comprising transmitting the vector adjustment values to the remote neural network.
11. The computer-implemented method of claim 10 , further comprising:
receiving a request signal from the remote neural network; and
in response to the request signal, transmitting a second vector to the remote neural network, wherein the second vector is generated by processing at least one of the additional images.
12. A computer-implemented method, comprising:
transmitting learned data for configuring a remote neural network to replicate a style specific to a subject by transferring the learned data to an object;
processing, by a generator neural network, an image of the object to generate a vector encoding attributes of the object, wherein the vector comprises landmark points that delineate positions of key points corresponding to the object; and
transmitting the vector to the remote neural network, wherein the remote neural network constructs output images by applying the learned data to vectors.
13. A computer-implemented method, comprising:
transmitting learned data for configuring a remote neural network to replicate a style specific to a subject by transferring the learned data to an object;
processing, by a generator neural network, an image of the object to generate a vector encoding attributes of the object;
separating at least the object in the image from background image data;
encoding the background image data;
transmitting the encoded background image data to the remote neural network; and
transmitting the vector to the remote neural network, wherein the remote neural network constructs output images by applying the learned data to vectors.
14. A system, comprising a processor configured to implement a generator neural network that is configured to:
transmit learned data for configuring a remote neural network to replicate a style specific to a subject by transferring the learned data to an object;
process an image of the object to generate a vector encoding attributes of the object; and
transmit the vector to the remote neural network, wherein the remote neural network constructs output images by applying the learned data to vectors, wherein the remote neural network is configured to apply the learned data to the vector to construct an output image of the object with at least one of the attributes of the object modified based on the style specific to the subject.
15. The system of claim 14 , wherein the attributes comprise head pose and facial expression.
16. The system of claim 14 , wherein the vector encodes at least one additional attribute associated with clothing, hairstyle, or lighting.
17. A system, comprising a processor configured to implement a generator neural network that is configured to:
transmit learned data for configuring a remote neural network to replicate a style specific to a subject by transferring the learned data to an object;
process an image of the object to generate a vector encoding attributes of the object, wherein the vector comprises landmark points that delineate positions of key points corresponding to the object; and
transmit the vector to the remote neural network, wherein the remote neural network constructs output images by applying the learned data to vectors.
18. A non-transitory, computer-readable storage medium storing instructions that, when executed by a processing unit, cause the processing unit to:
transmit learned data for configuring a remote neural network to replicate a style specific to a subject by transferring the learned data to an object;
process an image of the object to generate a vector encoding attributes of the object; and
transmit the vector to the remote neural network, wherein the remote neural network constructs output images by applying the learned data to vectors, wherein the remote neural network is configured to apply the learned data to the vector to construct an output image of the object with at least one of the attributes of the object modified based on the style specific to the subject.
19. The non-transitory, computer-readable storage medium of claim 18 , wherein the attributes comprise head pose and facial expression.
20. The non-transitory, computer-readable storage medium of claim 18 , wherein the vector encodes at least one additional attribute associated with clothing, hairstyle, or lighting.