Video synthesis using one or more neural networks

1. A processor, comprising:

one or more circuits to use one or more neural networks to generate one or more second images based, at least in part, on one or more point cloud representations of one or more first images.

2. The processor of claim 1 , wherein the one or more circuits are further to assign colors to one or more portions of the one or more point cloud representations, or one or more motion vector representations, based at least in part on the colors assigned to corresponding pixels of the one or more first images.

3. The processor of claim 2 , wherein the one or more first images and the one or more second images are generated using an image encoder network, to encode a prior generated image of the one or more first images, and an image generator network to generate a next generated image of the one or more second images based at least in part upon a vector representation of the one or more second images output by the image encoder network.

4. The processor of claim 3 , wherein the image generator network includes a sequence of translation blocks and upscaling layers, the translation blocks to generate images based on one or more inputs and each including a sequence of translation layers, at least one translation layer of the sequence to receive as input a guidance image generated by projecting the one or more point cloud representations onto a next viewpoint for the next generated image.

5. The processor of claim 4 , wherein at least one translation layer of the sequence of translation layers is to receive as input one or more style embeddings generated by a label embedding network, the label embedding network receiving as input concatenated semantic data, the semantic data including at least one of segmentation data, depth data, pose data, or edge data determined from one or more input images.

6. The processor of claim 4 , wherein at least one translation layer of the sequence of translation layers of the translation blocks is to receive as input one or more flow embeddings generated by a flow embedding network, the flow embedding network receiving as input optical flow data and a prior generated image of the one or more first images.

7. The processor of claim 1 , wherein sequential layers of the one or more neural networks receive inputs based, at least in part, on the one or more point cloud representations.

8. A system comprising:

one or more processors to use one or more neural networks to generate one or more second images based, at least in part, on one or more point cloud representations of one or more first images.

9. The system of claim 8 , wherein the one or more processors are further to assign one or more colors to one or more portions of the one or more point cloud representations based at least in part on the one or more colors assigned to corresponding pixels of the one or more first images.

10. The system of claim 9 , wherein the one or more first images and the one or more second images are generated using an image encoder network, to encode a prior generated image of the one or more first images, and an image generator network to generate a next generated image of the one or more second images based at least in part upon a vector representation of the one or more second images output by the image encoder network.

11. The system of claim 10 , wherein the image generator network includes a sequence of translation blocks and upscaling layers, the translation blocks to generate images based on one or more inputs and each including a sequence of translation layers, at least one translation layer of the sequence to receive as input a guidance image generated by projecting the one or more point cloud representations onto a next viewpoint for the next generated image.

12. The system of claim 11 , wherein at least one translation layer of the sequence of translation layers is to receive as input one or more style embeddings generated by a label embedding network, the label embedding network receiving as input concatenated semantic data, the semantic data including at least one of segmentation data, depth data, pose data, or edge data determined from one or more input images.

13. The system of claim 11 , wherein at least one translation layer of the sequence of translation layers of the translation blocks is to receive as input one or more flow embeddings generated by a flow embedding network, the flow embedding network receiving as input optical flow data and a prior generated image of the one or more first images.

14. A method comprising:

using one or more neural networks to generate one or more second images based, at least in part, on one or more point cloud representations of one or more first images.

15. The method of claim 14 , further comprising: assigning one or more colors to one or more portions of the one or more point cloud representations based at least in part on the one or more colors assigned to corresponding pixels of the one or more first images.

16. The method of claim 15 , wherein the one or more first images and the one or more second images are generated using an image encoder network, to encode a prior generated image of the one or more first images, and an image generator network to generate a next generated image of the one or more second images based at least in part upon a vector representation of the one or more second images output by the image encoder network.

17. The method of claim 16 , wherein the image generator network includes a sequence of translation blocks and upscaling layers, the translation blocks to generate images based on one or more inputs and each including a sequence of translation layers, at least one translation layer of the sequence to receive as input a guidance image generated by projecting the one or more point cloud representations onto a next viewpoint for the next generated image.

18. The method of claim 17 , wherein at least one translation layer of the sequence of translation layers is to receive as input one or more style embeddings generated by a label embedding network, the label embedding network receiving as input concatenated semantic data, the semantic data including at least one of segmentation data, depth data, pose data, or edge data determined from one or more input images.

19. The method of claim 17 , wherein at least one translation layer of the sequence of translation layers of the translation blocks is to receive as input one or more flow embeddings generated by a flow embedding network, the flow embedding network receiving as input optical flow data and a prior generated image of the one or more first images.

20. A non-transitory machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:

use one or more neural networks to generate one or more second images based, at least in part, on one or more point cloud representations of one or more first images.

21. The non-transitory machine-readable medium of claim 20 , wherein the instructions if performed further cause the one or more processors to:

assign one or more colors to one or more portions of the one or more point cloud representations based at least in part on the one or more colors assigned to corresponding pixels of the one or more first images.

22. The non-transitory machine-readable medium of claim 21 , wherein the one or more first images and the one or more second images are generated using an image encoder network, to encode a prior generated image of the one or more first images, and an image generator network to generate a next generated image of the one or more second images based at least in part upon a vector representation of the one or more second images output by the image encoder network.

23. The non-transitory machine-readable medium of claim 22 , wherein the image generator network includes a sequence of translation blocks and upscaling layers, the translation blocks to generate images based on one or more inputs and each including a sequence of translation layers, at least one translation layer of the sequence to receive as input a guidance image generated by projecting the one or more point cloud representations onto a next viewpoint for the next generated image.

24. The non-transitory machine-readable medium of claim 23 , wherein at least one translation layer of the sequence of translation layers is to receive as input one or more style embeddings generated by a label embedding network, the label embedding network receiving as input concatenated semantic data, the semantic data including at least one of segmentation data, depth data, pose data, or edge data determined from one or more input images.

25. The non-transitory machine-readable medium of claim 23 , wherein at least one translation layer of the sequence of translation layers of the translation blocks is to receive as input one or more flow embeddings generated by a flow embedding network, the flow embedding network receiving as input optical flow data and a prior generated image of the one or more first images.

26. A video synthesis system, comprising:

one or more processors to use one or more neural networks to generate one or more second video frames based, at least in part, on one or more point cloud representations of one or more first video frames; and

memory for storing network parameters for the one or more neural networks.

27. The video synthesis system of claim 26 , wherein the one or more processors are further to assign appearance data to one or more portions of the one or more point cloud representations based at least in part on the appearance data assigned to corresponding pixels of the one or more first video frames.

28. The video synthesis system of claim 27 , wherein the one or more first video frames and the one or more second video frames are generated using an image encoder network, to encode a prior generated video frame of the one or more first video frames, and an image generator network to generate a next generated video frame of the one or more second video frames based at least in part upon a vector representation of the one or more second images output by the image encoder network.

29. The video synthesis system of claim 28 , wherein the image generator network includes a sequence of translation blocks and upscaling layers, the translation blocks to generate video frames based on one or more inputs and each including a sequence of translation layers, at least one translation layer of the sequence to receive as input a guidance image generated by projecting the one or more point cloud representations onto a next viewpoint for the next generated video frame.

30. The video synthesis system of claim 29 , wherein at least one translation layer of the sequence of translation layers is to receive as input one or more style embeddings generated by a label embedding network, the label embedding network receiving as input concatenated semantic data, the semantic data including at least one of segmentation data, depth data, pose data, or edge data determined from one or more input video frames.

31. The video synthesis system of claim 29 , wherein at least one translation layer of the sequence of translation layers of the translation blocks is to receive as input one or more flow embeddings generated by a flow embedding network, the flow embedding network receiving as input optical flow data and a prior generated video frame of the one or more first video frames.