Arbitrary view generation
Techniques for generating an image are disclosed. In some embodiments, a received input image is transformed to generate an output image using a machine learning based framework that is trained on a constrained set of images. The generated output image comprises an attribute learned by the machine learning based framework from the set of images.
1. A method, comprising:
receiving an input image; and
transforming the input image to generate an output image using a machine learning based framework that is trained on a constrained set of images;
wherein the generated output image comprises an attribute learned by the machine learning based framework from the set of images.
2. The method of claim 1 , further comprising determining or predicting the attribute for the input image using the machine learning based framework.
3. The method of claim 2 , wherein the attribute is unknown or incomplete in the input image.
4. The method of claim 1 , wherein the set of images comprises photorealistic renderings.
5. The method of claim 1 , wherein the set of images comprises a prescribed texture.
6. The method of claim 1 , wherein the set of images is constrained to a prescribed environment.
7. The method of claim 1 , wherein the output image comprises a reference image that is used to generate other images.
8. The method of claim 1 , wherein the attribute comprises a metadata value.
9. The method of claim 1 , wherein the attribute comprises a texture value.
10. The method of claim 1 , wherein the attribute comprises a depth value.
11. The method of claim 1 , where the attribute comprises a surface normal vector value.
12. The method of claim 1 , wherein the attribute comprises coordinate values.
13. The method of claim 1 , wherein the machine learning based framework comprises a neural network.
14. The method of claim 1 , wherein the output image comprises a restored version of the input image.
15. The method of claim 1 , wherein the output image comprises an upscaled version of the input image.
16. The method of claim 1 , wherein the output image comprises a cleaned version of the input image.
17. The method of claim 1 , wherein the output image comprises a denoised version of the input image.
18. The method of claim 1 , wherein transforming comprises one or more of: removing a background of the input image, predicting a depth estimate of the input image, refining a depth estimate of the input image, predicting a perspective transformation estimate of the input image, and refining a perspective transformation estimate of the input image.
19. A system, comprising:
a processor configured to:
receive an input image; and
transform the input image to generate an output image using a machine learning based framework that is trained on a constrained set of images, wherein the generated output image comprises an attribute learned by the machine learning based framework from the set of images; and
a memory coupled to the processor and configured to provide the processor with instructions.
20. The system of claim 19 , wherein the processor is further configured to determine or predict the attribute for the input image using the machine learning based framework.
21. The system of claim 20 , wherein the attribute is unknown or incomplete in the input image.
22. The system of claim 19 , wherein the set of images comprises photorealistic renderings.
23. The system of claim 19 , wherein the set of images comprises a prescribed texture.
24. The system of claim 19 , wherein the set of images is constrained to a prescribed environment.
25. The system of claim 19 , wherein the output image comprises a reference image that is used to generate other images.
26. The system of claim 19 , wherein the attribute comprises a metadata value.
27. The system of claim 19 , wherein the attribute comprises a texture value.
28. The system of claim 19 , wherein the attribute comprises a depth value.
29. The system of claim 19 , where the attribute comprises a surface normal vector value.
30. The system of claim 19 , wherein the attribute comprises coordinate values.
31. The system of claim 19 , wherein the machine learning based framework comprises a neural network.
32. The system of claim 19 , wherein the output image comprises a restored version of the input image.
33. The system of claim 19 , wherein the output image comprises an upscaled version of the input image.
34. The system of claim 19 , wherein the output image comprises a cleaned version of the input image.
35. The system of claim 19 , wherein the output image comprises a denoised version of the input image.
36. The system of claim 19 , wherein to transform comprises one or more of: to remove a background of the input image, to predict a depth estimate of the input image, to refine a depth estimate of the input image, to predict a perspective transformation estimate of the input image, and to refine a perspective transformation estimate of the input image.
37. A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions which when executed cause a computer to:
receive an input image; and
transform the input image to generate an output image using a machine learning based framework that is trained on a constrained set of images;
wherein the generated output image comprises an attribute learned by the machine learning based framework from the set of images.
38. The computer program product of claim 37 , further comprising computer instructions which when executed cause a computer to determine or predict the attribute for the input image using the machine learning based framework.
39. The computer program product of claim 38 , wherein the attribute is unknown or incomplete in the input image.
40. The computer program product of claim 37 , wherein the set of images comprises photorealistic renderings.
41. The computer program product of claim 37 , wherein the set of images comprises a prescribed texture.
42. The computer program product of claim 37 , wherein the set of images is constrained to a prescribed environment.
43. The computer program product of claim 37 , wherein the output image comprises a reference image that is used to generate other images.
44. The computer program product of claim 37 , wherein the attribute comprises a metadata value.
45. The computer program product of claim 37 , wherein the attribute comprises a texture value.
46. The computer program product of claim 37 , wherein the attribute comprises a depth value.
47. The computer program product of claim 37 , where the attribute comprises a surface normal vector value.
48. The computer program product of claim 37 , wherein the attribute comprises coordinate values.
49. The computer program product of claim 37 , wherein the machine learning based framework comprises a neural network.
50. The computer program product of claim 37 , wherein the output image comprises a restored version of the input image.
51. The computer program product of claim 37 , wherein the output image comprises an upscaled version of the input image.
52. The computer program product of claim 37 , wherein the output image comprises a cleaned version of the input image.
53. The computer program product of claim 37 , wherein the output image comprises a denoised version of the input image.
54. The computer program product of claim 37 , wherein to transform comprises one or more of: to remove a background of the input image, to predict a depth estimate of the input image, to refine a depth estimate of the input image, to predict a perspective transformation estimate of the input image, and to refine a perspective transformation estimate of the input image.