Arbitrary view generation
A machine learning based image processing and generation framework is disclosed. In some embodiments, depth values of an object or asset in a received input image are at least in part determined using a machine learning based framework that is constrained to a known prescribed environment. Determined depth values facilitate generation of other views of the object or asset.
1. A method, comprising:
receiving an input image of an object or asset; and
determining depth values of the object or asset in the input image;
wherein depth values are at least in part determined using a machine learning based framework that is constrained to a known prescribed environment and wherein determined depth values facilitate generation of other views of the object or asset.
2. The method of claim 1 , wherein the known prescribed environment comprises a physical environment in which the input image is photographed and a model environment that simulates the physical environment for training datasets of the machine learning based framework.
3. The method of claim 1 , wherein the machine learning based framework is constrained to one or more known textures.
4. The method of claim 1 , wherein the input image comprises a camera captured photograph of the object or asset.
5. The method of claim 1 , wherein the input image comprises a plurality of images from different cameras or camera angles.
6. The method of claim 1 , wherein depth values are determined on a per pixel basis.
7. The method of claim 1 , wherein determining depth values comprises one or both of determining a depth estimate and refining the determined depth estimate.
8. The method of claim 7 , wherein the depth estimate is predicted using a neural network.
9. The method of claim 7 , wherein the depth estimate is determined from a left and right stereo pair comprising the input image.
10. The method of claim 7 , wherein the determined depth estimate is refined using a neural network.
11. The method of claim 1 , further comprising removing a background of the input image based at least in part on neural network based image segmentation.
12. The method of claim 1 , further comprising generating an output image comprising a prescribed perspective of the object or asset that is different than the input image perspective by performing a perspective transformation based on determined depth values.
13. The method of claim 12 , wherein the prescribed perspective comprises an orthographic view.
14. The method of claim 12 , wherein performing the perspective transformation comprises one or both of determining a perspective transformation estimate and refining the determined perspective transformation estimate.
15. The method of claim 14 , wherein the perspective transformation estimate is determined from a mathematical transformation.
16. The method of claim 14 , wherein the perspective transformation estimate is predicted using a neural network.
17. The method of claim 14 , wherein the perspective transformation estimate is refined using a neural network.
18. The method of claim 1 , wherein the machine learning based framework comprises a generative adversarial network (GAN).
19. A system, comprising:
a processor configured to:
receive an input image of an object or asset; and
determine depth values of the object or asset in the input image, wherein depth values are at least in part determined using a machine learning based framework that is constrained to a known prescribed environment and wherein determined depth values facilitate generation of other views of the object or asset; and
a memory coupled to the processor and configured to provide the processor with instructions.
20. The system of claim 19 , wherein the known prescribed environment comprises a physical environment in which the input image is photographed and a model environment that simulates the physical environment for training datasets of the machine learning based framework.
21. The system of claim 19 , wherein the machine learning based framework is constrained to one or more known textures.
22. The system of claim 19 , wherein the input image comprises a camera captured photograph of the object or asset.
23. The system of claim 19 , wherein the input image comprises a plurality of images from different cameras or camera angles.
24. The system of claim 19 , wherein depth values are determined on a per pixel basis.
25. The system of claim 19 , wherein to determine depth values comprises one or both of to determine a depth estimate and to refine the determined depth estimate.
26. The system of claim 25 , wherein the depth estimate is predicted using a neural network.
27. The system of claim 25 , wherein the depth estimate is determined from a left and right stereo pair comprising the input image.
28. The system of claim 25 , wherein the determined depth estimate is refined using a neural network.
29. The system of claim 19 , wherein the processor is further configured to remove a background of the input image based at least in part on neural network based image segmentation.
30. The system of claim 19 , wherein the processor is further configured to generate an output image comprising a prescribed perspective of the object or asset that is different than the input image perspective by performing a perspective transformation based on determined depth values.
31. The system of claim 30 , wherein the prescribed perspective comprises an orthographic view.
32. The system of claim 30 , wherein performing the perspective transformation comprises one or both of determining a perspective transformation estimate and refining the determined perspective transformation estimate.
33. The system of claim 32 , wherein the perspective transformation estimate is determined from a mathematical transformation.
34. The system of claim 32 , wherein the perspective transformation estimate is predicted using a neural network.
35. The system of claim 32 , wherein the perspective transformation estimate is refined using a neural network.
36. The system of claim 19 , wherein the machine learning based framework comprises a generative adversarial network (GAN).
37. A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions which when executed cause a computer to:
receive an input image of an object or asset; and
determine depth values of the object or asset in the input image;
wherein depth values are at least in part determined using a machine learning based framework that is constrained to a known prescribed environment and wherein determined depth values facilitate generation of other views of the object or asset.
38. The computer program product of claim 37 , wherein the known prescribed environment comprises a physical environment in which the input image is photographed and a model environment that simulates the physical environment for training datasets of the machine learning based framework.
39. The computer program product of claim 37 , wherein the machine learning based framework is constrained to one or more known textures.
40. The computer program product of claim 37 , wherein the input image comprises a camera captured photograph of the object or asset.
41. The computer program product of claim 37 , wherein the input image comprises a plurality of images from different cameras or camera angles.
42. The computer program product of claim 37 , wherein depth values are determined on a per pixel basis.
43. The computer program product of claim 37 , wherein to determine depth values comprises one or both of to determine a depth estimate and to refine the determined depth estimate.
44. The computer program product of claim 43 , wherein the depth estimate is predicted using a neural network.
45. The computer program product of claim 43 , wherein the depth estimate is determined from a left and right stereo pair comprising the input image.
46. The computer program product of claim 43 , wherein the determined depth estimate is refined using a neural network.
47. The computer program product of claim 37 , further comprising computer instructions which when executed cause a computer to remove a background of the input image based at least in part on neural network based image segmentation.
48. The computer program product of claim 37 , further comprising computer instructions which when executed cause a computer to generate an output image comprising a prescribed perspective of the object or asset that is different than the input image perspective by performing a perspective transformation based on determined depth values.
49. The computer program product of claim 48 , wherein the prescribed perspective comprises an orthographic view.
50. The computer program product of claim 48 , wherein performing the perspective transformation comprises one or both of determining a perspective transformation estimate and refining the determined perspective transformation estimate.
51. The computer program product of claim 50 , wherein the perspective transformation estimate is determined from a mathematical transformation.
52. The computer program product of claim 50 , wherein the perspective transformation estimate is predicted using a neural network.
53. The computer program product of claim 50 , wherein the perspective transformation estimate is refined using a neural network.
54. The computer program product of claim 37 , wherein the machine learning based framework comprises a generative adversarial network (GAN).