Arbitrary view generation
A machine learning based image processing and generation framework is disclosed. In some embodiments, depth values of an object or asset in a received input image are at least in part determined using a machine learning based framework that is constrained to a known prescribed environment. Determined depth values facilitate generation of other views of the object or asset.
1 . A method, comprising:
receiving an input image of an object or asset, wherein the input image comprises a stereo pair comprising corresponding left and right images;
determining depth values of the object or asset in the input image based at least in part on the left and right stereo pair comprising the input image; and
generating an output image of the object or asset comprising a prescribed perspective that is different than the input image perspective at least in part by performing a perspective transformation of the input image using the determined depth values.
2 . The method of claim 1 , wherein the input image of the object or asset comprises a photograph of the object or asset captured in a known physical environment.
3 . The method of claim 1 , wherein the input image of the object or asset comprises a photograph of the object or asset captured in a prescribed imaging apparatus.
4 . The method of claim 1 , wherein the left and right images comprising the stereo pair are captured by corresponding left and right cameras.
5 . The method of claim 1 , wherein determining depth values comprises one or both of determining a depth estimate and refining the determined depth estimate.
6 . The method of claim 1 , wherein depth values are determined on a per pixel basis.
7 . The method of claim 1 , wherein depth values are determined at least in part using a neural network.
8 . The method of claim 1 , further comprising removing a background of the input image.
9 . The method of claim 1 , wherein the prescribed perspective comprises any desired or requested camera view of the object or asset.
10 . The method of claim 1 , wherein the prescribed perspective comprises an orthographic view.
11 . The method of claim 1 , wherein performing the perspective transformation comprises one or both of determining a perspective transformation estimate and refining the determined perspective transformation estimate.
12 . The method of claim 1 , wherein the perspective transformation is at least in part determined from a mathematical transformation.
13 . The method of claim 1 , wherein the perspective transformation is at least in part determined from a neural network.
14 . The method of claim 1 , wherein determining depth values of the object or asset in the input image is based at least in part on using a machine learning based framework.
15 . The method of claim 14 , wherein the machine learning based framework is constrained to one or more textures.
16 . The method of claim 14 , wherein the machine learning based framework is constrained to a prescribed scene type to which the input image belongs.
17 . The method of claim 14 , wherein the machine learning based framework is constrained to a prescribed environment.
18 . The method of claim 17 , wherein the prescribed environment comprises a physical environment in which the input image is photographed and a model environment that simulates the physical environment for training datasets of the machine learning based framework.
19 . A system, comprising:
a processor configured to:
receive an input image of an object or asset, wherein the input image comprises a stereo pair comprising corresponding left and right images;
determine depth values of the object or asset in the input image based at least in part on the left and right stereo pair comprising the input image; and
generate an output image of the object or asset comprising a prescribed perspective that is different than the input image perspective at least in part by performing a perspective transformation of the input image using the determined depth values; and
a memory coupled to the processor and configured to provide the processor with instructions.
20 . The system of claim 19 , wherein the input image of the object or asset comprises a photograph of the object or asset captured in a known physical environment.
21 . The system of claim 19 , wherein the input image of the object or asset comprises a photograph of the object or asset captured in a prescribed imaging apparatus.
22 . The system of claim 19 , wherein the left and right images comprising the stereo pair are captured by corresponding left and right cameras.
23 . The system of claim 19 , wherein to determine depth values comprises one or both of to determine a depth estimate and to refine the determined depth estimate.
24 . The system of claim 19 , wherein depth values are determined on a per pixel basis.
25 . The system of claim 19 , wherein depth values are determined at least in part using a neural network.
26 . The system of claim 19 , wherein the processor is further configured to remove a background of the input image.
27 . The system of claim 19 , wherein the prescribed perspective comprises any desired or requested camera view of the object or asset.
28 . The system of claim 19 , wherein the prescribed perspective comprises an orthographic view.
29 . The system of claim 19 , wherein performing the perspective transformation comprises one or both of determining a perspective transformation estimate and refining the determined perspective transformation estimate.
30 . The system of claim 19 , wherein the perspective transformation is at least in part determined from a mathematical transformation.
31 . The system of claim 19 , wherein the perspective transformation is at least in part determined from a neural network.
32 . The system of claim 19 , wherein to determine depth values of the object or asset in the input image is based at least in part on using a machine learning based framework.
33 . The system of claim 32 , wherein the machine learning based framework is constrained to one or more textures.
34 . The system of claim 32 , wherein the machine learning based framework is constrained to a prescribed scene type to which the input image belongs.
35 . The system of claim 32 , wherein the machine learning based framework is constrained to a prescribed environment.
36 . The system of claim 35 , wherein the prescribed environment comprises a physical environment in which the input image is photographed and a model environment that simulates the physical environment for training datasets of the machine learning based framework.
37 . A computer program product embodied in a non-transitory computer readable storage medium comprising computer instructions which when executed cause a computer to:
receive an input image of an object or asset, wherein the input image comprises a stereo pair comprising corresponding left and right images;
determine depth values of the object or asset in the input image based at least in part on the left and right stereo pair comprising the input image; and
generate an output image of the object or asset comprising a prescribed perspective that is different than the input image perspective at least in part by performing a perspective transformation of the input image using the determined depth values.
38 . The computer program product of claim 37 , wherein the input image of the object or asset comprises a photograph of the object or asset captured in a known physical environment.
39 . The computer program product of claim 37 , wherein the input image of the object or asset comprises a photograph of the object or asset captured in a prescribed imaging apparatus.
40 . The computer program product of claim 37 , wherein the left and right images comprising the stereo pair are captured by corresponding left and right cameras.
41 . The computer program product of claim 37 , wherein determining depth values comprises one or both of determining a depth estimate and refining the determined depth estimate.
42 . The computer program product of claim 37 , wherein depth values are determined on a per pixel basis.
43 . The computer program product of claim 37 , wherein depth values are determined at least in part using a neural network.
44 . The computer program product of claim 37 , further comprising computer instructions for removing a background of the input image.
45 . The computer program product of claim 37 , wherein the prescribed perspective comprises any desired or requested camera view of the object or asset.
46 . The computer program product of claim 37 , wherein the prescribed perspective comprises an orthographic view.
47 . The computer program product of claim 37 , wherein performing the perspective transformation comprises one or both of determining a perspective transformation estimate and refining the determined perspective transformation estimate.
48 . The computer program product of claim 20 , wherein the perspective transformation is at least in part determined from a mathematical transformation.
49 . The computer program product of claim 20 , wherein the perspective transformation is at least in part determined from a neural network.
50 . The computer program product of claim 37 , wherein determining depth values of the object or asset in the input image is based at least in part on using a machine learning based framework.
51 . The computer program product of claim 50 , wherein the machine learning based framework is constrained to one or more textures.
52 . The computer program product of claim 50 , wherein the machine learning based framework is constrained to a prescribed scene type to which the input image belongs.
53 . The computer program product of claim 50 , wherein the machine learning based framework is constrained to a prescribed environment.
54 . The computer program product of claim 53 , wherein the prescribed environment comprises a physical environment in which the input image is photographed and a model environment that simulates the physical environment for training datasets of the machine learning based framework.