Employing three-dimensional (3D) data predicted from two-dimensional (2D) images using neural networks for 3D modeling applications and other applications
The disclosed subject matter is directed to employing machine learning models configured to predict 3D data from 2D images using deep learning techniques to derive 3D data for the 2D images. In some embodiments, a method is provided that comprises employing, by a system comprising a processor, one or more three-dimensional data from two-dimensional data (3D-from-2D) neural network models to derive three-dimensional data from one or more two-dimensional images captured of an object or environment from a current perspective of the object or environment viewed on or through a display of the device. The method further comprises, determining, by the system, a position for integrating a graphical data object on or within a representation of the object or environment viewed on or through the display based on the current perspective and the three-dimensional data.
1. A system comprising:
a memory that stores computer executable components; and
a processor that executes the computer executable components stored in the memory, wherein the computer executable components comprise:
a three-dimensional data derivation component configured to employ one or more three-dimensional data from two-dimensional data (3D-from-2D) neural network models to derive three-dimensional data from one or more two-dimensional images captured of an object or environment from a current perspective of the object or environment viewed on or through a display of a device, the one or more 3D-from-2D neural network models are trained based on weighted values applied to respective pixels of the one or more two-dimensional images in association with depth data from the respective pixels, the weighted values being varied based on an angular area of the respective pixels; and
a spatial alignment component configured to determine a position for integrating a graphical data object on or within a representation of the object or environment viewed on or through the display based on the current perspective and the three-dimensional data.
2. The system of claim 1 , wherein the computer executable components further comprise:
an integration component to integrate the graphical data object on or within the representation of the object or environment based on the position.
3. The system of claim 2 , wherein the spatial alignment component further comprises an occlusion mapping component configured to determine a relative position of the graphical data object to another object included in the representation of the object or environment based on the current perspective and the three-dimensional data.
4. The system of claim 3 , wherein based on a determination that the relative position of the graphical data object is behind the other object, the integration component is configured to occlude at least a portion of the graphical data object located behind the other object in association with integrating the graphical data object on or within the representation of the object or environment.
5. The system of claim 3 , wherein based on a determination that the relative position of the graphical data object is in front of the other object, the integration component is configured to occlude at least a portion of the other object located behind the graphical data object in association with integrating the graphical data object on or within the representation of the environment.
6. The system of claim 2 , wherein the representation of the object or environment comprises a two-dimensional image.
7. The system of claim 6 , wherein the graphical data object comprises a three-dimensional object.
8. The system of claim 2 , wherein the representation of the object or environment comprises a live view of the environment viewed through a transparent display, and wherein the integration component facilitates rendering the graphical data object on the display.
9. The system of claim 2 , wherein the representation of the object or environment comprises a video.
10. The system of claim 1 , further comprising:
an object segmentation component configured to extract object image data of an object included in a two-dimensional image, and wherein three-dimensional data derivation component is further configured to employ the one or more 3D-from-2D neural network models to derive object three-dimensional data from the object image data.
11. The system of claim 10 , wherein the graphical data object comprises the object image data, and wherein the spatial alignment component is further configured to determine the position for integrating the object image data on or within the representation of the object based on the object three-dimensional data.
12. A method, comprising:
employing, by a system comprising a processor, one or more three-dimensional data from two-dimensional data (3D-from-2D) neural network models to derive three-dimensional data from one or more two-dimensional images captured of an object or environment from a current perspective of the object or environment viewed on or through a display of a device, the 3D-from-2D neural network models are trained based on weighted values applied to respective pixels of the one or more two-dimensional images in association with depth data from the respective pixels, the weighted values being varied based on an angular area of the respective pixels; and
determining, by the system, a position for integrating a graphical data object on or within a representation of the object or environment viewed on or through the display based on the current perspective and the three-dimensional data.
13. The method of claim 12 , further comprising:
integrating, by the system, the graphical data object on or within the representation of the object or environment based on the position.
14. The method of claim 13 , further comprising:
determining, by the system, a relative position of the graphical data object to another object included in the representation of the object or environment based on the current perspective and the three-dimensional data.
15. The method of claim 14 , wherein based on a determination that the relative position of the graphical data object is behind the other object, the method further comprises:
occluding, by the system, at least a portion of the graphical data object located behind the other object in association with integrating the graphical data object on or within the representation of the object or environment.
16. The method of claim 14 , wherein based on a determination that the relative position of the graphical data object is behind the other object, the method further comprises:
occluding, by the system, at least a portion of the other object located behind the graphical data object in association with integrating the graphical data object on or within the representation of the environment.
17. The method of claim 13 , wherein the representation of the object or environment comprises a two-dimensional image or a video.
18. The method of claim 13 , wherein the representation of the object or environment comprises a live view of the environment viewed through a transparent display, and wherein the integrating comprises rendering the graphical data object on the display.
19. A non-transitory machine-readable storage medium comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising:
employing one or more three-dimensional data from two-dimensional data (3D-from-2D) neural network models to derive three-dimensional data from one or more two-dimensional images captured of an object or environment from a current perspective of the object or environment viewed on or through a display of a device, the 3D-from-2D neural network models are trained based on weighted values applied to respective pixels of the one or more two-dimensional images in association with depth data from the respective pixels, the weighted values being varied based on an angular area of the respective pixels;
determining a position for integrating a graphical data object on or within a representation of the object or environment viewed on or through the display based on the current perspective and the three-dimensional data; and
integrating the graphical data object on or within the representation of the object or environment based on the position.
20. The non-transitory machine-readable storage medium of claim 19 , the operations further comprising:
determining, a relative position of the graphical data object to another object included in the representation of the object or environment based on the current perspective and the three-dimensional data;
based on a determination that the relative position of the graphical data object is behind the other object, occluding least a portion of the graphical data object located behind the other object in association with integrating the graphical data object on or within the representation of the object or environment; and
based on a determination that the relative position of the graphical data object is behind the other object, occluding at least a portion of the other object located behind the graphical data object in association with integrating the graphical data object on or within the representation of the environment.