Employing three-dimensional (3D) data predicted from two-dimensional (2D) images using neural networks for 3D modeling applications and other applications
The disclosed subject matter is directed to employing machine learning models configured to predict 3D data from 2D images using deep learning techniques to derive 3D data for the 2D images. In some embodiments, a method is provided that comprises receiving, by a system comprising a processor, a panoramic image, and employing, by the system, a three-dimensional data from two-dimensional data (3D-from-2D) convolutional neural network model to derive three-dimensional data from the panoramic image, wherein the 3D-from-2D convolutional neural network model employs convolutional layers that wrap around the panoramic image as projected on a two-dimensional plane to facilitate deriving the three-dimensional data.
1. A method, comprising:
receiving, by a system comprising a processor, two-dimensional images; and
employing, by the system, a three-dimensional data from two-dimensional data (3D-from-2D) convolutional neural network model to derive three-dimensional data from the two-dimensional images, wherein the 3D-from-2D convolutional neural network model employs convolutional layers that wrap around the two-dimensional images as projected on a two-dimensional plane to facilitate deriving the three-dimensional data, wherein the 3D-from-2D convolutional neural network model was trained based on weighted values applied to respective pixels of projected panoramic images in association with deriving depth data for the respective pixels, wherein the weighted values varied based on an angular area of the respective pixels.
2. The method of claim 1 , wherein the two-dimensional images includes at least one panoramic image.
3. The method of claim 2 , wherein the receiving comprises receiving the panoramic image as a spherical or cylindrical panoramic image, and wherein the method further comprises:
projecting, by the system, the spherical or cylindrical panoramic image on the two-dimensional plane prior to the employing the 3D-from-2D convolutional neural network model to derive the three-dimensional data.
4. The method of claim 2 , wherein the convolutional layers minimize or eliminate edge effects associated with deriving the three-dimensional data based on wrapping around the panoramic image as projected on the two-dimensional plane.
5. The method of claim 1 , wherein the weighted values were decreased as the angular area of the respective pixels decreased.
6. A method comprising:
receiving, by a system comprising a processor, two-dimensional images; and
employing, by the system, a three-dimensional data from two-dimensional data (3D-from-2D) convolutional neural network model to derive three-dimensional data from the two-dimensional images, wherein the 3D-from-2D convolutional neural network model employs convolutional layers that wrap around the two-dimensional images as projected on a two-dimensional plane to facilitate deriving the three-dimensional data, wherein downstream convolutional layers of the convolutional layers that follow a preceding layer are configured to re-project a portion of the two-dimensional images processed by the preceding layer in association with deriving depth data for the two-dimensional images, resulting in generation of a re-projected version of the two-dimensional images for each of the downstream convolutional layers.
7. The method of claim 6 , wherein the downstream convolutional layers are further configured to employ input data from the preceding layer by extracting the input data from the re-projected version of the two-dimensional images.
8. The method of claim 7 , wherein the input data is exacted from the re-projected version of the two-dimensional images based on locations in the portion of the of the two-dimensional images that correspond to a defined angular receptive field based the re-projected version of the two-dimensional images.
9. A non-transitory machine-readable storage medium comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising:
receiving two-dimensional images; and
employing a three-dimensional data from two-dimensional data (3D-from-2D) convolutional neural network model to derive the three-dimensional data from the two-dimensional images, wherein the 3D-from-2D convolutional neural network model employs convolutional layers that wrap around the two-dimensional images as projected on a two-dimensional plane to facilitate deriving the three-dimensional data, wherein the 3D-from-2D convolutional neural network model was trained based on weighted values applied to respective pixels of projected panoramic images in association with deriving depth data for the respective pixels, wherein the weighted values varied based on an angular area of the respective pixels.
10. The non-transitory machine-readable storage medium of claim 9 , wherein the two-dimensional images include at least one panoramic image.
11. The non-transitory machine-readable storage medium of claim 10 , wherein the receiving comprises receiving the panoramic image as a spherical or cylindrical panoramic image, the operations further comprising:
projecting the spherical or cylindrical panoramic image on the two-dimensional plane prior to the employing the 3D-from-2D convolutional neural network model to derive the three-dimensional data.
12. The non-transitory machine-readable storage medium of claim 10 , wherein the convolutional layers minimize or eliminate edge effects associated with deriving the three-dimensional data based on wrapping around the panoramic image as projected on the two-dimensional plane.
13. The non-transitory machine-readable storage medium of claim 9 , wherein the weighted values were decreased as the angular area of the respective pixels decreased.
14. A non-transitory machine-readable storage medium comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising:
receiving two-dimensional images; and
employing a three-dimensional data from two-dimensional data (3D-from-2D) convolutional neural network model to derive the three-dimensional data from the two-dimensional images, wherein the 3D-from-2D convolutional neural network model employs convolutional layers that wrap around the two-dimensional images as projected on a two-dimensional plane to facilitate deriving the three-dimensional data, wherein downstream convolutional layers of the convolutional layers that follow a preceding layer are configured to re-project a portion of the two-dimensional images processed by the preceding layer in association with deriving depth data for the two-dimensional images, resulting in generation of a re-projected version of the two-dimensional images for each of the downstream convolutional layers.
15. The non-transitory machine-readable storage medium of claim 14 , wherein the downstream convolutional layers are further configured to employ input data from the preceding layer by extracting the input data from the re-projected version of the two-dimensional images.