Panorama generation using one or more neural networks
Apparatuses, systems, and techniques are presented to generate panoramas from a set of images. In at least one embodiment, one or more generative neural networks are used to generate a spherical panoramic image using features extracted from input images.
1 . One or more processors, comprising: circuitry to:
use one or more neural networks to:
generate encodings of one or more first features based, at least in part, on one or more second features extracted from a plurality of images of an environment captured from a same position in the environment facing different directions;
generate a spherical panoramic image based, at least in part, on the encodings; and
cause the spherical panoramic image to be presented using a display of a virtual reality headset.
2 . The one or more processors of claim 1 , wherein the circuitry is further to extract representative features from at least six images of the plurality of images, the representative features being provided as input to the one or more neural networks.
3 . The one or more processors of claim 2 , wherein the one or more neural networks include one or more generative adversarial networks (GANs) or variational autoencoders (VAEs) for generating the spherical panoramic image based at least in part on the representative features.
4 . The one or more processors of claim 2 , wherein the representative features are used to generate a cube map that is transformed into the spherical panoramic image.
5 . The one or more processors of claim 1 , wherein gaps in view directions of the plurality of images are filled with content generated by the one or more neural networks.
6 . The one or more processors of claim 1 , wherein the circuitry is further to perform post-processing of the spherical panoramic image to cause the spherical panoramic image to be in a format for a specified use.
7 . A system comprising:
one or more processors to:
use one or more neural networks to:
generate encodings of one or more first features based, at least in part, on one or more second features extracted from a plurality of images of an environment captured from a same position in the environment facing different directions;
generate a spherical panoramic image based, at least in part, on the encodings; and
cause the spherical panoramic images to be presented using a display of a virtual reality headset.
8 . The system of claim 7 , wherein the one or more processors are further to extract representative features from at least six images of the plurality of images, the representative features being provided as input to the one or more neural networks.
9 . The system of claim 8 , wherein the one or more neural networks include one or more generative adversarial networks (GANs) or variational autoencoders (VAEs) for generating the spherical panoramic image using the representative features.
10 . The system of claim 8 , wherein the representative features are used to generate a cube map that is transformed into the spherical panoramic image.
11 . The system of claim 7 , wherein gaps in view directions of the plurality of images are filled with content generated by the one or more neural networks.
12 . The system of claim 7 , wherein the one or more processors are further to perform post-processing of the spherical panoramic image to cause the spherical panoramic image to be in a format for a specified use.
13 . A method comprising:
using one or more neural networks to:
generate encodings of one or more first features based, at least in part, on one or more second features extracted from a plurality of images of an environment captured from a same position in the environment facing different directions;
generate a spherical panoramic image based, at least in part, on the encodings; and
causing the spherical image to be presented using a display of a virtual reality headset.
14 . The method of claim 13 , further comprising:
extracting representative features from at least six images of the plurality of images, the representative features being provided as input to the one or more neural networks.
15 . The method of claim 14 , wherein the one or more neural networks include one or more generative adversarial networks (GANs) or variational autoencoders (VAEs) for generating the spherical panoramic image using the representative features.
16 . The method of claim 14 , wherein the representative features are used to generate a cube map that is transformed into the spherical panoramic image.
17 . The method of claim 13 , wherein gaps in view directions of the plurality of images are filled with content generated by the one or more neural networks.
18 . The method of claim 13 , further comprising: performing post-processing of the spherical panoramic image to cause the spherical panoramic image to be in a format for a specified use.
19 . A non-transitory machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:
use one or more neural networks to:
generate encodings of one or more first features based, at least in part, on one or more second features extracts from a plurality of images of an environment captured from a same positions in the environment facing different directions;
generate a spherical panoramic image based, at least in part, on the encodings; and
cause the spherical image to be presented using a display of a virtual reality headset.
20 . The non-transitory machine-readable medium of claim 19 , wherein the instructions if performed further cause the one or more processors to:
extract representative features from at least six images of the plurality of images, the representative features being provided as input to the one or more neural networks.
21 . The non-transitory machine-readable medium of claim 20 , wherein the one or more neural networks include one or more generative adversarial networks (GANs) or variational autoencoders (VAEs) for generating the spherical panoramic image using the representative features.
22 . The non-transitory machine-readable medium of claim 20 , wherein the representative features are used to generate a cube map that is transformed into the spherical panoramic image.
23 . The non-transitory machine-readable medium of claim 19 , wherein gaps in view directions of the plurality of images are filled with content generated by the one or more neural networks.
24 . The non-transitory machine-readable medium of claim 19 , wherein the instructions if performed further cause the one or more processors to:
perform post-processing of the spherical panoramic image to cause the spherical panoramic image to be in a format for a specified use.
25 . A panorama generation system, comprising:
one or more processors to:
use one or more neural networks to:
generate encodings of one or more first features based, at least in part, on one or more second features extracted from a plurality of images of an environment captured from a same position in the environment facing different directions;
generate a spherical panoramic image based, at least in part, on the encodings; and
cause the spherical image to be presented using a display of a virtual reality headset.
26 . The panorama generation system of claim 25 , wherein the one or more processors are further to extract representative features from at least six images of the plurality of images, the representative features being provided as input to the one or more neural networks.
27 . The panorama generation system of claim 26 , wherein the one or more neural networks include one or more generative adversarial networks (GANs) or variational autoencoders (VAEs) for generating the spherical panoramic image using the representative features.
28 . The panorama generation system of claim 26 , wherein the representative features are used to generate a cube map that is transformed into the spherical panoramic image.
29 . The panorama generation system of claim 25 , wherein gaps in view directions of the plurality of images are filled with content generated by the one or more neural networks.
30 . The panorama generation system of claim 25 , wherein the one or more processors are further to perform post-processing of the spherical panoramic image to cause the spherical panoramic image to be in a format for a specified use.