Stereo-aware panorama conversion for immersive media
Techniques are disclosed for conversion of panoramas between different panoramic projections, including stereoscopic panoramic projections. The pixels of one panoramic image are mapped to the pixels of another panoramic image using ray tracing so that the images can be combined without introducing visual distortions caused by mismatches in the optics of each image. The conversion is performed using a process for mapping one or more pixels of an output image to pixels of an input image by tracing a ray between the UV coordinates corresponding to each pixel in the images. The conversion process accounts for arbitrary projections and for arbitrary stereo camera rig configurations by using metadata that describe the projection and positioning of each image being combined. The origin of each ray is offset to account for the interpupillary distance between left and right images and to account for any rotation of the scene.
1. In a digital medium environment for editing digital images, a computer-implemented method for generating a panoramic image from different panoramic projections, the method comprising:
calculating, by at least one processor and for each of a plurality of pixels in an output image, Euler angle coordinates at first UV coordinates corresponding to the respective pixel, the first UV coordinates representing a two-dimensional texture of a first three-dimensional surface associated with the respective pixel in the output image;
mapping, by the at least one processor and for each of the pixels in the output image, the Euler angle coordinates to a geometric ray;
calculating, by the at least one processor and for each geometric ray, second UV coordinates corresponding to an intersection between the respective geometric ray and a pixel of an input image, the second UV coordinates representing a two-dimensional texture of a second three-dimensional surface associated with the respective pixel in the input image, wherein the first three-dimensional surface of the output image is different from the second three-dimensional surface of the input image;
converting, by the at least one processor and for each intersection, the second UV coordinates at the respective intersection to one of the pixels in the input image; and
generating, by the at least one processor and for each intersection, a combined panoramic image, metadata descriptive of the combined panoramic image, or both, the generating being based on the converted pixel in the input image, the combined panoramic image representing a combination of the output image and the input image, wherein the combined panoramic image retains at least a portion of content of the output image and at least a portion of content of the input image.
2. The method of claim 1 , further comprising determining, by the at least one processor and for each pixel in an output image, whether the respective pixel is within a left eye sub-window of the output image or within a right eye sub-window of the output image.
3. The method of claim 2 , wherein the ray has an origin in a coordinate system based at least in part on the determination of whether the respective pixel is within the left eye sub-window of the output image or within the right eye sub-window of the output image.
4. The method of claim 3 , wherein mapping the Euler angle coordinates to the ray includes offsetting the ray from the origin of the coordinate system based on an interpupillary distance associated with the output image.
5. The method of claim 1 , wherein at least one of the output image and the input image represents a stereoscopic panoramic projection.
6. The method of claim 1 , wherein the output image represents a first panoramic projection, wherein the input image represents a second panoramic projection, and wherein optics of the first panoramic projection and optics of the second panoramic projection are mismatched.
7. The method of claim 1 , wherein at least one of the output image and the input image represents a monoscopic panoramic projection.
8. A computer program product including one or more non-transitory machine-readable mediums having instructions encoded thereon that when executed by one or more processors cause a process to be carried out for generating a panoramic image from different panoramic projections, the process comprising:
calculating, for each of a plurality of pixels in an output image, Euler angle coordinates at first UV coordinates corresponding to the respective pixel, the first UV coordinates representing a two-dimensional texture of a first three-dimensional surface associated with the respective pixel in the output image;
mapping, for each of the pixels in the output image, the Euler angle coordinates to a geometric ray;
calculating, for each geometric ray, second UV coordinates corresponding to an intersection between the respective geometric ray and a pixel of an input image, the second UV coordinates representing a two-dimensional texture of a second three-dimensional surface associated with the respective pixel in the input image, wherein the first three-dimensional surface of the output image is different from the second three-dimensional surface of the input image;
converting, for each intersection, the UV coordinates at the respective intersection to one of the pixels in the input image; and
generating, for each intersection, a combined panoramic image, metadata descriptive of the combined panoramic image, or both, the generating being based on the converted pixel in the input image, the combined panoramic image representing a combination of the output image and the input image, wherein the combined panoramic image retains at least a portion of content of the output image and at least a portion of content of the input image.
9. The computer program product of claim 8 , wherein the process further includes determining, for each pixel in an output image, whether the respective pixel is within a left eye sub-window of the output image or within a right eye sub-window of the output image.
10. The computer program product of claim 9 , wherein the ray has an origin in a coordinate system based at least in part on the determination of whether the respective pixel is within the left eye sub-window of the output image or within the right eye sub-window of the output image.
11. The computer program product of claim 10 , wherein mapping the Euler angle coordinates to the ray includes offsetting the ray from the origin of the coordinate system based on an interpupillary distance associated with the output image.
12. The computer program product of claim 8 , wherein at least one of the output image and the input image represents a stereoscopic panoramic projection.
13. The computer program product of claim 8 , wherein the output image represents a first panoramic projection, wherein the input image represents a second panoramic projection, and wherein optics of the first panoramic projection and optics of the second panoramic projection are mismatched.
14. The computer program product of claim 8 , wherein at least one of the output image and the input image represents a monoscopic panoramic projection.
15. A system for generating a panoramic image from different panoramic projections, the system comprising:
at least one processor; and
a storage operatively coupled to the at least one processor and for storing instructions that when executed by the at least one processor cause the at least one processor to
cause a pixel-to-UV module to obtain, for each of a plurality of pixels in an output image, first UV coordinates corresponding to the respective pixel in the output image based on metadata associated with the output image, wherein the first UV coordinates represent a two-dimensional texture of a first three-dimensional surface associated with the respective pixel in the output image;
cause a Euler angle at UV module to calculate, for each of the pixels in the output image, Euler angle coordinates at the respective UV coordinates based on the metadata associated with the output image;
cause a ray at Euler angle module to map, for each of the pixels in the output image, the Euler angle coordinates to a geometric ray;
cause a UV on ray module to calculate, for each geometric ray, an intersection between the respective geometric ray and a pixel of an input image, wherein the intersection represents second UV coordinates corresponding to one of a plurality of pixels in an input image, the second UV coordinates representing a two-dimensional texture of a second three-dimensional surface associated with the respective pixel in the input image, wherein the first three-dimensional surface of the output image is different from the second three-dimensional surface of the input image;
cause a UV-to-pixel module to convert, for each intersection, the UV coordinates at the respective intersection to one of the pixels in the input image based on metadata associated with the input image; and
generate, by the at least one processor and for each intersection, a combined panoramic image, metadata descriptive of the combined panoramic image, or both, the generating being based on the converted pixel in the input image, the combined panoramic image representing a combination of the output image and the input image, wherein the combined panoramic image retains at least a portion of content of the output image and at least a portion of content of the input image.
16. The system of claim 15 , wherein the instructions when executed by the at least one processor further cause the at least one processor to cause a sub-window module to determine, for each pixel in an output image, whether the respective pixel is within a left eye sub-window of the output image or within a right eye sub-window of the output image.
17. The system of claim 16 , wherein the ray has an origin in a coordinate system based at least in part on the determination of whether the respective pixel is within the left eye sub-window of the output image or within the right eye sub-window of the output image.
18. The system of claim 17 , wherein mapping the Euler angle coordinates to the ray includes offsetting the ray from the origin of the coordinate system based on an interpupillary distance associated with the output image.
19. The system of claim 15 , wherein at least one of the output image and the input image represents a stereoscopic panoramic projection.
20. The system of claim 15 , wherein the output image represents a first panoramic projection, wherein the input image represents a second panoramic projection, and wherein optics of the first panoramic projection and optics of the second panoramic projection are mismatched.