Aggregating images and audio data to generate content
The disclosure includes generating a stream of panoramic images. A method includes determining a first matching camera module. The method includes constructing a first camera map that associates a first pixel location in a first panoramic image to the first matching camera module, wherein the first pixel location corresponds to the point in a panorama from a first viewing direction. The method includes generating, based on the first camera map, a stream of first panoramic images.
1. A method comprising:
determining a first matching camera module by:
determining a set of camera modules that have a point in their respective fields of view;
determining a first eye viewing direction from a first eye position to the point;
determining a set of camera viewing directions to the point for the set of camera modules; and
selecting the first matching camera module from the set of camera modules based on the first matching camera module having a first camera viewing direction that is substantially parallel to the first eye viewing direction, wherein the first camera viewing direction is determined to be substantially parallel to the first eye viewing direction based on the first camera viewing direction forming a smaller angle with the first eye viewing direction as compared to other angles formed between the first eye viewing direction and other camera viewing directions associated with other camera modules from the set of camera modules;
constructing a first camera map that associates a first pixel location in a first panoramic image to the first matching camera module, wherein the first pixel location corresponds to the point in a panorama from the first eye viewing direction, wherein the first camera map indicates a weight to be used to blend a pixel value for the first pixel location with a pixel value for a bordering pixel location associated with another camera module in the set of camera modules; and
generating, based on the first camera map, a stream of first panoramic images.
2. The method of claim 1 , further comprising
constructing a second camera map that associates a second pixel location in a second panoramic image to a second matching camera module, wherein the second pixel location corresponds to the point in the panorama from a second eye viewing direction and the second matching camera module is selected based on having a second field of view that includes a second camera viewing direction that is most parallel to the second eye viewing direction as compared to the other camera viewing directions associated with the other camera modules; and
generating, based on the second camera map, a stream of second panoramic images.
3. The method of claim 2 , further comprising generating a stream of three dimensional (3D) video data from the stream of first panoramic images and the stream of second panoramic images.
4. The method of claim 3 , wherein generating the stream of 3D video data comprises:
identifying first matching camera modules for first panoramic images based on a first camera map;
identifying second matching camera modules for second panoramic images based on a second camera map;
stitching first image frames captured by the first matching camera modules at a particular time to form a corresponding first panoramic image in the stream of first panoramic images; and
stitching second image frames captured by the second matching camera modules at a particular time to form a corresponding second panoramic image in the stream of second panoramic images.
5. The method of claim 3 , wherein:
for a pixel with a yaw value and a pitch value in a panorama:
the first camera map identifies a first matching camera module for the pixel in the panorama and matches the pixel in the panorama to a pixel in an image plane of the first matching camera module; and
the second camera map identifies a second matching camera module for the pixel in the panorama and matches the pixel in the panorama to a pixel in an image plane of the second matching camera module.
6. The method of claim 3 , wherein:
the first camera map associates a pixel location in first panoramic images to a corresponding first matching camera module, wherein the pixel location corresponds to a point of a panorama in a first viewing direction;
the corresponding first matching camera module has a field of view that includes a viewing direction to the point of the panorama; and
the viewing direction of the corresponding first matching camera module is closer to the first viewing direction than other viewing directions associated with other camera modules.
7. The method of claim 3 , wherein generating the stream of 3D video data comprises:
determining a current viewing direction associated with a user; and
generating the stream of first panoramic images and the stream of second panoramic images based on the current viewing direction.
8. The method of claim 2 , wherein:
the first eye viewing direction is a right eye viewing direction from a right eye position to the point; and
the second eye viewing direction is a left eye viewing direction from a left eye position to the point.
9. A system comprising:
one or more processors;
one or more non-transitory tangible computer-readable mediums communicatively coupled to the one or more processors and storing executable instructions executable by the one or more processors to perform operations comprising:
determining a first matching camera module by:
determining a set of camera modules that have a point in their respective fields of view;
determining a first eye viewing direction from a first eye position to the point;
determining a set of camera viewing directions to the point for the set of camera modules; and
selecting the first matching camera module from the set of camera modules based on the first matching camera module having a first camera viewing direction that is substantially parallel to the first eye viewing direction, wherein the first camera viewing direction is determined to be substantially parallel to the first eye viewing direction based on the first camera viewing direction forming a smaller angle with the first eye viewing direction as compared to other angles formed between the first eye viewing direction and other camera viewing directions associated with other camera modules from the set of camera modules;
constructing a first camera map that associates a first pixel location in a first panoramic image to the first matching camera module, wherein the first pixel location corresponds to the point in a panorama from the first eye viewing direction, wherein the first camera map indicates a weight to be used to blend a pixel value for the first pixel location with a pixel value for a bordering pixel location associated with another camera module in the set of camera modules; and
generating, based on the first camera map, a stream of first panoramic images.
10. The system of claim 9 , wherein the instructions executable by the one or more processors are executable to cause the processors to perform additional operations including:
constructing a second camera map that associates a second pixel location in a second panoramic image to a second matching camera module, wherein the second pixel location corresponds to the point in the panorama from a second eye viewing direction and the second matching camera module is selected based on having a second field of view that includes a second camera viewing direction that is most parallel to the second eye viewing direction as compared to the other camera viewing directions associated with the other camera modules; and
generating, based on the second camera map, a stream of second panoramic images.
11. The system of claim 10 , wherein the instructions executable by the one or more processors are executable to cause the processors to perform additional operations including generating a stream of three dimensional (3D) video data from the stream of first panoramic images and the stream of second panoramic images.
12. The system of claim 11 , wherein generating the stream of 3D video data comprises:
identifying first matching camera modules for first panoramic images based on a first camera map;
identifying second matching camera modules for second panoramic images based on a second camera map;
stitching first image frames captured by the first matching camera modules at a particular time to form a corresponding first panoramic image in the stream of first panoramic images; and
stitching second image frames captured by the second matching camera modules at a particular time to form a corresponding second panoramic image in the stream of second panoramic images.
13. The system of claim 11 , wherein:
the first camera map associates a pixel location in first panoramic images to a corresponding first matching camera module, wherein the pixel location corresponds to a point of a panorama in a first viewing direction;
the corresponding first matching camera module has a field of view that includes a viewing direction to the point of the panorama; and
the viewing direction of the corresponding first matching camera module is closer to the first viewing direction than other viewing directions associated with other camera modules.
14. The system of claim 11 , wherein:
for a pixel with a yaw value and a pitch value in a panorama:
the first camera map identifies a first matching camera module for the pixel in the panorama and matches the pixel in the panorama to a pixel in an image plane of the first matching camera module; and
the second camera map identifies a second matching camera module for the pixel in the panorama and matches the pixel in the panorama to a pixel in an image plane of the second matching camera module.
15. The system of claim 11 , wherein generating the stream of 3D video data comprises:
determining a current viewing direction associated with a user; and
generating the stream of first panoramic images and the stream of second panoramic images based on the current viewing direction.
16. A computer program product comprising a non-transitory computer-usable medium including a computer-readable program, wherein the computer-readable program when executed on a computer causes the computer to:
determine a first matching camera module by:
determining a set of camera modules that have a point in their respective fields of view;
determining a first eye viewing direction from a first eye position to the point;
determining a set of camera viewing directions to the point for the set of camera modules; and
selecting the first matching camera module from the set of camera modules based on the first matching camera module having a first camera viewing direction that is substantially parallel to the first eye viewing direction, wherein the first camera viewing direction is determined to be substantially parallel to the first eye viewing direction based on the first camera viewing direction forming a smaller angle with the first eye viewing direction as compared to other angles formed between the first eye viewing direction and other camera viewing directions associated with other camera modules from the set of camera modules;
construct a first camera map that associates a first pixel location in a first panoramic image to the first matching camera module, wherein the first pixel location corresponds to the point in a panorama from the first eye viewing direction, wherein the first camera map indicates a weight to be used to blend a pixel value for the first pixel location with a pixel value for a bordering pixel location associated with another camera module in the set of camera modules; and
generate, based on the first camera map, a stream of first panoramic images.
17. The compute program product of claim 16 , wherein computer-readable program when executed on the computer further cause the computer to:
construct a second camera map that associates a second pixel location in a second panoramic image to a second matching camera module, wherein the second pixel location corresponds to the point in the panorama from a second eye viewing direction and the second matching camera module is selected based on having a second field of view that includes a second camera viewing direction that is most parallel to the second eye viewing direction as compared to the other camera viewing directions associated with the other camera modules; and
generate, based on the second camera map, a stream of second panoramic images.
18. The computer program product of claim 17 , wherein the computer-readable program when executed on the computer further cause the computer to generate a stream of three dimensional (3D) video data from the stream of first panoramic images and the stream of second panoramic images.
19. The computer program product of claim 18 , wherein generating the stream of 3D video data comprises:
identifying first matching camera modules for first panoramic images based on a first camera map;
identifying second matching camera modules for second panoramic images based on a second camera map;
stitching first image frames captured by the first matching camera modules at a particular time to form a corresponding first panoramic image in the stream of first panoramic images; and
stitching second image frames captured by the second matching camera modules at a particular time to form a corresponding second panoramic image in the stream of second panoramic images.
20. The computer program product of claim 18 , wherein:
the first camera map associates a pixel location in first panoramic images to a corresponding first matching camera module, wherein the pixel location corresponds to a point of a panorama in a first viewing direction;
the corresponding first matching camera module has a field of view that includes a viewing direction to the point of the panorama; and
the viewing direction of the corresponding first matching camera module is closer to the first viewing direction than other viewing directions associated with other camera modules.