Stereo image generation and interactive playback

1. A method for preparing video data of an environment for stereoscopic presentation to a viewer in an experience selected from virtual reality and augmented reality experiences, the method comprising:

at a processor, designating a plurality of locations, distributed throughout a viewing volume, at which a plurality of vantages are to be positioned to facilitate viewing of the environment from proximate the locations in the experience;

at the processor, for each location of the plurality of the locations:

retrieving a plurality of images of the environment captured from viewpoints proximate the location; and

combining the images to generate a combined image to generate a vantage, the combined image comprising a plurality of subject pixels; and

at a data store, storing each of the vantages such that the vantages can be used to generate stereoscopic viewpoint video of the environment, as viewed from at least two virtual viewpoints corresponding to viewpoints of an actual viewer's eyes within the viewing volume, further comprising, at the processor:

receiving designations of the at least two virtual viewpoints; and

using the vantages to generate the stereoscopic viewpoint video.

2. The method of claim 1 , wherein using the vantages to generate the stereoscopic viewpoint video comprises, for each of the virtual viewpoints, combining the combined images from a plurality of the vantages.

3. The method of claim 2 , wherein combining the combined images from a plurality of the vantages comprises:

assigning a fitness value to each candidate pixel of each of the combined images that corresponds to the subject pixel; and

setting the subject pixel to be the same as the candidate pixel that corresponds to the subject pixel and has the highest fitness value;

wherein the fitness value for each of the candidate pixels represents a degree to which it is desirable to include the subject pixel in the experience.

4. The method of claim 2 , wherein combining the combined images from a plurality of the vantages comprises, for each subject pixel of the stereoscopic viewpoint video, using linear interpolation to select color and/or depth data for the subject pixel based on corresponding color and/or depth data for each candidate pixel of each of the combined images that corresponds to the subject pixel.

5. The method of claim 2 , wherein using the vantages to generate the stereoscopic viewpoint video comprises, for each of the virtual viewpoints:

identifying a subset of the vantages that are proximate the virtual viewpoint; and

reprojecting at least portions of the combined images of the subset of the vantages to the virtual viewpoint.

6. The method of claim 5 , wherein identifying the subset of the vantages comprises identifying vertices of a polyhedron containing the virtual viewpoint.

7. The method of claim 6 , wherein:

the polyhedron comprises a tetrahedron; and

identifying the vertices of the polyhedron comprises identifying four of the vantages that define corners of the tetrahedron.

8. A method for preparing video data of an environment for stereoscopic presentation to a viewer in an experience selected from virtual reality and augmented reality experiences, the method comprising:

at a processor, designating a plurality of locations, distributed throughout a viewing volume, at which a plurality of vantages are to be positioned to facilitate viewing of the environment from proximate the locations in the experience;

at the processor, for each location of the plurality of the locations:

retrieving a plurality of images of the environment captured from viewpoints proximate the location; and

combined image comprising a plurality of subject pixels; and

at a data store, storing each of the vantages such that the vantages can be used to generate stereoscopic viewpoint video of the environment, as viewed from at least two virtual viewpoints corresponding to viewpoints of an actual viewer's eyes within the viewing volume, wherein designating the plurality of locations comprises spacing the plurality of locations apart at irregular intervals.

9. The method of claim 8 , wherein:

each combined image comprises color data and depth data for each of a plurality of pixels; and

the depth data comprises an indication of a distance between a vantage location of the plurality of locations that corresponds to the combined image, and an object within the environment that is aligned with the pixel and the vantage location.

10. A non-transitory computer-readable medium for preparing video data of an environment for stereoscopic presentation to a viewer in an experience selected from virtual reality and augmented reality experiences, comprising instructions stored thereon, that when executed by a processor, cause the processor to perform the steps of:

designating a plurality of locations, distributed throughout a viewing volume, at which a plurality of vantages are to be positioned to facilitate viewing of the environment from proximate the locations in the experience;

for each location of the plurality of the locations:

retrieving a plurality of images of the environment captured from viewpoints proximate the location; and

combining the images to generate a combined image to generate a vantage, the combined image comprising a plurality of subject pixels; and

causing a data store to store each of the vantages such that the vantages can be used to generate stereoscopic viewpoint video of the environment, as viewed from at least two virtual viewpoints corresponding to viewpoints of an actual viewer's eyes within the viewing volume, wherein combining the images comprises, for each subject pixel of each combined image, using linear interpolation to select color and/or depth data for the subject pixel based on corresponding color and/or depth data for each candidate pixel of each of the plurality of images that corresponds to the subject pixel.

11. A non-transitory computer-readable medium for preparing video data of an environment for stereoscopic presentation to a viewer in an experience selected from virtual reality and augmented reality experiences, comprising instructions stored thereon, that when executed by a processor, cause the processor to perform the steps of:

designating a plurality of locations, distributed throughout a viewing volume, at which a plurality of vantages are to be positioned to facilitate viewing of the environment from proximate the locations in the experience;

for each location of the plurality of the locations:

retrieving a plurality of images of the environment captured from viewpoints proximate the location; and

combining the images to generate a combined image to generate a vantage, the combined image comprising a plurality of subject pixels; and

causing a data store to store each of the vantages such that the vantages can be used to generate stereoscopic viewpoint video of the environment, as viewed from at least two virtual viewpoints corresponding to viewpoints of an actual viewer's eyes within the viewing volume

receiving designations of the at least two virtual viewpoints; and

using the vantages to generate the stereoscopic viewpoint video.

12. The non-transitory computer-readable medium of claim 11 , wherein:

using the vantages to generate the stereoscopic viewpoint video comprises combining the combined images from a plurality of the vantages; and

combining the combined images from a plurality of the vantages comprises:

assigning a fitness value to each candidate pixel of each of the combined images that corresponds to the subject pixel; and

setting the subject pixel to be the same as the candidate pixel that corresponds to the subject pixel and has the highest fitness value;

wherein the fitness value for each of the candidate pixels represents a degree to which it is desirable to include the subject pixel in the experience.

13. The non-transitory computer-readable medium of claim 11 , wherein:

using the vantages to generate the stereoscopic viewpoint video comprises, for each of the virtual viewpoints, combining the combined images from a plurality of the vantages; and

combining the combined images from a plurality of the vantages comprises, for each subject pixel of the stereoscopic viewpoint video, using linear interpolation to select color and/or depth data for the subject pixel based on corresponding color and/or depth data for each candidate pixel of each of the combined images that corresponds to the subject pixel.

14. The non-transitory computer-readable medium of claim 11 , wherein using the vantages to generate the stereoscopic viewpoint video comprises, for each of the virtual viewpoints:

identifying a subset of the vantages that are proximate the virtual viewpoint by identifying vertices of a polyhedron containing the virtual viewpoint; and

combining the combined images from the subset of the vantages by reprojecting at least portions of the combined images of the subset of the vantages to the virtual viewpoint.

15. The non-transitory computer-readable medium of claim 14 , wherein:

the polyhedron comprises a tetrahedron; and

identifying the vertices of the polyhedron comprises identifying four of the vantages that define corners of the tetrahedron.

16. A system for preparing video data of an environment for stereoscopic presentation to a viewer in an experience selected from virtual reality and augmented reality experiences, the system comprising:

a processor configured to:

designate a plurality of locations, distributed throughout a viewing volume, at which a plurality of vantages are to be positioned to facilitate viewing of the environment from proximate the locations in the experience; and

for each location of the plurality of the locations:

retrieve a plurality of images of the environment captured from viewpoints proximate the location; and

combine the images to generate a combined image to generate a vantage, the combined image comprising a plurality of subject pixels; and

a data store communicatively coupled to the processor, configured to store each of the vantages such that the vantages can be used to generate stereoscopic viewpoint video of the environment, as viewed from at least two virtual viewpoints corresponding to viewpoints of an actual viewer's eyes within the viewing volume, wherein the processor is further configured to:

receive designations of the at least two virtual viewpoints; and

use the vantages to generate the stereoscopic viewpoint video.

17. The system of claim 16 , wherein the processor is further configured to use the vantages to generate the stereoscopic viewpoint video by, for each of the virtual viewpoints, combining the combined images from a plurality of the vantages by, for each subject pixel of the stereoscopic viewpoint video:

assigning a fitness value to each candidate pixel of each of the combined images that corresponds to the subject pixel; and

setting the subject pixel to be the same as the candidate pixel that corresponds to the subject pixel and has the highest fitness value;

wherein the fitness value for each of the candidate pixels represents a degree to which it is desirable to include the subject pixel in the experience.

18. The system of claim 16 , wherein the processor is further configured to use the vantages to generate the stereoscopic viewpoint video by combining the combined images from a plurality of the vantages by, for each subject pixel of the stereoscopic viewpoint video, using linear interpolation to select color and/or depth data for the subject pixel based on corresponding color and/or depth data for each candidate pixel of each of the combined images that corresponds to the subject pixel.

19. The system of claim 16 , wherein the processor is further configured to use the vantages to generate the stereoscopic viewpoint video by, for each of the virtual viewpoints:

identifying a subset of the vantages that are proximate the virtual viewpoint by identifying vertices of a polyhedron containing the virtual viewpoint; and

combining the combined images from the subset of the vantages by reprojecting at least portions of the combined images of the subset of the vantages to the virtual viewpoint.

20. The system of claim 19 , wherein:

the polyhedron comprises a tetrahedron; and

the processor is further configured to identify the vertices of the polyhedron by identifying four of the vantages that define corners of the tetrahedron.