Layered content delivery for virtual and augmented reality experiences
A virtual reality or augmented reality experience of a scene may be presented to a viewer using layered data retrieval and/or processing. A first layer of a video stream may be retrieved, and a first viewer position and/or orientation may be received. The first layer may be processed to generate first viewpoint video of the scene from a first virtual viewpoint corresponding to the first viewer position and/or orientation. The first viewpoint video may be displayed for the viewer. Then, a second layer of the video stream may be retrieved, and a second viewer position and/or orientation may be received. The second layer may be processed to generate second viewpoint video of the scene from a second virtual viewpoint corresponding to the second viewer position and/or orientation, with higher quality than the first viewpoint video. The second viewpoint video may be displayed for the viewer.
1. A method for presenting a virtual reality or augmented reality experience of a scene, the method comprising:
from a data store, retrieving a first layer of a video stream;
at an input device, receiving a first viewer position and/or orientation;
at a processor, processing the first layer of the video stream to generate first viewpoint video of the scene from a first virtual viewpoint corresponding to the first viewer position and/or orientation, the first viewpoint video having a first quality level;
at a display device, displaying the first viewpoint video;
from the data store, after displaying the first viewpoint video, retrieving a second layer of the video stream;
at the input device, receiving a second viewer position and/or orientation;
at the processor, processing the second layer of the video stream to generate second viewpoint video of the scene from a second virtual viewpoint corresponding to the second viewer position and/or orientation, the second viewpoint video having a second quality level higher than the first quality level; and
at the display device, displaying the second viewpoint video, wherein:
the video stream comprises a plurality of vantages, each of which comprises a view of the scene from one of a plurality of virtual viewpoints within a viewing volume;
each of the vantages comprises a plurality of tiles, each of which depicts the scene from the virtual viewpoint associated with the vantage, within a limited field of view;
the first layer of the video stream comprises a first set of the tiles; and
the second layer of the video stream comprises a second set of the tiles, each of which corresponds to one of the tiles of the first set of the tiles and comprises a level of view-dependent lighting greater than that of the corresponding tile of the first set of the tiles.
2. A non-transitory computer-readable medium for presenting a virtual reality or augmented reality experience of a scene, comprising instructions stored thereon, that when executed by one or more processors, perform the steps of:
causing a data store to retrieve a first layer of a video stream;
causing an input device to receive a first viewer position and/or orientation;
processing the first layer of the video stream to generate first viewpoint video of the scene from a first virtual viewpoint corresponding to the first viewer position and/or orientation, the first viewpoint video having a first quality level;
causing a display device, displaying the first viewpoint video;
after displaying the first viewpoint video, causing the data store to retrieve a second layer of the video stream;
causing the input device to receive a second viewer position and/or orientation;
processing the second layer of the video stream to generate second viewpoint video of the scene from a second virtual viewpoint corresponding to the second viewer position and/or orientation, the second viewpoint video having a second quality level higher than the first quality level; and
causing the display device to display the second viewpoint video, wherein:
the video stream comprises a plurality of vantages, each of which comprises a view of the scene from one of a plurality of virtual viewpoints within a viewing volume;
each of the vantages comprises a plurality of tiles, each of which depicts the scene from the virtual viewpoint associated with the vantage, within a limited field of view;
the first layer of the video stream comprises a first set of the tiles; and
the second layer of the video stream comprises a second set of the tiles, each of which corresponds to one of the tiles of the first set of the tiles and comprises a level of view-dependent lighting greater than that of the corresponding tile of the first set of the tiles.
3. A system for presenting a virtual reality or augmented reality experience of a scene, the system comprising:
a data store configured to retrieve a first layer of a video stream;
an input device configured to receive a first viewer position and/or orientation;
a processor configured to process the first layer of the video stream to generate first viewpoint video of the scene from a first virtual viewpoint corresponding to the first viewer position and/or orientation, the first viewpoint video having a first quality level; and
a display device configured to display the first viewpoint video;
wherein:
the data store is further configured, after the first viewpoint video has been displayed, to retrieve a second layer of the video stream;
the input device is further configured to receive a second viewer position and/or orientation;
the processor is further configured to process the second layer of the video stream to generate second viewpoint video of the scene from a second virtual viewpoint corresponding to the second viewer position and/or orientation, the second viewpoint video having a second quality level higher than the first quality level; and
the display device is further configured to display the second viewpoint video, wherein:
the video stream comprises a plurality of vantages, each of which comprises a view of the scene from one of a plurality of virtual viewpoints within a viewing volume;
each of the vantages comprises a plurality of tiles, each of which depicts the scene from the virtual viewpoint associated with the vantage, within a limited field of view;
the first layer of the video stream comprises a first set of the tiles; and
the second layer of the video stream comprises a second set of the tiles, each of which corresponds to one of the tiles of the first set of the tiles and comprises a level of view-dependent lighting greater than that of the corresponding tile of the first set of the tiles.