Encoding and decoding virtual reality video

1. A method for decoding a video stream of a scene for a virtual reality or augmented reality experience, the method comprising:

from a data store, retrieving at least part of the video stream;

at an input device, receiving a first viewer position and/or orientation;

at a processor, partially decoding the video stream to generate a partially-decoded bitstream;

at a graphics processor, further decoding the partially-decoded bitstream to generate viewpoint video of the scene from a first virtual viewpoint corresponding to the first viewer position and/or orientation; and

at a display device, displaying the viewpoint video, wherein a handoff of the partially-decoded bitstream from the processor to the graphics processor is based upon a scanline iteration, the scanline iteration being indicative of a manner in which storage occurs in the graphics processor.

2. The method of claim 1 , wherein:

the video stream comprises a plurality of vantages, each of which comprises a view of the scene from one of a plurality of virtual viewpoints within a viewing volume; and

each of the vantages comprises a plurality of tiles, each of which depicts the scene from the virtual viewpoint associated with the vantage, within a limited field of view.

3. The method of claim 2 , wherein partially decoding the video stream comprises partially decoding a plurality of the tiles in parallel.

4. The method of claim 2 , wherein partially decoding the video stream comprises:

dividing each of the tiles into a plurality of macroblocks; and

removing serial dependencies from each of the tiles and/or macroblocks.

5. The method of claim 1 , wherein further decoding the partially-decoded bitstream comprises storing the partially-decoded bitstream in a graphics processor memory of the graphics processor in a manner that reduces access latency for the graphics processor memory.

6. The method of claim 1 , wherein:

the graphics processor is configured for multi-threaded operation; and

further decoding the partially-decoded bitstream comprises dividing the partially-decoded bitstream into a plurality of thread-groups, each of which can be processed by a thread of the graphics processor.

7. The method of claim 6 , wherein further decoding the partially-decoded bitstream further comprises synchronizing decoding between at least some of the thread-groups.

8. The method of claim 1 , wherein:

the video stream comprises a depth channel and one or more color channels;

at least one of partially decoding the video stream and further decoding the partially-decoded bitstream comprises:

decoding one or more color channels; and

decoding the depth channel.

9. The method of claim 1 , wherein partially decoding the video stream comprises performing at least one selection from the group consisting of:

performing Huffman decoding on the video stream;

performing inverse run length encoding on the video stream; and

performing de-zigzagging on the video stream.

10. The method of claim 1 , wherein further decoding the partially-decoded bitstream comprises performing at least one selection from the group consisting of:

performing de-quantization on the partially-decoded bitstream;

performing inverse discrete cosine transformation on the partially-decoded bitstream; and

performing color conversion on the partially-decoded bitstream.

11. A method for encoding a volumetric video data of a scene for a virtual reality or augmented reality experience, the method comprising:

from a data store, retrieving the volumetric video data; and

at a processor, encoding the volumetric video data to generate a video stream configured to facilitate decoding via a decoding method comprising the steps of:

at a processor, partially decoding the video stream to generate a partially-decoded bitstream; and

at a graphics processor, further decoding the partially-decoded bitstream to generate viewpoint video of the scene from a virtual viewpoint corresponding to a viewer-designated viewer position and/or orientation, wherein a handoff of the partially-decoded bitstream from the processor to the graphics processor is based upon a scanline iteration, the scanline iteration being indicative of a manner in which storage occurs in the graphics processor.

12. The method of claim 11 , wherein:

the video stream comprises a plurality of vantages, each of which comprises a view of the scene from one of a plurality of virtual viewpoints within a viewing volume; and

each of the vantages comprises a plurality of tiles, each of which depicts the scene from the virtual viewpoint associated with the vantage, within a limited field of view.

13. A non-transitory computer-readable medium for decoding a video stream of a scene for a virtual reality or augmented reality experience, comprising instructions stored thereon, that when executed by one or more processors, perform the steps of:

causing a data store to retrieve at least part of the video stream;

causing an input device to receive a first viewer position and/or orientation;

partially decoding the video stream to generate a partially-decoded bitstream;

causing a graphics processor to further decode the partially-decoded bitstream to generate viewpoint video of the scene from a first virtual viewpoint corresponding to the first viewer position and/or orientation; and

causing a display device to display the viewpoint video, wherein a handoff of the partially-decoded bitstream from the processor to the graphics processor is based upon a scanline iteration, the scanline iteration being indicative of a manner in which storage occurs in the graphics processor.

14. The non-transitory computer-readable medium of claim 13 , wherein:

the video stream comprises a plurality of vantages, each of which comprises a view of the scene from one of a plurality of virtual viewpoints within a viewing volume; and

each of the vantages comprises a plurality of tiles, each of which depicts the scene from the virtual viewpoint associated with the vantage, within a limited field of view.

15. The non-transitory computer-readable medium of claim 13 , wherein further decoding the partially-decoded bitstream comprises storing the partially-decoded bitstream in a graphics processor memory of the graphics processor in a manner that reduces access latency for the graphics processor memory.

16. The non-transitory computer-readable medium of claim 13 , wherein:

the graphics processor is configured for multi-threaded operation;

further decoding the partially-decoded bitstream comprises dividing the partially-decoded bitstream into a plurality of thread-groups, each of which can be processed by a thread of the graphics processor; and

further decoding the partially-decoded bitstream further comprises synchronizing decoding between at least some of the thread-groups.

17. The non-transitory computer-readable medium of claim 13 , wherein:

the video stream comprises a depth channel and one or more color channels;

at least one of partially decoding the video stream and further decoding the partially-decoded bitstream comprises:

decoding one or more color channels; and

decoding the depth channel.

18. The non-transitory computer-readable medium of claim 13 , wherein partially decoding the video stream comprises performing at least one selection from the group consisting of:

performing Huffman decoding on the video stream;

performing inverse run length encoding on the video stream; and

performing de-zigzagging on the video stream.

19. The non-transitory computer-readable medium of claim 13 , wherein further decoding the partially-decoded bitstream comprises performing at least one selection from the group consisting of:

performing de-quantization on the partially-decoded bitstream;

performing inverse discrete cosine transformation on the partially-decoded bitstream; and

performing color conversion on the partially-decoded bitstream.

20. A non-transitory computer-readable medium for encoding a volumetric video data of a scene for a virtual reality or augmented reality experience, comprising instructions stored thereon, that when executed by one or more processors, perform the steps of:

causing a data store to retrieve the volumetric video data; and

encoding the volumetric video data to generate a video stream configured to facilitate decoding via a decoding method comprising the steps of:

at a processor, partially decoding the video stream to generate a partially-decoded bitstream; and

at a graphics processor, further decoding the partially-decoded bitstream to generate viewpoint video of the scene from a virtual viewpoint corresponding to a viewer-designated viewer position and/or orientation, wherein a handoff of the partially-decoded bitstream from the processor to the graphics processor is based upon a scanline iteration, the scanline iteration being indicative of a manner in which storage occurs in the graphics processor.

21. The non-transitory computer-readable medium of claim 20 , wherein:

the video stream comprises a plurality of vantages, each of which comprises a view of the scene from one of a plurality of virtual viewpoints within a viewing volume; and

each of the vantages comprises a plurality of tiles, each of which depicts the scene from the virtual viewpoint associated with the vantage, within a limited field of view.

22. A system for decoding a video stream of a scene for a virtual reality or augmented reality experience, the system comprising:

a data store configured to retrieve at least part of the video stream;

an input device configured to receive a first viewer position and/or orientation;

a processor configured to partially decode the video stream to generate a partially-decoded bitstream;

a graphics processor configured to further decode the partially-decoded bitstream to generate viewpoint video of the scene from a first virtual viewpoint corresponding to the first viewer position and/or orientation; and

a display device configured to display the viewpoint video, wherein a handoff of the partially-decoded bitstream from the processor to the graphics processor is based upon a scanline iteration, the scanline iteration being indicative of a manner in which storage occurs in the graphics processor.

23. The system of claim 22 , wherein:

the video stream comprises a plurality of vantages, each of which comprises a view of the scene from one of a plurality of virtual viewpoints within a viewing volume; and

each of the vantages comprises a plurality of tiles, each of which depicts the scene from the virtual viewpoint associated with the vantage, within a limited field of view.

24. The system of claim 22 , wherein the graphics processor is configured to further decode the partially-decoded bitstream by storing the partially-decoded bitstream in a graphics processor memory of the graphics processor in a manner that reduces access latency for the graphics processor memory.

25. The system of claim 22 , wherein:

the graphics processor is configured for multi-threaded operation;

the graphics processor is configured to further decode the partially-decoded bitstream by:

dividing the partially-decoded bitstream into a plurality of thread-groups, each of which can be processed by a thread of the graphics processor; and

synchronizing decoding between at least some of the thread-groups.

26. The system of claim 22 , wherein:

the video stream comprises a depth channel and one or more color channels;

at least one of the processor and the graphics processor is configured to partially decode the video stream or further decode the partially-decoded bitstream by:

decoding one or more color channels; and

decoding the depth channel.

27. The system of claim 22 , wherein the processor is configured to partially decode the video stream by performing at least one selection from the group consisting of:

performing Huffman decoding on the video stream;

performing inverse run length encoding on the video stream; and

performing de-zigzagging on the video stream.

28. The system of claim 22 , wherein the graphics processor is configured to further decode the partially-decoded bitstream by performing at least one selection from the group consisting of:

performing de-quantization on the partially-decoded bitstream;

performing inverse discrete cosine transformation on the partially-decoded bitstream; and

performing color conversion on the partially-decoded bitstream.

29. A system for encoding a volumetric video data of a scene for a virtual reality or augmented reality experience, the system comprising:

a data store configured to retrieve the volumetric video data; and

a first processor configured to encode the volumetric video data to generate a video stream configured to facilitate decoding via a decoding method comprising the steps of:

at a second processor, partially decoding the video stream to generate a partially-decoded bitstream; and

at a graphics processor, further decoding the partially-decoded bitstream to generate viewpoint video of the scene from a virtual viewpoint corresponding to a viewer-designated viewer position and/or orientation, wherein a handoff of the partially-decoded bitstream from the processor to the graphics processor is based upon a scanline iteration, the scanline iteration being indicative of a manner in which storage occurs in the graphics processor.

30. The system of claim 29 , wherein:

the video stream comprises a plurality of vantages, each of which comprises a view of the scene from one of a plurality of virtual viewpoints within a viewing volume; and

each of the vantages comprises a plurality of tiles, each of which depicts the scene from the virtual viewpoint associated with the vantage, within a limited field of view.