Processing omnidirectional media with dynamic region-wise packing

1. A method of processing video data, the method comprising:

processing a first frame of a set of file format samples of an omnidirectional video stream, the set of file format samples referring to a sample entry, the first frame having a first region-wise packing scheme, wherein the omnidirectional video stream includes a scheme type syntax element of file format information, the scheme type syntax element having a value indicating that the omnidirectional video stream includes equirectangular projected video with dynamic region-wise packing; and

processing a second frame of the set of file format samples referring to the sample entry, the second frame having a second region-wise packing scheme different than the first region-wise packing scheme.

2. The method of claim 1 , wherein processing the first frame and processing the second frame comprises processing the omnidirectional video stream as a single omnidirectional video stream including an entire virtual reality (VR) video content having one or more most requested regions that are optimized with a largest number of color samples per unit of area, the single omnidirectional video stream comprising the set of file format samples including the first frame and the second frame.

3. The method of claim 1 , wherein the value of the scheme type syntax element is ‘erp2’.

4. The method of claim 1 , wherein the scheme type syntax element has a value of either ‘podv’ and ‘erpv’ or ‘podv’ and ‘erp2’.

5. The method of claim 4 , wherein when the scheme type ‘erpv’ is present in the omnidirectional video stream, the method includes determining that a region wise packing box is present in the omnidirectional video stream and that the region wise packing box signals information also signaled in one or more region-wise supplemental enhancement information (SEI) messages for the omnidirectional video stream.

6. The method of claim 5 , wherein the region wise packing box is included in file format information of a file included in the omnidirectional video stream, the file including the set of file format samples.

7. The method of claim 1 , wherein at least one of ‘erpv’ or ‘erp2’ scheme types are signaled in the omnidirectional video stream as optional MIME type parameters for a network streaming protocol.

8. The method of claim 7 , wherein the network streaming protocol comprises Dynamic Adaptive Streaming over HTTP (DASH).

9. The method of claim 7 , further comprising processing a manifest file for the network streaming protocol, the manifest file comprising the ‘erpv’ or ‘erp2’ scheme types.

10. The method of claim 9 , wherein the manifest file comprises a media presentation description (MPD).

11. The method of claim 1 , wherein the omnidirectional video stream conforms to a high efficiency video coding (HEVC) media profile that defines the scheme type syntax element value as indicating that the omnidirectional video stream includes the equirectangular projected video with dynamic region-wise packing.

12. The method of claim 1 ,

wherein processing the first frame comprises encoding or transmitting the first frame, the method further comprising:

determining the first region-wise packing scheme of the first frame; and

encoding a first scheme type value for the first frame to indicate the first region-wise packing scheme of the first frame; and

wherein processing the second frame comprises encoding or transmitting the second frame, the method further comprising:

determining the second region-wise packing scheme of the second frame; and

encoding a second scheme type value for the second frame to indicate the second region-wise packing scheme of the second frame.

13. The method of claim 1 ,

wherein processing the first frame comprises receiving or decoding the first frame, the method further comprising:

decoding a first scheme type value for the first frame; and

determining the first region-wise packing scheme of the first frame from the first scheme type value; and

wherein processing the second frame comprises receiving or decoding the second frame, the method further comprising:

decoding a second scheme type value for the second frame; and

determining the second region-wise packing scheme of the second frame from the second scheme type value.

14. The method of claim 1 , further comprising:

decoding the first frame;

rearranging one or more regions of the first decoded frame according to the first region-wise packing scheme;

decoding the second frame; and

rearranging one or more regions of the second decoded frame according to the second region-wise packing scheme.

15. A device for processing video data, the device comprising:

a memory configured to store an omnidirectional video stream, wherein the omnidirectional video stream includes a scheme type syntax element of file format information, the scheme type syntax element having a value indicating that the omnidirectional video stream includes equirectangular projected video with dynamic region-wise packing; and

a processor implemented in circuitry and configured to:

process a first frame of a set of file format samples of the omnidirectional video stream, the set of file format samples referring to a sample entry, the first frame having a first region-wise packing scheme; and

process a second frame of the set of file format samples referring to the sample entry, the second frame having a second region-wise packing scheme different than the first region-wise packing scheme.

16. The device of claim 15 , wherein the processor is configured to process the omnidirectional video stream as a single omnidirectional video stream including an entire virtual reality (VR) video content having one or more most requested regions that are optimized with a largest number of color samples per unit of area, the single omnidirectional video stream comprising the set of file format samples including the first frame and the second frame.

17. The device of claim 15 , wherein the value of the scheme type syntax element is ‘erp2’.

18. The device of claim 15 , wherein the scheme type syntax element has a value of either ‘podv’ and ‘erpv’ or ‘podv’ and ‘erp2’.

19. The device of claim 18 , wherein the processor is configured to, when the scheme type ‘erpv’ is present in the omnidirectional video stream, determine that a region wise packing box is present in the omnidirectional video stream and that the region wise packing box signals information also signaled in one or more region-wise supplemental enhancement information (SEI) messages for the omnidirectional video stream.

20. The device of claim 19 , wherein the region wise packing box is included in file format information of a file included in the omnidirectional video stream, the file including the set of file format samples.

21. The device of claim 15 , wherein at least one of ‘erpv’ or ‘erp2’ scheme types are signaled as optional MIME type parameters for a network streaming protocol.

22. The device of claim 21 , wherein the network streaming protocol comprises Dynamic Adaptive Streaming over HTTP (DASH).

23. The device of claim 21 , wherein the processor is further configured to process a manifest file for the network streaming protocol, the manifest file comprising the ‘erpv’ or ‘erp2’ scheme types.

24. The device of claim 23 , wherein the manifest file comprises a media presentation description (MPD).

25. The device of claim 15 , wherein the omnidirectional video stream conforms to a high efficiency video coding (HEVC) media profile that defines the scheme type syntax element value as indicating that the omnidirectional video stream includes the equirectangular projected video with dynamic region-wise packing.

26. The device of claim 15 ,

wherein to process the first frame, the processor is configured to encode or transmit the first frame, and wherein the processor is further configured to:

determine the first region-wise packing scheme of the first frame; and

encode a first scheme type value for the first frame to indicate the first region-wise packing scheme of the first frame; and

wherein to process the second frame, the processor is further configured to encode or transmit the second frame, and wherein the processor is further configured to:

determine the second region-wise packing scheme of the second frame; and

encode a second scheme type value for the second frame to indicate the second region-wise packing scheme of the second frame.

27. The device of claim 15 ,

wherein to process the first frame, the processor is configured to receive or decode the first frame, and wherein the processor is further configured to:

decode a first scheme type value for the first frame; and

determine the first region-wise packing scheme of the first frame from the first scheme type value; and

wherein to process the second frame, the processor is configured to receive or decode the second frame, and wherein the processor is further configured to:

decode a second scheme type value for the second frame; and

determine the second region-wise packing scheme of the second frame from the second scheme type value.

28. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause a processor to:

process a first frame of a set of file format samples of an omnidirectional video stream, the set of file format samples referring to a sample entry, the first frame having a first region-wise packing scheme, wherein the omnidirectional video stream includes a scheme type syntax element of file format information, the scheme type syntax element having a value indicating that the omnidirectional video stream includes equirectangular projected video with dynamic region-wise packing; and

process a second frame of the set of file format samples referring to the sample entry, the second frame having a second region-wise packing scheme different than the first region-wise packing scheme.