Methods and apparatus for deriving composite tracks

1. An encoding method for encoding a composition operation for a plurality of two-dimensional sub-picture tracks, the method comprising:

encoding video data in a hierarchical track structure, comprising:

receiving the video data including the plurality of two-dimensional sub-picture tracks associated with a viewport, wherein the plurality of two-dimensional sub-picture tracks are at a first level of the hierarchical track structure; and

encoding a composition operation for composing the plurality of two-dimensional sub-picture tracks for the viewport, wherein the composition operation is associated with a composite track at a second level in the hierarchical track structure that is above the first level of the plurality of two-dimensional sub-picture tracks, and comprises data indicative of:

a composition to perform on the plurality of two-dimensional sub-picture tracks to compose the plurality of two-dimensional sub-picture tracks into a canvas associated with the viewport, wherein the canvas comprises a plurality of portions and each of the plurality of two-dimensional sub-picture tracks comprises video data for at least an associated portion of the plurality of portions; and

a composition layout operation to adjust the composition if the canvas comprises a composition layout created by two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas; and

providing the received video data and the encoded composition operation.

2. The encoding method of claim 1 , wherein the composition layout comprises a gap between the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas, an overlap of the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas, or both.

3. The encoding method of claim 1 , wherein encoding the composition layout operation of the composition operation comprises encoding one or more of a background color, a background image, or a background video to use to fill a gap between the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas.

4. The encoding method of claim 1 , wherein encoding the composition layout operation of the composition operation comprises encoding blending data to use to blend an overlap of the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas.

5. The encoding method of claim 1 , wherein encoding the composition of the composition operation comprises selecting the composition from the group consisting of:

a track overlay composition specifying an overlay operation and an order for overlaying each of the plurality of two-dimensional sub-picture tracks on the canvas;

a track grid composition specifying a grid order for overlaying each of the plurality of two-dimensional sub-picture tracks on the canvas; and

a track matrix composition specifying an order and a matrix used to overlay each of the plurality of two-dimensional sub-picture tracks on the canvas.

6. A decoding method for decoding video data, the method comprising:

receiving:

(a) video data including a plurality of two-dimensional sub-picture tracks associated with a viewport, wherein the plurality of two-dimensional sub-picture tracks are at a first level of a hierarchical track structure; and

(b) a composition operation to compose the plurality of two-dimensional sub-picture tracks for the viewport, wherein the composition operation is associated with a composite track at a second level in the hierarchical track structure that is above the first level of the plurality of two-dimensional sub-picture tracks, and comprises data indicative of:

a composition to perform on the plurality of two-dimensional sub-picture tracks to compose the plurality of two-dimensional sub-picture tracks into a canvas associated with the viewport, wherein the canvas comprises a plurality of portions and each of the plurality of two-dimensional sub-picture tracks comprises video data for at least an associated portion of the plurality of portions; and

a composition layout operation to adjust the composition if the canvas comprises a composition layout created by two or more of the plurality of two- dimensional sub-picture tracks composed on the canvas; and

composing the plurality of two-dimensional sub-picture tracks into the canvas according to the composition, comprising:

determining two or more of the composed two-dimensional sub-picture tracks comprise the composition layout; and

adjusting the composition based on the composition layout operation to compensate for the composition layout.

7. The decoding method of claim 6 , wherein the composition layout comprises a gap between the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas, an overlap of the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas, or both.

8. The decoding method of claim 6 , wherein decoding the composition layout operation of the composition operation comprises:

decoding one or more of a background color, a background image, or a background video; and

composing the plurality of two-dimensional sub-picture tracks comprises filling a gap between the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas.

9. The decoding method of claim 6 , wherein decoding the composition layout operation of the composition operation comprises:

decoding blending data; and

composing the plurality of two-dimensional sub-picture tracks comprises blending an overlap of the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas.

10. The decoding method of claim 6 , wherein decoding the composition of the composition operation comprises decoding the composition from the group consisting of:

a track overlay composition specifying an overlay operation and an order for overlaying each of the plurality of two-dimensional sub-picture tracks on the canvas;

a track grid composition specifying a grid order for overlaying each of the plurality of two-dimensional sub-picture tracks on the canvas; and

a track matrix composition specifying an order and a matrix used to overlay each of the plurality of two-dimensional sub-picture tracks on the canvas.

11. An apparatus configured to decode video data, the apparatus comprising a processor in communication with memory, the processor being configured to execute instructions stored in the memory that cause the processor to:

receive:

(a) video data including a plurality of two-dimensional sub-picture tracks associated with a viewport, wherein the plurality of two-dimensional sub-picture tracks are at a first level of a hierarchical track structure; and

(b) a composition operation to compose the plurality of two-dimensional sub-picture tracks for the viewport, wherein the composition operation is associated with a composite track at a second level in the hierarchical track structure that is above the first level of the plurality of two-dimensional sub-picture tracks, and comprises data indicative of:

a composition to perform on the plurality of two-dimensional sub-picture tracks to compose the plurality of two-dimensional sub-picture tracks into a canvas associated with the viewport, wherein the canvas comprises a plurality of portions and each of the plurality of two-dimensional sub-picture tracks comprises video data for at least an associated portion of the plurality of portions; and

a composition layout operation to adjust the composition if the canvas comprises a composition layout created by two or more of the plurality of two- dimensional sub-picture tracks composed on the canvas; and

compose the plurality of two-dimensional sub-picture tracks into the canvas according to the composition, comprising:

determining two or more of the composed two-dimensional sub-picture tracks comprise the composition layout; and

adjusting the composition based on the composition layout operation to compensate for the composition layout.

12. The apparatus of claim 11 , wherein the composition layout comprises a gap between the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas, an overlap of the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas, or both.

13. The apparatus of claim 11 , wherein decoding the composition layout operation of the composition operation comprises:

decoding one or more of a background color, a background image, or a background video; and

composing the plurality of two-dimensional sub-picture tracks comprises filling a gap between the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas.

14. The apparatus of claim 11 , wherein decoding the composition layout operation of the composition operation comprises:

decoding blending data; and

composing the plurality of two-dimensional sub-picture tracks comprises blending an overlap of the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas.

15. The apparatus of claim 11 , wherein decoding the composition of the composition operation comprises decoding the composition from the group consisting of:

a track overlay composition specifying an overlay operation and an order for overlaying each of the plurality of two-dimensional sub-picture tracks on the canvas;

a track grid composition specifying a grid order for overlaying each of the plurality of two-dimensional sub-picture tracks on the canvas; and

a track matrix composition specifying an order and a matrix used to overlay each of the plurality of two-dimensional sub-picture tracks on the canvas.

16. An apparatus for encoding video data, the apparatus comprising a processor in communication with memory, the processor being configured to execute instructions stored in the memory that cause the processor to:

encode video data in a hierarchical track structure, comprising:

receiving the video data including a plurality of two-dimensional sub-picture tracks associated with a viewport, wherein the plurality of two-dimensional sub-picture tracks are at a first level of the hierarchical track structure; and

encoding a composition operation for composing the plurality of two-dimensional sub-picture tracks for the viewport, wherein the composition operation is associated with a composite track at a second level in the hierarchical track structure that is above the first level of the plurality of two-dimensional sub-picture tracks, and comprises data indicative of:

a composition to perform on the plurality of two-dimensional sub-picture tracks to compose the plurality of two-dimensional sub-picture tracks into a canvas associated with the viewport, wherein the canvas comprises a plurality of portions and each of the plurality of two-dimensional sub-picture tracks comprises video data for at least an associated portion of the plurality of portions; and

a composition layout operation to adjust the composition if the canvas comprises a composition layout created by two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas; and

provide the received video data and the encoded composition operation.

17. The apparatus of claim 16 , wherein the composition layout comprises a gap between the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas, an overlap of the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas, or both.

18. The apparatus of claim 16 , wherein encoding the composition layout operation of the composition operation comprises encoding one or more of a background color, a background image, or a background video to use to fill a gap between the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas.

19. The apparatus of claim 16 , wherein encoding the composition layout operation of the composition operation comprises encoding blending data to use to blend an overlap of the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas.

20. The apparatus of claim 16 , wherein encoding the composition of the composition operation comprises selecting the composition from the group consisting of:

a track overlay composition specifying an overlay operation and an order for overlaying each of the plurality of two-dimensional sub-picture tracks on the canvas;

a track grid composition specifying a grid order for overlaying each of the plurality of two-dimensional sub-picture tracks on the canvas; and

a track matrix composition specifying an order and a matrix used to overlay each of the plurality of two-dimensional sub-picture tracks on the canvas.