Inter-layer prediction for extended spatial scalability in video coding
View Patent ↗An improved system and method for providing improved inter-layer prediction for extended spatial scalability in video coding, as well as improving inter-layer prediction for motion vectors in the case of extended spatial scalability. In various embodiments, for the prediction of macroblock mode, the actual reference frame index and motion vectors from the base layer are used in determining if two blocks should be merged. Additionally, multiple representative pixels in a 4×4 block can be used to represent each 4×4 block in a virtual base layer macroblock. The partition and motion vector information for the relevant block in the virtual base layer macroblock can be derived from all of the partition information and motion vectors of those 4×4 blocks.
1. A method comprising:
deriving a coding mode indicator specifying a partition of enhancement layer block based at least in part on a plurality of reference frame index values corresponding to a plurality of reference layer blocks, the reference layer blocks representing an area in a base layer that, after upsampling, covers the enhancement layer block,
wherein the partition's size is determined by merging reference layer blocks that have the same reference frame index and that have motion vectors having a difference metric less than a predetermined threshold,
wherein the difference metric is derived by a sum of absolute difference of motion vector components;
deriving a motion vector associated with the partition of the enhancement layer block, wherein the derived motion vector comprises a weighted average of the motion vectors of the merged reference layer blocks before merging, wherein weights for the weighted average are based on differentials between the motion vectors of the reference layer blocks before merging, partition sizes associated with the reference layer blocks before merging, or a macroblock type or mode associated with the reference layer blocks before merging; and
encoding the enhancement layer block utilizing the coding mode indicator and the derived motion vector;
wherein the predetermined threshold is equal to one.
2. The method of claim 1 , wherein the merged reference layer blocks have the same motion vectors.
3. The method of claim 1 , wherein a plurality of representative pixels for locating corresponding reference layer blocks are used to select partition and motion vector information for each respective reference layer block.
4. The method of claim 1 , wherein, if the plurality of reference blocks have different reference frame indices, then the motion vector information from the reference block with the lowest reference frame indices is selected.
5. A computer program product, comprising a non-transitory computer-readable medium storing computer code, the computer code being configured to, upon execution, cause an apparatus to at least
encode an enhancement layer block representing at least a portion of a video frame within a scalable bit stream by
deriving a coding mode indicator specifying a partition of the enhancement layer block based at least in part on a plurality of reference frame index values corresponding to a plurality of reference layer blocks, the reference layer blocks representing an area in a base layer that, after upsampling, covers the enhancement layer block, and
determining the partition's size by merging reference layer blocks that have the same reference frame index and that have motion vectors having a difference metric less than a predetermined threshold, wherein the difference metric is derived by a sum of absolute difference of motion vector components, and
deriving a motion vector associated with the partition of the enhancement layer block, wherein the derived motion vector comprises a weighted average of the motion vectors of the merged reference layer blocks before merging, wherein weights for weighted average are based on differentials between the motion vectors of the reference layer blocks before merging, partition sizes associated with the reference layer blocks before merging, or a macroblock type or mode associated with the reference layer blocks before merging; and
encoding the enhancement layer block utilizing the coding mode indicator and the derived motion vector;
wherein the predetermined threshold is equal to one.
6. An apparatus, comprising: a processor; and a memory communicatively connected to the processor and including computer code, the memory and computer code being configured to, with the processor, cause the apparatus to at least:
derive a coding mode indicator specifying a partition of an enhancement layer block based at least in part on a plurality of reference frame index values corresponding to a plurality of reference layer blocks, the reference layer blocks representing an area in a base layer that, after upsampling, covers the enhancement layer block;
determine the partition's size by merging reference layer blocks that have the same reference frame index and that have motion vectors having a different metric less than a predetermined threshold, wherein the difference metric is derived by a sum of absolute difference of motion vector components;
deriving a motion vector associated with the partition of the enhancement layer block, wherein the derived motion vector comprises a weighted average of the motion vectors of the merged reference layer blocks before merging, wherein weights for the weighted average are based on differentials between the motion vectors of the reference layer blocks before merging, partition sizes associated with the reference layer blocks before merging, or a macroblock type or mode associated with the reference layer blocks before merging; and
encode the enhancement layer block utilizing the coding mode indicator and the derived motion vector;
wherein the predetermined threshold is equal to one.
7. The apparatus of claim 6 , wherein the merged reference layer blocks have the same motion vectors.
8. The apparatus of claim 6 , wherein a plurality of representative pixels for locating corresponding reference layer block are used to select partition and motion vector information for each respective reference layer block.
9. The apparatus of claim 6 , wherein, if the plurality of reference blocks have different reference frame indices, then the motion vector information from the reference block with the lowest reference frame indices is selected.
10. An apparatus, comprising:
a processor configured to
derive a coding mode indicator specifying a partition of an enhancement layer block based at least in part on a plurality of reference frame index values corresponding to a plurality of reference layer blocks, the reference layer blocks representing an area in a base layer that, after upsampling, covers the enhancement layer block,
determine the partition's size by merging the reference layer blocks that have the same reference frame index and that have motion vectors having a difference metric less than a predetermined threshold, wherein the difference metric is derived by a sum of absolute difference of motion vector components, and
derive a motion vector associated with the partition of the enhancement layer block, wherein the derived motion vector comprises a weighted average of the motion vectors of the merged reference layer blocks before merging, wherein weights for the weighted average are based on differentials between the motion vectors of the reference layer blocks before merging, partition sizes associated with the references layer blocks before merging, or a macroblock type or mode associated with the reference layer blocks before merging; and
an encoder configured to encode the enhancement layer block utilizing the coding mode indicator and the derived motion vector;
wherein the predetermined threshold is equal to one.
11. A method comprising:
deriving a coding mode indicator specifying a partition of an enhancement layer block based at least in part on a plurality of reference frame index values corresponding to a plurality of reference layer blocks, the reference layer blocks representing an area in a base layer that, after upsampling, covers the enhancement layer block,
wherein the partition's size is determined by merging the reference layer blocks that have the same reference frame index and that have motion vectors having a difference metric less than a predetermined threshold,
wherein the difference metric is derived by a sum of absolute difference of motion vector components;
deriving a motion vector associated with the partition of the enhancement layer block, wherein the derived motion vector comprises a weighted average of the motion vectors of the merged reference layer blocks before merging, wherein weights for the weighted average are based an differentials between the motion vectors of the reference layer blocks before merging, partition sizes associated with the reference layer blocks before merging, or a macroblock type or mode associated with the reference layer blocks before merging; and
decoding the enhancement layer block utilizing the coding mode indicator and the derived motion vector;
wherein the predetermined threshold is equal to one.
12. The method of claim 11 , wherein the merged reference layer blocks have the same motion vector information.
13. The method of claim 11 , wherein a plurality of representative pixels for locating corresponding reference layer blocks are used to select partition and motion vector information for the respective reference layer block.
14. The method of claim 11 , wherein, if the plurality of reference blocks have different reference frame indices, then the motion vector information from the block with the lowest reference frame indices is selected.
15. A computer program product comprising a non-transitory computer-readable medium storing computer code, the computer code being configured to, upon execution, cause an apparatus to at least
decode an enhancement layer block representing at least a portion of a video frame within a scalable bit stream by:
deriving a coding mode indicator specifying a partition of the enhancement layer block based at least in part on a plurality of reference frame index values corresponding to a plurality of reference layer blocks, the reference layer blocks representing an area in a base layer that, after upsampling, covers the enhancement layer block;
determining the partition's size by merging the reference layer blocks that have the same reference frame index and that have a motion vectors having a difference metric less than a predetermined threshold, wherein the difference metric is derived by a sum of absolute difference of motion vector components;
deriving a motion vector associated with the partition of the enhancement layer block, wherein the derived motion vector comprises a weighted average of the motion vectors of the merged reference layer blocks before merging, wherein weights for the weighted average are based on differentials between the motion vectors of the reference layer blocks before merging, partition sizes associated with the reference layer blocks before merging, or a macroblock type or mode associated with the reference layer blocks before merging; and
decoding the enhancement layer block utilizing the coding mode indicator and the derived motion vector;
wherein the predetermined threshold is equal to one.
16. An apparatus, comprising: a processor; and a memory communicatively connected to the processor and including computer code, the memory and computer code being configured to, with the processor, cause the apparatus to at least:
derive a coding mode indicator specifying a partition of an enhancement layer block based at least in part on a plurality of reference frame index values corresponding to a plurality of reference layer blocks, the reference layer blocks representing an area in a base layer that, after upsampling, covers the enhancement layer block;
determine the partition's size by merging the reference layer blocks that have the same reference frame index and that have motion vectors having a difference metric less than a predetermined threshold, wherein the difference metric is derived by a sum of absolute difference of motion vector components;
derive a motion vector associated with the partition of the enhancement layer block, wherein the derived motion vector comprises a weighted average of the motion vectors of the merged reference layer blocks before merging, wherein weights for the weighted average are based on differentials between the motion vectors of the reference layer blocks before merging, partition sizes associated with the reference layer blocks before merging, or a macroblock type or mode associated with the reference layer blocks before merging; and
decode the enhancement layer block utilizing the coding mode indicatorand the derived motion vector;
wherein the predetermined threshold is equal to one.
17. The apparatus of claim 16 , wherein the merged reference layer blocks have the same motion vectors.
18. The apparatus of claim 16 , wherein a plurality of representative pixels are used to select partition and motion vector information for the respective reference layer block.
19. The apparatus of claim 16 , wherein, if the plurality of reference blocks have different reference frame indices, then the motion vector information from the block with the lowest reference frame indices is selected.
20. An apparatus, comprising:
a processor configured to
derive a coding mode indicator specifying a partition of an enhancement layer block based at least in part on a plurality of reference frame index values corresponding to a plurality of reference layer blocks, the reference layer blocks representing an area in a base layer that, after upsampling, covers the enhancement layer block,
determine the partition's size by merging the reference layer blocks that have the same reference frame index and that have motion vectors having a difference metric less than a predetermined threshold, wherein the difference metric is derived by a sum of absolute difference of motion vector components,
derive a motion vector associated with the partition of the enhancement layer block, wherein the derived motion vector comprises a weighted average of the motion vectors of the merged reference layer blocks before merging, wherein weights for the weighted average are based on differentials between the motion vectors of the reference layer blocks before merging, partition sizes associated with the reference layer blocks before merging, or a macroblock type or mode associated with the reference layer blocks merging; and
a decoder configured to decode the enhancement layer block utilizing the coding mode indicator and the derived motion vector;
wherein the predetermined threshold is equal to one.