Motion compensated bi-prediction based on local illumination compensation
Systems, methods, and Instrumentalities are described herein for calculating local Illumination compensation (LIC) parameters for bi-predicted coding unit (CU). The LIC parameters may be used to generate adjusted samples for the current CU and to address local illumination changes that may exist among temporal neighboring pictures. LIC parameters may be calculated based on bi-predicted reference template samples and template samples for a current CU. Bi-predicted reference template samples may be generated based on reference template samples neighboring temporal reference CUs. For example, the bi-predicted reference template samples may be generated based on averaging the reference template samples. The reference template samples may correspond to template samples for the current CU. A CU may be or may include a coding block and/or a sub-block that may be derived by dividing the coding block.
1. A method for coding comprising:
obtaining a current coding unit (CU) in a video bitstream;
determining whether the current CU is bi-predicted;
based on a determination that the current CU is bi-predicted, identifying a first set of reference template samples neighboring a first temporal reference CU and a second set of reference template samples neighboring a second temporal reference CU, the first set of reference template samples and the second set of reference template samples corresponding to a set of template samples neighboring the current CU;
obtaining a set of bi-predicted reference template samples based on the identified first set of reference template samples and the identified second set of reference template samples;
obtaining a local illumination compensation (LIC) scaling factor based on the obtained set of bi-predicted reference template samples and the set of template samples for the current CU; and
obtaining an adjusted sample within the current CU based on the obtained LIC scaling factor.
2. The method of claim 1 , wherein the current CU comprises a current block or a current sub-block.
3. The method of claim 1 , wherein obtaining the set of bi-predicted reference template samples comprises averaging the first set of reference template samples neighboring the first temporal reference CU and the second set of reference template samples neighboring the second temporal reference CU.
4. The method of claim 1 , wherein the first set of reference template samples neighboring the first temporal reference CU and the second set of reference template samples neighboring the second temporal reference CU are identified based on motion vectors of the current CU.
5. The method of claim 1 , wherein the LIC scaling factor is obtained based on minimizing differences between the set of bi-predicted reference template samples and the set of template samples for the current CU.
6. The method of claim 5 , wherein the differences between the set of bi-predicted reference template samples and the set of template samples for the current CU are minimized based on a least mean square error (LMSE) approach.
7. The method of claim 1 , wherein the LIC scaling factor is obtained based on a linear model approach using a minimum value and a maximum value associated with the set of bi-predicted reference template samples and the set of template samples for the current CU.
8. The method of claim 1 , further comprising:
determining whether a sub-block of the current CU is at a boundary of the current CU;
based on a determination that the sub-block of the current CU is at the boundary of the current CU, identifying a plurality of neighboring sub-blocks of the sub-block;
applying a plurality of motion vectors associated with the identified plurality of neighboring sub-blocks to a sample in the sub-block to derive a plurality of template reference samples;
obtaining an overlapped block motion compensation (OBMC) predicted sample based on the plurality of template reference samples; and
applying the obtained LIC scaling factor to the OBMC predicted sample.
9. The method of claim 1 , wherein the set of template samples for the current CU comprises a set of neighboring samples of the current CU.
10. The method of claim 1 , wherein an LIC parameter comprises the LIC scaling factor and an LIC offset, and wherein the method comprises obtaining the LIC offset based on obtained set of bi-predicted reference template samples and the set of template samples for the current CU.
11. A video coding device comprising:
a processor configured to:
obtain a current coding unit (CU) in a video bitstream;
determine whether the current CU is bi-predicted;
based on a determination that the current CU is bi-predicted, identify a first set of reference template samples neighboring a first temporal reference CU and a second set of reference template samples neighboring a second temporal reference CU, the first set of reference template samples and the second set of reference template samples corresponding to a set of template samples neighboring the current CU;
obtain a set of bi-predicted reference template samples based on the identified first set of reference template samples and the identified second set of reference template samples;
obtain a local illumination compensation (LIC) scaling factor based on the obtained set of bi-predicted reference template samples and the set of template samples for the current CU; and
obtain an adjusted sample within the current CU based on the obtained LIC scaling factor.
12. The video coding device of claim 11 , wherein the current CU comprises a current block or a current sub-block.
13. The video coding device of claim 11 , wherein the processor configured to obtain the set of bi-predicted reference template samples comprises the processor configured to average the first set of reference template samples neighboring the first temporal reference CU and the second set of reference template samples neighboring the second temporal reference CU.
14. The video coding device of claim 11 , wherein the first set of reference template samples neighboring the first temporal reference CU and the second set of reference template samples neighboring the second temporal reference CU are identified based on motion vectors of the current CU.
15. The video coding device of claim 11 , wherein the LIC scaling factor is obtained based on the processor configured to minimize differences between the set of bi-predicted reference template samples and the set of template samples for the current CU.
16. The video coding device of claim 15 , wherein the differences between the set of bi-predicted reference template samples and the set of template samples for the current CU are minimized based on a least mean square error (LMSE) approach.
17. The video coding device of claim 11 , wherein the LIC scaling factor is obtained based on the processor configured to use a linear model approach using a minimum value and a maximum value associated with the set of bi-predicted reference template samples and the set of template samples for the current CU.
18. The video coding device of claim 11 , wherein the processor is further configured to:
determine whether a sub-block of the current CU is at a boundary of the current CU;
based on a determination that the sub-block of the current CU is at the boundary of the current CU, identify a plurality of neighboring sub-blocks of the sub-block;
apply a plurality of motion vectors associated with the identified plurality of neighboring sub-blocks to a sample in the sub-block to derive a plurality of template reference samples;
obtain an overlapped block motion compensation (OBMC) predicted sample based on the plurality of template reference samples; and
apply the obtained LIC scaling factor to the OBMC predicted sample.
19. The video coding device of claim 11 , wherein the set of template samples for the current CU comprises a set of neighboring samples of the current CU.
20. The video coding device of claim 11 , wherein an LIC parameter comprises the LIC scaling factor and an LIC offset, and wherein the processor is configured to obtain the LIC offset based on obtained set of bi-predicted reference template samples and the set of template samples for the current CU.