System and method for signaling of motion merge modes in video coding

1. A computer implemented method for motion merge modes, comprising:

encoding, by an encoder, a regular merge flag for a coding unit (CU) that is coded as merge mode and merge related modes;

in response to determining that the regular merge flag is encoded as one, indicating that a regular merge mode or merge mode with motion vector differences (MMVD) is used by the CU, constructing, by the encoder, a motion vector merge list for the CU and using regular merge index to indicate which candidate is used; and

in response to determining that the regular merge flag is encoded as zero, indicating the regular merge mode is not used by the CU, and further encoding, by the encoder, mode flags to indicate associated merge related modes are used in response to determining that a mode flag's constraints are met.

2. The method of claim 1 , further comprising:

obtaining, by the encoder, a MMVD flag;

signaling, by the encoder, a MMVD merge flag, MMVD distance index, and MMVD direction index in response to determining that the MMVD flag is equal to one.

3. The method of claim 1 , further comprising encoding a subblock flag and the subblock flag's constraints includes:

generating, by the encoder, a coding block, wherein the coding block has a width and height;

specifying, by the encoder, whether maximum number of subblock-based merging MVP candidates (MaxNumSubblockMergeCand) is greater than zero;

specifying, by the encoder, whether coding block width is greater than equal to eight; and

specifying, by the encoder, whether coding block height is greater than equal to eight.

4. The method of claim 1 , wherein the mode flag is a combined inter and intra prediction (CIIP) flag and the CIIP flag's constraints include:

generating, by the encoder, a coding block, wherein the coding block has a width and height;

specifying, by the encoder, whether sps_mh_intra_enabled_flag is set;

specifying, by the encoder, whether cu_skip_flag is equal to zero;

specifying, by the encoder, whether the coding block width times the coding block height is greater than equal to sixty-four;

specifying, by the encoder, whether coding block width is less than one hundred and twenty-eight; and

specifying, by the encoder, whether coding block height is less than one hundred and twenty-eight.

5. The method of claim 1 , further comprising:

signaling, by the encoder, a regular merge flag before signaling, by the encoder, the mode flag in response to determining that the mode flag's constraints are met.

6. The method of claim 5 , further comprising:

signaling, by the encoder, a regular merge flag before signaling, by the encoder, a merge mode with motion vector differences (MMVD) flag in response to determining that the merge mode with motion vector differences (MMVD) flag's constraints are met.

7. The method of claim 5 , further comprising:

signaling, by the encoder, a regular merge flag before signaling, by the encoder, a combined inter and intra prediction (CIIP) flag in response to determining that the CIIP flag's constraints are met.

8. The method of claim 1 , further comprising:

signaling, by the encoder, a regular merge flag after signaling, by the encoder, a mode flag in response to determining that the mode flag's constraints are met.

9. The method of claim 8 , further comprising:

signaling, by the encoder, a regular merge flag after signaling, by the encoder, a subblock merge mode flag in response to determining that the subblock merge mode flag's constraints are met.

10. The method of claim 1 , further comprising:

signaling, by the encoder, the regular merge flag using Context-adaptive binary arithmetic coding (CABAC) with multiple context models and the selection of the context model is based on coded information.

11. A computing device, comprising:

one or more processors;

a non-transitory computer-readable memory storing instructions executable by the one or more processors, wherein the one or more processors are configured to:

encode, by an encoder, a regular merge flag for a coding unit (CU) that is coded as merge mode and merge related modes;

in response to determining that the regular merge flag is encoded as one, indicating that a regular merge mode or merge mode with motion vector differences (MMVD) is used by the CU, construct, by an encoder, a motion vector merge list for the CU and using regular merge index to indicate which candidate is used; and

in response to determining that the regular merge flag is encoded as zero, indicating the regular merge mode is not used by the CU, further encode mode flags to indicate associated merge related modes are used in response to determining that a mode flag's constraints are met.

12. The computing device of claim 11 , wherein the one or more processors are configured to:

obtain, by the encoder, a MMVD flag,

signal, by the encoder a MMVD merge flag, MMVD distance index, and MMVD direction index in response to determining that the MMVD flag is equal to one.

13. The computing device of claim 11 , wherein the one or more processors are further configured to signal a subblock flag and the subblock flag's constraints includes:

generating, by the encoder, a coding block, wherein the coding block has a width and height;

specify, by the encoder, whether maximum number of subblock-based merging MVP candidates (MaxNumSubblockMergeCand) is greater than zero;

specify, by the encoder, whether coding block width is greater than equal to eight; and

specify, by the encoder, whether coding block height is greater than equal to eight.

14. The computing device of claim 11 , wherein the mode flag is a combined inter and intra prediction (CIIP) flag and the CIIP flag's constraints include:

generate, by the encoder, a coding block, wherein the coding block has a width and height;

specify, by the encoder, whether sps_mh_intra_enabled_flag is set;

specify, by the encoder, whether cu_skip_flag is equal to zero;

specify, by the encoder, whether the coding block width times the coding block height is greater than equal to sixty-four;

specify, by the encoder, whether coding block width is less than one hundred and twenty-eight; and

specify, by the encoder, whether coding block height is less than one hundred and twenty-eight.

15. The computing device of claim 11 , wherein the one or more processors are configured to:

signal, by the encoder, a regular merge flag before signaling, by the encoder, the mode flag in response to determining that the mode flag's constraints are met.

16. The computing device of claim 15 , wherein the one or more processors are configured to:

signal, by the encoder, a regular merge flag before signaling, by the encoder, a combined inter and intra prediction (CIIP) flag in response to determining that the CIIP flag's constraints are met.

17. The computing device of claim 11 , wherein the one or more processors are configured to:

signal, by the encoder, a regular merge flag after signaling, by the encoder, a mode flag in response to determining that the mode flag's constraints are met.

18. The computing device of claim 17 , wherein the one or more processors are configured to:

signal, by the encoder, a regular merge flag after signaling, by the encoder, a subblock merge mode flag in response to determining that the subblock merge mode flag's constraints are met.

19. The computing device of claim 11 , wherein the one or more processors are configured to:

signal, by the encoder, the regular merge flag using Context-adaptive binary arithmetic coding (CABAC) with multiple context models and the selection of the context model is based on coded information.

20. A non-transitory computer-readable storage medium storing a plurality of programs for execution by a computing device having one or more processors, wherein the plurality of programs when executed by the one or more processors, cause the computing device to perform operations including:

encoding, by an encoder, a regular merge flag for a coding unit (CU) that is coded as merge mode and merge related modes;

in response to determining that the regular merge flag is encoded as one, indicating that a regular merge mode or merge mode with motion vector differences (MMVD) is used by the CU, constructing, by the encoder, a motion vector merge list for the CU and using regular merge index to indicate which candidate is used; and

in response to determining that the regular merge flag is encoded as zero, indicating the regular merge mode is not used by the CU, and further encoding, by the encoder, mode flags to indicate associated merge related modes are used in response to determining that a mode flag's constraints are met.