Template matching with merge mode with motion vector difference
In a method for video decoding in a video decoder, prediction information of a current block in a coded bitstream is decoded. The prediction information indicates that template matching (TM) is applied with a merge mode with motion vector difference (MMVD) mode and indicates motion vector difference (MVD) information of the current block. Using the MMVD mode, one or more MVD candidates is determined based on the MVD information. TM is performed to search for a matched MVD between a current template of the current block and a reference template of a reference block of the current block based on the one or more MVD candidates determined using the MMVD mode. The current block is reconstructed based on a motion vector (MV) that is a summation of a motion vector predictor (MVP) and the matched MVD.
1 . A method for video decoding in a video decoder, comprising:
decoding prediction information of a current block in a coded bitstream, the prediction information indicating whether template matching (TM) is applied with a merge mode with motion vector difference (MMVD) mode and indicating motion vector difference (MVD) information of the current block;
determining, using the MMVD mode, one or more MVD candidates based on the MVD information;
when the prediction information indicates that the TM is applied with the MMVD mode, performing the TM to search for a matched MVD between a current template of the current block and a reference template of a reference block of the current block in a respective search region for each of the one or more MVD candidates determined using the MMVD mode; and
reconstructing the current block based on a motion vector (MV) that is a summation of a motion vector predictor (MVP) and the matched MVD.
2 . The method of claim 1 , wherein
the one or more MVD candidates is one MVD candidate;
the MVD information includes distance information indicating a magnitude of the one MVD candidate and direction information indicating a direction of the one MVD candidate;
the determining the one or more MVD candidates includes determining the one MVD candidate based on the magnitude of the one MVD candidate and the direction of the one MVD candidate; and
the performing the TM includes performing the TM in the search region around the one MVD candidate.
3 . The method of claim 1 , wherein
the coded bitstream further includes a TM flag,
the TM flag indicates that the TM is enabled for the current block, and
which of a plurality of sets of distances is used in the MMVD mode is determined based on whether the TM is enabled for the current block.
4 . The method of claim 2 , wherein
the distance information includes a distance index pointing to a distance in a first plurality of distances, the distance being the magnitude of the one MVD candidate; and
the first plurality of distances is based on a search range of the TM.
5 . The method of claim 4 , wherein
the search range of the TM is in a range of [−L, +L] in a horizontal direction and in a vertical direction, L being a positive integer; and
the smallest distance in the first plurality of distances is equal to L or L/2.
6 . The method of claim 2 , wherein
the distance information includes a distance index pointing to a distance in a first plurality of distances, the distance being the magnitude of the one MVD candidate; and
the first plurality of distances is in an arithmetic progression.
7 . The method of claim 1 , wherein
the one or more MVD candidates include a plurality of MVD candidates that have a same direction;
the MVD information includes direction information indicating the direction of the plurality of MVD candidates;
the determining the one or more MVD candidates includes determining the plurality of MVD candidates based on the direction of the plurality of MVD candidates and a first plurality of distances, the plurality of MVD candidates being ordered based on the first plurality of distances; and
the performing the TM includes:
applying, for each of the plurality of MVD candidates, the TM to the respective search region of the respective MVD candidate to obtain a TM cost and a corresponding updated MVD candidate corresponding to the respective MVD candidate; and
determining the matched MVD based on the TM costs and the corresponding updated MVD candidates.
8 . The method of claim 7 , wherein
the MVD information includes a distance index; and
the performing the TM includes:
reordering the plurality of MVD candidates according to the TM costs of the plurality of MVD candidates; and
determining the matched MVD to be the updated MVD candidate corresponding to the MVD candidate indicated by the distance index in the plurality of reordered MVD candidates.
9 . The method of claim 7 , wherein the performing the TM comprises:
determining the matched MVD to be the updated MVD candidate that corresponds to the smallest TM cost of the TM costs of the plurality of MVD candidates.
10 . The method of claim 1 , wherein
the one or more MVD candidates include a plurality of MVD candidates that have a same magnitude;
the MVD information includes distance information indicating the magnitude of the plurality of MVD candidates;
the determining the one or more MVD candidates includes determining the plurality of MVD candidates based on the magnitude of the plurality of MVD candidates and a plurality of directions, the plurality of MVD candidates being ordered according to the plurality of directions; and
the performing the TM includes:
applying, for each of the plurality of MVD candidates, the TM to the respective search region of the respective MVD candidate to obtain a TM cost and a corresponding updated MVD candidate corresponding to the respective MVD candidate; and
determining the matched MVD based on the TM costs and the corresponding updated MVD candidates.
11 . The method of claim 10 , wherein
the MVD information includes a direction index; and
the performing the TM includes:
reordering the plurality of MVD candidates according to the TM costs of the plurality of MVD candidates; and
determining the matched MVD to be the updated MVD candidate corresponding to the MVD candidate indicated by the direction index in the plurality of reordered MVD candidates.
12 . The method of claim 10 , wherein the performing the TM comprises:
determining the matched MVD to be the updated MVD candidate corresponding to the smallest TM cost of the TM costs of the plurality of MVD candidates.
13 . A method for video encoding in a video encoder, comprising:
determining, using a merge mode with motion vector difference (MMVD) mode, one or more motion vector difference (MVD) candidates of a current block when template matching (TM) is to be applied with the MMVD mode to encode the current block;
when the TM is determined to be applied with the MMVD mode, performing the TM to search for a matched MVD between a current template of the current block and a reference template of a reference block of the current block in a respective search region for each of the one or more MVD candidates determined using the MMVD mode; and
encoding the current block in a bitstream based on a motion vector (MV) that is a summation of a motion vector predictor (MVP) and the matched MVD.
14 . The method of claim 13 , wherein
the one or more MVD candidates is one MVD candidate;
the determining the one or more MVD candidates includes determining the one MVD candidate based on a magnitude of the one MVD candidate that is one of a first plurality of distances and a direction of the one MVD candidate; and
the performing the TM includes performing the TM in the search region around the one MVD candidate.
15 . The method of claim 13 , wherein
the bitstream further includes a TM flag,
the TM flag indicates that the TM is enabled for the current block, and
which of a plurality of sets of distances is used in the MMVD mode is determined based on whether the TM is enabled for the current block.
16 . The method of claim 14 , wherein
the bitstream includes distance information, the distance information including a distance index pointing to a distance in the first plurality of distances, the distance being the magnitude of the one MVD candidate; and
the first plurality of distances is based on a search range of the TM.
17 . The method of claim 16 , wherein
the search range of the TM is in a range of [−L, +L] in a horizontal direction and in a vertical direction, L being a positive integer; and
the smallest distance in the first plurality of distances is equal to L or L/2.
18 . The method of claim 14 , wherein
the bitstream includes distance information, the distance information including a distance index pointing to a distance in the first plurality of distances, the distance being the magnitude of the one MVD candidate; and
the first plurality of distances is in an arithmetic progression.
19 . The method of claim 13 , wherein
the one or more MVD candidates include a plurality of MVD candidates that have a same direction;
the determining the one or more MVD candidates includes determining the plurality of MVD candidates based on the direction of the plurality of MVD candidates and a first plurality of distances, the plurality of MVD candidates being ordered based on the first plurality of distances; and
the performing the TM includes:
applying, for each of the plurality of MVD candidates, the TM to the respective search region of the respective MVD candidate to obtain a TM cost and a corresponding updated MVD candidate corresponding to the respective MVD candidate; and
determining the matched MVD based on the TM costs and the corresponding updated MVD candidates.
20 . A method of processing visual media data, the method comprising:
processing a bitstream that includes the visual media data according to a format rule, wherein
the bitstream includes prediction information of a current block, the prediction information indicating that template matching (TM) is applied with a merge mode with motion vector difference (MMVD) and motion vector difference (MVD) information of the current block; and
the format rule specifies that:
one or more MVD candidates are determined using the MMVD mode based on the MVD information,
when the prediction information indicates that the TM is applied with the MMVD mode, the TM is performed to search for a matched MVD between a current template of the current block and a reference template of a reference block of the current block in a respective search region for each of the one or more MVD candidates determined using the MMVD mode, and
the current block is reconstructed based on a motion vector (MV) that is a summation of a motion vector predictor (MVP) and the matched MVD.