Virtual merge candidates

1. A video processing method, comprising:

generating, for a conversion between a video block in a video picture and a bitstream of the video block, a list of merge candidates for the video block, wherein the list of merge candidates includes at least a first merge candidate that is a virtual merge candidate derived by modifying a motion vector and a reference picture of a second merge candidate, wherein the modifying includes scaling the motion vector to a different reference picture; and

performing, using the list of merge candidates, the conversion between the video block and the video bitstream.

2. The method of claim 1 , wherein the virtual merge candidate inherits a prediction direction from the second merge candidate.

3. The method of claim 1 , wherein the second merge candidate is derived using a bi-prediction in which two reference picture lists are used, and wherein the virtual merge candidate is derived by modifying motion information associated only one of the two reference picture lists, or

the second merge candidate is derived using a uni-prediction in which only one reference picture list is used, and wherein the virtual merge candidate uses a bi-prediction in which two reference picture lists are used.

4. The method of claim 3 , wherein, for a first reference picture list of the two reference picture lists, the virtual merge candidate inherits a first reference picture associated with the second merge candidate that is in the first reference picture list, and/or for a second reference picture list of the two reference picture lists, the virtual merge candidate is derived by scaling a motion vector with respect to a second reference picture that is in the second reference picture list.

5. The method of claim 1 , wherein the list of merge candidates includes multiple virtual merge candidates derived based on the second merge candidate with respect to multiple corresponding reference pictures, or derived based on multiple corresponding merge candidates.

6. The method of claim 1 , comprising:

selecting, based on a first criterion, based on a second criterion, or based on a third criterion, one or more merge candidates for deriving the virtual merge candidate.

7. The method of claim 6 , wherein the first criterion includes at least one of:

an insertion order of the one or more merge candidates into the list;

whether a merge candidate corresponds to bi-prediction;

a type of a merge candidate, wherein the type indicates whether the merge candidate is a spatial merge candidate, a temporal merge candidate, a combined bi-prediction merge candidate, a zero motion vector merge candidate, or a sub-block merge candidate;

a coded mode of a merge candidate, wherein the coded mode indicates whether the merge candidate is coded using affine motion, non-affine motion, illumination compensation, or non-illumination compensation;

a size of the video block;

a shape of the video block; or

a coding mode of a neighboring block.

8. The method of claim 7 , comprising:

selecting one or more spatial merge candidates for deriving the virtual merge candidate, or

selecting one or more merge candidates coded using non-affine motion or non-illumination compensation for deriving the virtual merge candidate.

9. The method of claim 6 , wherein the second criterion includes at least one of:

an index of a reference picture;

a picture order count different between a first reference picture and a second reference picture associated with the second merge candidate;

a quantization parameter different between the first reference picture and the second reference picture associated with the second merge candidate; or

a number of temporal layers of a reference picture.

10. The method of claim 1 , wherein the virtual merge candidate is placed at a first position in the list of merge candidate list.

11. The method of claim 10 , wherein the first position includes one of: before temporal motion vector prediction (TMVP) candidates, after TMVP candidates and before combined bi-predictive merge candidates, or after combined bi-predictive merge candidates.

12. The method of claim 10 , wherein a bi-prediction based merge candidate has a higher priority than a uni-prediction based merge candidate.

13. The method of claim 1 , wherein multiple virtual merge candidates are derived by modifying the motion vector of the second merge candidate based on multiple reference pictures, and wherein the multiple reference pictures are ordered by a third criterion.

14. The method of claim 13 , wherein the third criterion includes at least one of:

an increasing order of a reference picture index;

a sequential order of checking pictures in different picture lists;

an interleaved order of checking pictures in different picture lists; or

a combination thereof.

15. The method of claim 1 , further comprising:

comparing sub-block motion candidate against at least a subset of existing candidates;

determining that the sub-block motion candidate is identical to an existing candidate; and

refraining, due to the determining, from adding the sub-block motion candidate to a list of merge candidates.

16. The method of claim 1 , wherein the conversion includes decoding the video bitstream to generate the current video block.

17. The method of claim 1 , wherein the conversion includes encoding the current video block to generate the video bitstream.

18. A video encoding apparatus comprising a processor configured to implement a method of video processing, comprising:

generating, for a conversion between a video block in a video picture and a bitstream of the video block, a list of merge candidates for the video block, wherein the list of merge candidates includes at least a first merge candidate that is a virtual merge candidate derived by modifying a motion vector and a reference picture of a second merge candidate, wherein the modifying includes scaling the motion vector to a different reference picture; and

performing, using the list of merge candidates, the conversion between the video block and the video bitstream.

19. A video decoding apparatus comprising a processor configured to implement a method of video processing comprising:

generating, for a conversion between a video block in a video picture and a bitstream of the video block, a list of merge candidates for the video block, wherein the list of merge candidates includes at least a first merge candidate that is a virtual merge candidate derived by modifying a motion vector and a reference picture of a second merge candidate, wherein the modifying includes scaling the motion vector to a different reference picture; and

performing, using the list of merge candidates, the conversion between the video block and the video bitstream.

20. A non-transitory computer-readable program medium having code stored thereupon, the code comprising instructions that, when executed by a processor, causing the processor to implement a method of video processing, comprising:

generating, for a conversion between a video block in a video picture and a bitstream of the video block, a list of merge candidates for the video block, wherein the list of merge candidates includes at least a first merge candidate that is a virtual merge candidate derived by modifying a motion vector and a reference picture of a second merge candidate, wherein the modifying includes scaling the motion vector to a different reference picture; and

performing, using the list of merge candidates, the conversion between the video block and the video bitstream.