Side motion refinement in video encoding/decoding systems
A method for decoding a current block in a current picture of a video bitstream includes decoding, from the video bitstream, a first motion vector for the current block relative to a first reference block of a first reference picture having a first picture order count, and decoding, from the video bitstream, a second motion vector for the current block relative to a second reference block of a second reference picture having a second picture order count. A similarity metric is generated based on a comparison of the first motion vector and the second motion vector. The method further includes determining whether to refine the first motion vector based on the similarity metric, generating a first refined motion vector from the first motion vector, and performing motion compensation to derive a first reference block from the first reference picture using the first refined motion vector.
1. A method, performed by a decoder, for decoding a current block in a current picture of a video bitstream, the current picture having a current picture order count, the method comprising:
decoding, from the video bitstream, a first motion vector for the current block relative to a first reference block of a first reference picture having a first picture order count;
decoding, from the video bitstream, a second motion vector for the current block relative to a second reference block of a second reference picture having a second picture order count;
generating a similarity metric based on a comparison of the first motion vector and the second motion vector;
determining whether to refine the first motion vector based on the similarity metric;
responsive to determining whether to refine the first motion vector, generating a first refined motion vector from the first motion vector; and
performing motion compensation to derive an updated first reference block from the first reference picture using the first refined motion vector,
wherein the first and second motion vectors each comprise a three-dimensional motion vector including an x-component in a plane of the current picture, a y-component in the plane of the current picture and a z-component, wherein the z-component represents a time component;
wherein the similarity metric measures a similarity between at least one of the x-component, the y-component and the z-component of the first motion vector to a corresponding component of the second motion vector,
wherein generating the similarity metric comprises generating a difference between the z-components of the first and second motion vectors, and
wherein determining whether to refine the first motion vector based on the similarity metric comprises comparing the difference between the z-components of the first and second motion vectors to a third threshold and determining to refine the first motion vector in response to the difference between the z-components of the first and second motion vectors being less than the third threshold.
2. The method of claim 1 , wherein generating the similarity metric comprises generating a difference between the x-components of the first and second motion vectors; and
wherein determining whether to refine the first motion vector based on the similarity metric comprises comparing a difference between the x-components of the first and second motion vectors to a first threshold and determining to refine the first motion vector in response to the difference between the x-components of the first and second motion vectors being less than the first threshold.
3. The method of claim 2 , wherein generating the difference between the x-components of the first and second motion vectors comprises generating a difference between an absolute value of the x-component of the first motion vector and an absolute value of the x-component of the second motion vector.
4. The method of claim 1 , wherein generating the similarity metric comprises generating a difference between the y-components of the first and second motion vectors; and
wherein determining whether to refine the first motion vector based on the similarity metric comprises comparing a difference between the y-components of the first and second motion vectors to a second threshold and determining to refine the first motion vector in response to the difference between the y-components of the first and second motion vectors being less than the second threshold.
5. The method of claim 4 , wherein generating the difference between the y-components of the first and second motion vectors comprises generating a difference between an absolute value of the y-component of the first motion vector and an absolute value of the y-component of the second motion vector.
6. The method of claim 1 , wherein the z-component of the first motion vector comprises a difference between the current picture order count and the first picture order count, and the z-component of the second motion vector comprises a difference between the current picture order count and the second picture order count; and
wherein generating the difference between the z-components of the first and second motion vectors comprises generating a difference between an absolute value of the z-component of the first motion vector and an absolute value of the z-component of the second motion vector.
7. A computer program product comprising a non-transitory computer-readable medium including computer-executable instructions configured to cause a decoder to perform the method according to claim 1 , when the computer-executable instructions are executed on a processor circuit comprised in the decoder.
8. A method, performed by a decoder, for decoding a current block in a current picture of a video bitstream, the current picture having a current picture order count, the method comprising:
decoding, from the video bitstream, a first motion vector for the current block relative to a first reference block of a first reference picture having a first picture order count;
decoding, from the video bitstream, a second motion vector for the current block relative to a second reference block of a second reference picture having a second picture order count;
generating a similarity metric based on a comparison of the first motion vector and the second motion vector;
determining whether to refine the first motion vector based on the similarity metric;
responsive to determining whether to refine the first motion vector, generating a first refined motion vector from the first motion vector; and
performing motion compensation to derive an updated first reference block from the first reference picture using the first refined motion vector,
wherein determining whether to refine the first motion vector based on the similarity metric further comprises determining whether an absolute difference between the current picture order count and the first picture order count is equal to an absolute difference between the current picture order count and the second picture order count.
9. The method of claim 8 , wherein generating the first refined motion vector comprises performing bi-directional optical flow, BIO, processing on the first motion vector.
10. The method of claim 8 , wherein generating the first refined motion vector comprises performing decoder-side motion vector refinement, DMVR, processing on the first motion vector.
11. The method of claim 8 , further comprising:
scaling the first motion vector or the second motion vector based on the first picture order count, the second picture order count and the current picture order count.
12. The method of claim 11 , comprising scaling the first motion vector and the second motion vector in accordance with the following equations:
MV
0
′
.
x
=
M
V
0
.
x
(
POC
1
-
POC
0
)
MV
0
′
.
y
=
M
V
0
.
y
(
POC
1
-
POC
0
)
MV
1
′
.
x
=
M
V
1
.
x
(
POC
2
-
POC
0
)
MV
1
′
.
y
=
M
V
1
.
y
(
POC
2
-
POC
0
)
where MV0.x is the x-component of the first motion vector, MV0.y is the y-component of the first motion vector, MV0′.x is a scaled x-component of the first motion vector, MV0′.y is a scaled y-component of the first motion vector, where MV1.x is the x-component of the second motion vector, MV1.y is the y-component of the second motion vector, MV1′.x is a scaled x-component of the second motion vector, MV1′.y is a scaled y-component of the second motion vector, POC0 is the current picture order count, POC1 is the first picture order count, and POC2 is the second picture order count.
13. A method, performed by a decoder, for decoding a current block in a current picture of a video bitstream, the current picture having a current picture order count, the method comprising:
decoding, from the video bitstream, a first motion vector for the current block relative to a first reference block of a first reference picture having a first picture order count;
decoding, from the video bitstream, a second motion vector for the current block relative to a second reference block of a second reference picture having a second picture order count;
generating a similarity metric based on a comparison of the first motion vector and the second motion vector;
determining whether to refine the first motion vector based on the similarity metric;
responsive to determining whether to refine the first motion vector, generating a first refined motion vector from the first motion vector; and
performing motion compensation to derive an updated first reference block from the first reference picture using the first refined motion vector,
wherein scaling the first motion vector is performed in accordance with the following equations:
MV
0
′
.
x
=
MV
0
.
x
×
(
POC
2
-
POC
0
)
(
POC
1
-
POC
0
)
MV
0
′
.
y
=
M
V
0
.
y
×
(
POC
2
-
POC
0
)
(
POC
1
-
POC
0
)
where MV0.x is the x-component of the first motion vector, MV0.y is the y-component of the first motion vector, MV0′.x is a scaled x-component of the first motion vector, MV0′.y is a scaled y-component of the first motion vector, POC0 is the current picture order count, POC1 is the first picture order count, and POC2 is the second picture order count.
14. A method, performed by an encoder, for encoding a current block in a current picture of a video bitstream, the current picture having a current picture order count, the method comprising:
generating a first motion vector for the current block relative to a first reference block of a first reference picture having a first picture order count;
generating a second motion vector for the current block relative to a second reference block of a second reference picture having a second picture order count;
generating a similarity metric based on a comparison of the first motion vector and the second motion vector;
determining whether to refine the first motion vector based on the similarity metric;
responsive to determining whether to refine the first motion vector, generating a first refined motion vector from the first motion vector; and
performing motion compensation to derive an updated first reference block from the first reference picture using the first refined motion vector,
wherein the first and second motion vectors each comprise a three-dimensional motion vector including an x-component in a plane of the current picture, a y-component in the plane of the current picture and a z-component, wherein the z-component represents a time component;
wherein the similarity metric measures a similarity between at least one of the x-component, the y-component and the z-component of the first motion vector to a corresponding component of the second motion vector,
wherein generating the similarity metric comprises generating a difference between the z-components of the first and second motion vectors, and
wherein determining whether to refine the first motion vector based on the similarity metric comprises comparing the difference between the z-components of the first and second motion vectors to a third threshold and determining to refine the first motion vector in response to the difference between the z-components of the first and second motion vectors being less than the third threshold.
15. The method of claim 14 , wherein generating the similarity metric comprises generating a difference between the x-components of the first and second motion vectors; and
wherein determining whether to refine the first motion vector based on the similarity metric comprises comparing a difference between the x-components of the first and second motion vectors to a first threshold and determining to refine the first motion vector in response to the difference between the x-components of the first and second motion vectors being less than the first threshold.
16. The method of claim 15 , wherein generating the difference between the x-components of the first and second motion vectors comprises generating a difference between an absolute value of the x-component of the first motion vector and an absolute value of the x-component of the second motion vector.
17. The method of claim 14 , wherein generating the similarity metric comprises generating a difference between the y-components of the first and second motion vectors; and
wherein determining whether to refine the first motion vector based on the similarity metric comprises comparing a difference between the y-components of the first and second motion vectors to a second threshold and determining to refine the first motion vector in response to the difference between the y-components of the first and second motion vectors being less than the second threshold.
18. The method of claim 17 , wherein generating the difference between the y-components of the first and second motion vectors comprises generating a difference between an absolute value of the y-component of the first motion vector and an absolute value of the y-component of the second motion vector.
19. The method of claim 14 , wherein the z-component of the first motion vector comprises a difference between the current picture order count and the first picture order count, and the z-component of the second motion vector comprises a difference between the current picture order count and the second picture order count; and
wherein generating the difference between the z-components of the first and second motion vectors comprises generating a difference between an absolute value of the z-component of the first motion vector and an absolute value of the z-component of the second motion vector.
20. The method of claim 14 , wherein generating the first refined motion vector comprises performing bi-directional optical flow, BIO, processing on the first motion vector.
21. An encoder for encoding a current block in a current picture of a video bitstream, the current picture having a current picture order count, the encoder comprising:
a processor circuit; and
a memory coupled with the processor circuit, wherein the memory comprises instructions that when executed by the processor circuit cause the processor circuit to perform operations according to claim 14 .
22. A computer program product comprising a non-transitory computer-readable medium including computer-executable instructions configured to cause an encoder to perform the method according to claim 14 , when the computer-executable instructions are executed on a processor circuit comprised in the encoder.
23. A method, performed by an encoder, for encoding a current block in a current picture of a video bitstream, the current picture having a current picture order count, the method comprising:
generating a first motion vector for the current block relative to a first reference block of a first reference picture having a first picture order count;
generating a second motion vector for the current block relative to a second reference block of a second reference picture having a second picture order count;
generating a similarity metric based on a comparison of the first motion vector and the second motion vector;
determining whether to refine the first motion vector based on the similarity metric;
responsive to determining whether to refine the first motion vector, generating a first refined motion vector from the first motion vector; and
performing motion compensation to derive an updated first reference block from the first reference picture using the first refined motion vector
scaling the first motion vector or the second motion vector based on the first picture order count, the second picture order count and the current picture order count,
wherein scaling the first motion vector comprises is performed in accordance with the following equations:
MV
0
′
.
x
=
MV
0
.
x
×
(
POC
2
-
POC
0
)
(
POC
1
-
POC
0
)
MV
0
′
.
y
=
M
V
0
.
y
×
(
POC
2
-
POC
0
)
(
POC
1
-
POC
0
)
where MV0.x is the x-component of the first motion vector, MV0.y is the y-component of the first motion vector, MV0′.x is a scaled x-component of the first motion vector, MV0′.y is a scaled y-component of the first motion vector, POC0 is the current picture order count, POC1 is the first picture order count, and POC2 is the second picture order count.
24. The method of claim 23 , comprising scaling the first motion vector and the second motion vector in accordance with the following equations:
MV
0
′
.
x
=
M
V
0
.
x
(
POC
1
-
POC
0
)
MV
0
′
.
y
=
M
V
0
.
y
(
POC
1
-
POC
0
)
MV
1
′
.
x
=
M
V
1
.
x
(
POC
2
-
POC
0
)
MV
1
′
.
y
=
M
V
1
.
y
(
POC
2
-
POC
0
)
where MV0.x is the x-component of the first motion vector, MV0.y is the y-component of the first motion vector, MV0′.x is a scaled x-component of the first motion vector, MV0′.y is a scaled y-component of the first motion vector, where MV1.x is the x-component of the second motion vector, MV1.y is the y-component of the second motion vector, MV1′.x is a scaled x-component of the second motion vector, MV1′.y is a scaled y-component of the second motion vector, POC0 is the current picture order count, POC1 is the first picture order count, and POC2 is the second picture order count.
25. A decoder for decoding a current block in a current picture of a video bitstream, the current picture having a current picture order count , the decoder comprising:
a processor circuit; and
a memory coupled with the processor circuit, wherein the memory comprises instructions that when executed by the processor circuit cause the processor circuit to perform operations comprising,
decoding, from the video bitstream, a first motion vector for the current block relative to a first reference block of a first reference picture having a first picture order count;
decoding, from the video bitstream, a second motion vector for the current block relative to a second reference block of a second reference picture having a second picture order count;
generating a similarity metric based on a comparison of the first motion vector and the second motion vector;
determining whether to refine the first motion vector based on the similarity metric;
responsive to determining whether to refine the first motion vector, generating a first refined motion vector from the first motion vector; and
performing motion compensation to derive an updated first reference block from the first reference picture using the first refined motion vector,
wherein the first and second motion vectors each comprise a three-dimensional motion vector including an x-component in a plane of the current picture, a y-component in the plane of the current picture and a z-component, wherein the z-component represents a time component;
wherein the similarity metric measures a similarity between at least one of the x-component, the y-component and the z-component of the first motion vector to a corresponding component of the second motion vector,
wherein generating the similarity metric comprises generating a difference between the z-components of the first and second motion vectors, and
wherein determining whether to refine the first motion vector based on the similarity metric comprises comparing the difference between the z-components of the first and second motion vectors to a third threshold and determining to refine the first motion vector in response to the difference between the z-components of the first and second motion vectors being less than the third threshold.