Motion field estimation based on motion trajectory derivation
A motion field estimate determined using motion vector information of two or more reference frames of a current/encoded frame is used to derive a motion vector for inter-prediction of the current/encoded frame. Motion trajectory information, including concatenated motion vectors and locations of the current/encoded frame at which those concatenated motion vectors point, is determined by concatenating motion vectors of the reference frames. A motion field estimate is determined using the motion trajectory information and, in some cases, by interpolating unavailable motion vectors using neighbors. The motion field estimate is used to determine a co-located reference frame for the current/encoded frame, and an inter-prediction process is performed for the current/encoded frame using a motion vector derived using the co-located reference frame. During decoding, the motion field estimate may be determined using motion vectors signaled within a bitstream and without additional side information, thereby improving prediction coding efficiency.
1. A non-transitory computer readable medium having stored thereon an encoded bitstream, wherein the encoded bitstream is configured for decoding by operations comprising:
determining a concatenated motion vector representing a motion trajectory intersecting a current frame, a first reference frame of the current frame, and a second reference frame of the current frame by concatenating one or more motion vectors signaled within the encoded bitstream and associated with the first reference frame and the second reference frame;
determining a motion field estimate for decoding the current frame using the concatenated motion vector;
interpolating a co-located reference frame for decoding the current frame using motion information from the motion field estimate and pixel information from the first reference frame and the second reference frame, wherein the co-located reference frame is co-located with the current frame; and
performing an inter-prediction process for decoding the current frame using a motion vector derived from the co-located reference frame.
2. The non-transitory computer readable medium of claim 1 , wherein determining the concatenated motion vector comprises:
concatenating a first motion vector pointing from the first reference frame toward the second reference frame and a second motion vector pointing from the second reference frame toward the current frame, wherein the motion trajectory corresponds to the first motion vector and the second motion vector, and wherein the second motion vector is previously derived between the second reference frame and a third reference frame.
3. The non-transitory computer readable medium of claim 1 , wherein determining the concatenated motion vector comprises:
determining an interpolated motion vector from a first motion vector pointing from the first reference frame toward a third reference frame and a second motion vector pointing from the second reference frame toward the third reference frame, wherein the motion trajectory corresponds to the interpolated motion vector.
4. The non-transitory computer readable medium of claim 1 , further comprising:
evaluating a quality measurement of the motion vector; and
deriving the motion vector based on the evaluating.
5. The non-transitory computer readable medium of claim 4 , wherein quality measurements are evaluated for multiple motion vectors of a pixel of the co-located reference frame, wherein the motion vector is derived responsive to determining that the quality measurement evaluated for the motion vector is a highest one of the quality measurements.
6. The non-transitory computer readable medium of claim 1 , wherein determining the motion field estimate for decoding the current frame using the concatenated motion vector comprises:
responsive to determining that a given motion vector is unavailable at a location of the motion field estimate, interpolating the given motion vector using one or more neighboring motion vectors within the motion field estimate.
7. The non-transitory computer readable medium of claim 6 , wherein the one or more neighboring motion vectors are weighted according to a relative importance for interpolating the given motion vector.
8. The non-transitory computer readable medium of claim 1 , further comprising:
decoding one or more syntax elements indicating whether to determine the motion field estimate using motion vector information of one or more reference frames, wherein the concatenated motion vector is determined responsive to the one or more syntax elements indicating to determine the motion field estimate using the motion vector information.
9. The non-transitory computer readable medium of claim 8 , further comprising:
responsive to determining that the one or more syntax elements do not indicate to determine the motion field estimate using the motion vector information, performing the inter-prediction process for decoding the current frame using an optical flow reference frame generated based on an optical flow estimation of the current frame, the first reference frame, and the second reference frame.
10. The non-transitory computer readable medium of claim 9 , wherein the optical flow estimation is performed by minimizing a Lagrangian function for respective pixels of the first reference frame.
11. A non-transitory computer readable medium having stored thereon an encoded bitstream, wherein the encoded bitstream is configured for decoding by operations comprising:
determining a concatenated motion vector representing a motion trajectory intersecting a current frame, a first reference frame of the current frame, and a second reference frame of the current frame by concatenating motion vectors signaled within the bitstream and associated with the first reference frame and the second reference frame;
interpolating a co-located reference frame for decoding the current frame using motion information from a motion field estimate determined based on the concatenated motion vector and using pixel information from the first reference frame and the second reference frame, wherein the co-located reference frame is co-located with the current frame; and
performing an inter-prediction process for decoding the current frame using a motion vector derived using the co-located reference frame.
12. The non-transitory computer readable medium of claim 11 , wherein interpolating the co-located reference frame comprises:
determining the motion field estimate using the concatenated motion vector; and
determining the co-located reference frame by interpolating the motion information using the motion field estimate.
13. The non-transitory computer readable medium of claim 12 , further comprising:
using the motion field estimate for motion vector prediction.
14. The non-transitory computer readable medium of claim 11 , wherein, when the motion trajectory is a non-linear motion trajectory, the co-located reference frame is used to adjust an offset between the first reference frame and the second reference frame.
15. The non-transitory computer readable medium of claim 11 , wherein the motion vectors associated with the first reference frame and the second reference frame include one or more motion vectors associated with the first reference frame and one or more motion vectors associated with the second reference frame.
16. A non-transitory computer readable medium having stored thereon an encoded bitstream, wherein the encoded bitstream is configured for decoding by operations comprising:
determining a motion field estimate for decoding a current frame using a concatenated motion vector determined by concatenating motion vectors signaled within the encoded bitstream and associated with a first reference frame of the current frame and a second reference frame of the current frame, the concatenated motion vector representing a motion trajectory intersecting the first reference frame, the second reference frame, and the current frame;
interpolating a co-located reference frame for decoding the current frame using motion information from the motion field estimate and pixel information from the first reference frame and the second reference frame, wherein the co-located reference frame is co-located with the current frame; and
performing an inter-prediction process for decoding the current frame using a motion vector derived using the co-located reference frame.
17. The non-transitory computer readable medium of claim 16 , wherein determining the motion field estimate for decoding the current frame using the concatenated motion vector comprises:
determining the concatenated motion vector by concatenating one or more motion vectors of a first set of motion vectors between the first reference frame and the current frame and one or more motion vectors of a second set of motion vectors between the second reference frame and the current frame.
18. The non-transitory computer readable medium of claim 17 , wherein the concatenated motion vector is determined by concatenating the one or more motion vectors of the first set of motion vectors, the one or more motion vectors of the second set of motion vectors, and one or more motion vectors of a third set of motion vectors between the current frame and a third reference frame.
19. The non-transitory computer readable medium of claim 16 , wherein interpolating a co-located reference frame comprises:
interpolating the co-located reference frame directly from the motion field estimate using the first reference frame and the second reference frame.
20. The non-transitory computer readable medium of claim 16 , wherein the co-located reference frame is used to adjust an offset between the first reference frame and the second reference frame.