Frame-rate up conversion with low complexity
Systems and methods are described for selecting a motion vector (MV) to use in frame-rate up conversion (FRUC) coding of a block of video. In one embodiment, a first set of motion vector candidates is identified for FRUC prediction of the block. A search center is defined based on the first set of motion vector candidates, and a search window is determined, the search window having a selected width and being centered on the search center. A search for a selected MV is performed within the search window. In some embodiments, an initial set of MVs is processed with a clustering algorithm to generate a smaller number of MVs that are used as the first set. The selected MV may be subject to a motion refinement search, which may also be performed over a constrained search range. In additional embodiments, search iterations are constrained to limit complexity.
1. A method comprising:
identifying a first set of motion vector candidates for frame-rate up conversion (FRUC) prediction of a current block in a video;
defining a search center based on a first set of motion vector candidates, where the search center is an average of one or more motion vectors in the first set of motion vector candidates;
determining a search window, the search window having a selected width and being centered on the search center;
processing the first set of motion vector candidates by clipping any motion vector in the first set that falls outside the search window so as to fall within the search window; and
performing a search for a selected motion vector from among the processed first set of motion vector candidates.
2. The method of claim 1 , further comprising:
performing a motion refinement search based on the selected motion vector candidate to generate a refined motion vector; and
predicting the block with frame-rate up conversion using the refined motion vector.
3. The method of claim 1 , wherein identifying the first set of motion vector candidates is performed by a method comprising:
clustering an initial set of motion vector candidates into a plurality of clusters; and
for each cluster, calculating a centroid of the respective cluster and contributing a centroid motion vector representing the centroid of the respective cluster to the first set of motion vector candidates.
4. The method of claim 3 , wherein identifying the first set of motion vector candidates comprises limiting the first set of motion vector candidates to selected maximum number of motion vectors.
5. The method of claim 1 , wherein the search center is one of the motion vectors in the first set of motion vector candidates.
6. The method of claim 1 , wherein the search center is an average of the first set of motion vector candidates.
7. The method of claim 3 , wherein the search center is the center of the cluster having the most motion vector candidates.
8. The method of claim 1 wherein performing a search for a selected motion vector from among the processed first set of motion vector candidates comprises selecting, from among the processed first set of motion vector candidates, a motion vector that achieves a lowest matching cost.
9. The method of claim 1 wherein the current block is a coding unit (CU), further comprising,
performing a motion refinement search based on the selected motion vector candidate to generate a refined CU-level motion vector; and
for each sub-CU in the coding unit:
using the refined CU-level motion vector as a motion vector candidate in a sub-CU-level motion vector search for a selected sub-CU-level motion vector;
refining the sub-CU-level motion vector; and
predicting the sub-CU using the refined sub-CU-level motion vector.
10. The method of claim 1 , wherein the current block is a coding unit (CU), further comprising:
performing a motion refinement search based on the selected motion vector candidate to generate a refined CU-level motion vector; and
for each sub-CU-level block within the coding unit:
identifying a set of sub-CU initial motion vectors including the refined CU-level motion vector;
clustering the sub-CU initial motion vectors into a plurality of clusters, each cluster having an associated centroid motion vector;
processing the centroid motion vectors by clipping any centroid motion vector in the set that falls outside a sub-CU search window so as to fall within the sub-CU search window;
performing a search for a selected sub-CU motion vector from among the processed set of centroid motion vectors;
performing a motion refinement search within the search window to generate a refined sub-CU motion vector; and
predicting the sub-CU-level block with frame-rate up-conversion (FRUC) using the refined sub-CU motion vector.
11. The method of claim 10 , wherein the sub-CU search window is the same as the search window used in determining the CU-level motion vector.
12. The method of claim 1 , further comprising selecting at least one reference picture for frame-rate up conversion (FRUC) prediction of the block, wherein the selected width for the current picture is determined based at least in part on a POC distance between the current picture and the at least one reference picture.
13. The method of claim 1 , further comprising:
performing a motion refinement search based on the selected motion vector candidate to generate a refined motion vector, wherein the motion refinement search is limited to a selected maximum number of iterations.
14. The method of claim 1 , wherein the method is performed by a decoder.
15. A system comprising a processor configured to perform at least:
identifying a first set of motion vector candidates for frame-rate up conversion (FRUC) prediction of a current block in a video;
defining a search center based on a first set of motion vector candidates, where the search center is an average of one or more motion vectors in the first set of motion vector candidates;
determining a search window, the search window having a selected width and being centered on the search center;
processing the first set of motion vector candidates by clipping any motion vector in the first set that falls outside the search window so as to fall within the search window; and
performing a search for a selected motion vector from among the processed first set of motion vector candidates.
16. The system of claim 15 , further configured to perform:
performing a motion refinement search based on the selected motion vector candidate to generate a refined motion vector; and
predicting the block with frame-rate up conversion using the refined motion vector.
17. The system of claim 15 , wherein identifying the first set of motion vector candidates is performed by a method comprising:
clustering an initial set of motion vector candidates into a plurality of clusters; and
for each cluster, calculating a centroid of the respective cluster and contributing a centroid motion vector representing the centroid of the respective cluster to the first set of motion vector candidates.
18. The system of claim 15 wherein performing a search for a selected motion vector from among the processed first set of motion vector candidates comprises selecting, from among the processed first set of motion vector candidates, a motion vector that achieves a lowest matching cost.
19. The system of claim 15 wherein the current block is a coding unit (CU), the processor further being configured to perform:
performing a motion refinement search based on the selected motion vector candidate to generate a refined CU-level motion vector; and
for each sub-CU in the coding unit:
using the refined CU-level motion vector as a motion vector candidate in a sub-CU-level motion vector search for a selected sub-CU-level motion vector;
refining the sub-CU-level motion vector; and
predicting the sub-CU using the refined sub-CU-level motion vector.
20. The system of claim 15 , wherein the current block is a coding unit (CU), further comprising:
performing a motion refinement search based on the selected motion vector candidate to generate a refined CU-level motion vector; and
for each sub-CU-level block within the coding unit:
identifying a set of sub-CU initial motion vectors including the refined CU-level motion vector;
clustering the sub-CU initial motion vectors into a plurality of clusters, each cluster having an associated centroid motion vector;
processing the centroid motion vectors by clipping any centroid motion vector in the set that falls outside a sub-CU search window so as to fall within the sub-CU search window;
performing a search for a selected sub-CU motion vector from among the processed set of centroid motion vectors;
performing a motion refinement search within the search window to generate a refined sub-CU motion vector; and
predicting the sub-CU-level block with frame-rate up-conversion (FRUC) using the refined sub-CU motion vector.