Systems and methods for motion compensated temporal filtering during image processing operations
The present disclosure is directed towards image processing circuitry that applies temporal filtering to video image data along motion trajectories in the video image data. The temporal filtering may be applied along motion trajectories in the image data, by filtering source pixels by reference pixel values and the refined motion vectors. The temporal filtering circuitry may fetch source and reference pixel values based on received motion vectors from an encoding pipeline. Additionally, the temporal filtering circuitry may include a motion vector refinement block along with a temporal filtering block, such that the video image data may be filtered based on refined motion vectors and source and reference pixel values.
1. A video encoding system configured to encode source image data corresponding with an image, comprising:
a low-resolution encoding pipeline and a full-resolution encoding pipeline configured to generate a plurality of motion vectors; and
motion compensated temporal filtering circuitry configured to:
receive the plurality of motion vectors from the low-resolution encoding pipeline and the full-resolution encoding pipeline;
determine a plurality of source pixel values and a plurality of reference pixel values based on the plurality of motion vectors;
generate a plurality of refined motion vectors by refining the plurality of motion vectors based at least in part on the plurality of source pixel values and reference pixel values and low-resolution motion vectors;
generate a plurality of filtered pixel values by filtering the plurality of source pixel values based on determining a weighted combination of the plurality of the reference pixel values and the plurality of source pixel values, wherein the weighted combination comprises a weight value determined based on multiplying a pixel weight by a motion vector weight and a frame weight; and
output the plurality of refined motion vectors and the plurality of filtered pixel values.
2. The video encoding system of claim 1 , wherein the video encoding system is configured to direct the plurality of refined motion vectors and the plurality of filtered pixel values output by the motion compensated temporal filtering circuitry to the motion compensated temporal filtering circuitry to represent neighbor motion vectors in refinement operations.
3. The video encoding system of claim 1 , comprising display circuitry configured to receive the plurality of refined motion vectors and the plurality of filtered pixel values output by the motion compensated temporal filtering circuitry.
4. The video encoding system of claim 1 , wherein the motion compensated temporal filtering circuitry is configured to refine the plurality of motion vectors at least partially by determining a cost for each of the plurality of motion vectors.
5. The video encoding system of claim 4 , wherein the motion compensated temporal filtering circuitry is configured to refine the plurality of motion vectors by:
determining a best candidate motion vector of the plurality of motion vectors based on the determined cost; and
refining the best candidate motion vector around a window size value in sub-pel resolution.
6. The video encoding system of claim 4 , wherein the motion compensated temporal filtering circuitry is configured to determine the cost by calculating a difference between the plurality of source pixels and the plurality of reference pixels.
7. The video encoding system of claim 4 , wherein the motion compensated temporal filtering circuitry is configured to determine the cost by calculating a penalty term that is representative of a smoothness of a current candidate motion vector relative to one or more surrounding motion vectors.
8. The video encoding system of claim 1 , wherein the plurality of motion vectors is divided by source pixel blocks and refined within an increment value.
9. The video encoding system of claim 1 , wherein the weighted combination comprises a weight value determined based on a difference between each of the plurality of source pixels and the plurality of reference pixels.
10. The video encoding system of claim 1 , wherein the motion compensated temporal filtering circuitry comprises:
a source and reference fetch block configured to determine and obtain the plurality of source pixel values and the plurality of reference pixel values;
a motion vector refinement block configured to refine the plurality of motion vectors; and
a temporal filter block configured to generate the plurality of filtered pixel values.
11. A method comprising:
generating, via a low-resolution encoding pipeline and a full-resolution encoding pipeline, a plurality of motion vectors;
determining, via motion compensated temporal filtering circuitry, a plurality of source pixel values and a plurality of reference pixel values based on the plurality of motion vectors;
generating, via the motion compensated temporal filtering circuitry, a plurality of refined motion vectors by refining the plurality of motion vectors based at least in part on the plurality of source pixel values and reference pixel values and low-resolution motion vectors;
generating, via the motion compensated temporal filtering circuitry, a plurality of filtered pixel values by filtering the plurality of source pixel values based on determining a weighted combination of the plurality of the reference pixel values and the plurality of source pixel values, wherein the weighted combination comprises a weight value determined based on multiplying a pixel weigh by a motion vector weight and a frame weight; and
outputting, via the motion compensated temporal filtering circuitry, the plurality of refined motion vectors and the plurality of filtered pixel values.
12. The method of claim 11 , comprising using the refined motion vectors as neighbor motion vectors in refinement operations.
13. The method of claim 11 , comprising sending the plurality of refined motion vectors and the plurality of filtered pixel values to display circuitry.
14. The method of claim 11 , wherein refining the plurality of motion vectors based on the plurality of source pixel values and reference pixel values comprises computing a cost for each of the plurality of motion vectors.
15. The method of claim 14 , wherein refining the plurality of motion vectors based on the plurality of source pixel values and reference pixel values comprises:
determining a best candidate motion vector of the plurality of motion vectors based on the computed cost; and
refining the best candidate motion vector around a window size value in sub-pel resolution.
16. An electronic device comprising:
a display; and
a video encoding system configured to encode source image data corresponding with an image, the video encoding system comprising:
a low-resolution encoding pipeline and a full-resolution encoding pipeline configured to generate a plurality of motion vectors; and
motion compensated temporal filtering circuitry configured to:
receive the plurality of motion vectors from the low-resolution encoding pipeline and the full-resolution encoding pipeline;
determine a plurality of source pixel values and a plurality of reference pixel values based on the plurality of motion vectors;
generate a plurality of refined motion vectors by refining the plurality of motion vectors based at least in part on the plurality of source pixel values and reference pixel values and low-resolution motion vectors;
generate a plurality of filtered pixel values by filtering the plurality of source pixel values based on determining a weighted combination of the plurality of the reference pixel values and the plurality of source pixel values, wherein the weighted combination comprises a weight value determined based on multiplying a pixel weight by a motion vector weight and a frame weight; and
output the plurality of refined motion vectors and the plurality of filtered pixel values.
17. The electronic device of claim 16 , wherein the display is configured to receive the plurality of refined motion vectors and the plurality of filtered pixel values output by the motion compensated temporal filtering circuitry.
18. The video encoding system of claim 1 , wherein refining the plurality of motion vectors is also based at least in part on previously refined motion vectors and full resolution motion vectors.
19. The video encoding system of claim 1 , wherein filtering the plurality of source pixel values is performed on a pixel-by-pixel basis.
20. The video encoding system of claim 1 , wherein the pixel weight and the motion vector weight are each associated with a lookup table.