Method, apparatus and computer program product for image motion prediction
View Patent ↗A video processing system, method and computer program storage device cooperate to provide a resource and performance efficient sub-pixel motion search operation. Using on the fly calculations it is possible to provide approximations of sub-pixel positions on one image with respect to another image in an image sequence. A sub-pixel position used to set a motion vector for an encoding process may then be identified with reduced processing and memory resources relative to conventional techniques. Also, by later performing a full encoding cost analysis on the earlier identified sub-pixel position, and keeping track of differences, an interpolation accuracy may be dynamically improved by applying the differences to subsequent image frames.
1. A video encoder comprising:
an interface configured to receive pixels of an image and information from a reference image; and
a processor configured to perform a first sub-pixel search including
performing a plurality of dynamic bilinear calculations on said pixels to develop approximated interpolated sub-pixel values,
comparing said approximated sub-pixels to a first threshold to identify a motion vector corresponding to a lowest cost sub-pixel of said approximated interpolated sub-pixels,
comparing the lowest cost sub-pixel of said approximated interpolated sub-pixels to a second threshold;
when the lowest cost sub-pixel of said approximated interpolated sub-pixels exceeds the second threshold, calculating actual interpolated sub-pixel values using a finite impulse response filter; and
when the lowest cost sub-pixel of said approximated interpolated sub-pixels is less than the second threshold, performing a second sub-pixel search different from the first sub-pixel search.
2. The video encoder of claim 1 , wherein the first threshold is a fixed threshold, and the second threshold is a function of a maximum pixel value and a quality parameter.
3. The video encoder of claim 1 , wherein said processor is configured to compare said approximated sub-pixel values to one another to identify the lowest cost sub-pixel.
4. The video encoder of claim 1 , wherein said cost is a function of a sum of absolute differences between said respective sub-pixels in said image frame and corresponding sub-pixels from the reference frame.
5. The video encoder of claim 1 , wherein
said processor is configured to track a difference in cost between the lowest cost subpixel as determined in said comparing and as determined using said finite impulse response filter.
6. The video encoder of claim 5 , wherein the processor reduces an interpolation bias in a motion vector search for a later image by applying the difference to a lowest cost subpixel in the later image having a corresponding subpixel position to said lowest cost subpixel determined by said processor in said comparing.
7. The video encoder of claim 1 , wherein the processor is configured to perform the plurality of dynamic bilinear calculations using bilinear average instructions.
8. A video encoding method comprising:
receiving pixels of an image and information from a reference image;
performing a first sub-pixel search with a processor, said search including
performing a plurality of on the fly bilinear calculations on said pixels to develop approximated interpolated sub-pixel values, and
comparing said approximated sub-pixels to a first threshold to identify a motion vector corresponding to a lowest cost sub-pixel of said approximated interpolated sub-pixels,
comparing the lowest cost sub-pixel of said approximated interpolated sub-pixels to a second threshold;
when the lowest cost sub-pixel of said approximated interpolated sub-pixels exceeds the second threshold, calculating actual interpolated sub-pixel values using a multi-tap filter; and
when the lowest cost sub-pixel of said approximated interpolated sub-pixels is less than the second threshold, performing a second sub-pixel search different from the first sub-pixel search.
9. The method of claim 8 , wherein said comparing includes comparing said approximated interpolated sub-pixels to a fixed threshold.
10. The method of claim 8 , wherein said comparing includes comparing said approximated sub-pixel values to one another to identify the lowest cost sub-pixel.
11. The method of claim 8 , wherein said cost is a function of a sum of absolute differences between said respective sub-pixels in said image frame and corresponding sub-pixels from the reference frame.
12. The method of claim 8 , further comprising:
determining an actual cost of the lowest cost sub-pixel using the multi-tap filter after said processor completes said comparing.
13. The method of claim 8 , further comprising:
reducing an interpolation bias in a motion vector search for a later image by applying the difference to a lowest cost subpixel in the later image having a corresponding subpixel position to said lowest cost subpixel determined said comparing.
14. The method of claim 8 , wherein the performing step includes performing the on the fly calculations using bilinear average instructions.
15. A non-transitory computer readable storage device having instruction stored therein that when executed by a processor perform a method comprising:
receiving pixels of an image and information from a reference image;
performing a sub-pixel search with a processor, said search including
performing a plurality of bilinear calculations on said pixels to develop approximated interpolated sub-pixel values, and
comparing said approximated sub-pixels to a threshold to identify a motion vector corresponding to a lowest cost sub-pixel of said approximated interpolated sub-pixels,
comparing the lowest cost sub-pixel of said approximated interpolated sub-pixels to a second threshold;
when the lowest cost sub-pixel of said approximated interpolated sub-pixels exceeds the second threshold, calculating actual interpolated sub-pixel values using a finite impulse response filter; and
when the lowest cost sub-pixel of said approximated interpolated sub-pixels is less than the second threshold, performing a second sub-pixel search different from the first sub-pixel search.
16. The non-transitory computer readable storage device of claim 15 , wherein said comparing includes
comparing said approximated interpolated sub-pixels to a fixed threshold.
17. The non-transitory computer readable storage device of claim 15 , wherein said comparing includes comparing said approximated sub-pixel values to one another to identify the lowest cost sub-pixel.
18. The non-transitory computer readable storage device of claim 15 , wherein said cost is a function of a sum of absolute differences between said respective sub-pixels in said image frame and corresponding sub-pixels from the reference frame.
19. The non-transitory computer readable storage device of claim 15 , wherein said method further comprising:
determining an actual cost of the lowest cost sub-pixel using the multi-tap filter after said processor completes said comparing.
20. The non-transitory computer readable storage device of claim 19 , wherein said method further comprising:
determining a difference in cost between the lowest cost subpixel as determined in said comparing and as determined using said multi-tap filter.
21. The non-transitory computer readable storage device of claim 20 , wherein said method further comprising:
reducing an interpolation bias in a motion vector search for a later image by applying the difference to a lowest cost subpixel in the later image having a corresponding subpixel position to said lowest cost subpixel determined said comparing.
22. The non-transitory computer readable storage device of claim 15 , wherein the performing step includes performing the bilinear calculations using bilinear average instructions.