System and method for frame rate conversion that utilizes motion estimation and motion compensated temporal interpolation employing embedded video compression
View Patent ↗A technique for frame rate conversion that utilizes motion estimation and motion compensated temporal interpolation includes obtaining a first image and a second image, where the first and second images correspond to different instances in time, compressing the second image using multiple motion vectors that result from motion estimation between the first image and the second image to generate a compressed image, and generating an interpolated image using the compressed image.
1. A method of generating interpolated images, the method comprising:
obtaining at a compression unit a first image and a second image, wherein the first and second images correspond to different instances in time;
compressing the second image with the compression unit using a plurality of motion vectors that result from motion estimation between the first image and the second image to generate a compressed image;
decompressing at least a part of the compressed image, and
generating an interpolated image using the decompressed image;
wherein the interpolated image is generated after the compressed image has been decompressed and before an output image is decoded, without using the reconstructed image.
2. The method of claim 1 , wherein compressing the second image includes, for each group of pixels of the second image, obtaining a corresponding group of pixels of the first image using a motion vector from the plurality of motion vectors, processing the group of pixels of the second image and the corresponding group of pixels of the first image to generate residual data, and compressing the residual data to generate compressed residual data.
3. The method of claim 2 , wherein processing the group of pixels of the second image and the corresponding group of pixels of the first image includes one of subtracting the corresponding group of pixels of the first image from the group of pixels of the second image and subtracting the group of pixels of the second image from the corresponding group of pixels of the first image.
4. The method of claim 2 further comprising buffering the compressed residual data within an integrated circuit while compressing the second image within the integrated circuit.
5. The method of claim 2 , wherein compressing the residual data includes performing differential pulse code modulation on the residual data.
6. The method of claim 2 , wherein compressing the residual data includes performing a transform coding, quantization, and entropy coding on the residual data.
7. The method of claim 1 , wherein decompressing at least a part of the compressed image includes, for groups of pixels of the second image, decompressing compressed residual data to generate reconstructed residual data, obtaining a corresponding group of pixels of the first image using at least one of the motion vectors, and processing the reconstructed residual data and the corresponding group of pixels of the first image to generate a group of pixels of the second image.
8. The method of claim 7 , wherein processing the reconstructed residual data and the corresponding group of pixels of the first image to generate the group of pixels of the second image includes one of adding the reconstructed residual data to the corresponding group of pixels of the first image and subtracting the reconstructed residual data from the corresponding group of pixels of the first image.
9. The method of claim 1 further comprising spatially compressing at least one of the first and second images to generate a spatially compressed image and spatially decompressing the spatially compressed image to reconstruct the at least one of the first and second images.
10. A system for frame rate conversion, comprising:
a compression unit configured to compress a second image using a plurality of motion vectors that result from motion estimation between a first image and the second image to generate a compressed image, wherein the first and second images correspond to different instances in time; and
an interpolator after the compressor and before an output decoder to generate an interpolated image using the compressed image without using a reconstructed image.
11. The system of claim 10 , wherein the compression unit is configured, for each group of pixels of the second image, to obtain a corresponding group of pixels of the first image using a motion vector from the plurality of motion vectors, to subtract the corresponding group of pixels of the first image from the group of pixels of the second image or subtract the group of pixels of the second image from the corresponding group of pixels of the first image to generate residual data, and to compress the residual data to generate compressed residual data.
12. The system of claim 11 further comprising a buffer unit configured to buffer the compressed residual data within an integrated circuit, wherein the compression unit is within the integrated circuit.
13. The system of claim 11 , wherein the compression unit is configured to perform differential pulse code modulation on the residual data.
14. The system of claim 11 , wherein the compression unit is configured to perform a transform coding, quantization, and entropy coding on the residual data.
15. The system of claim 10 further comprising a decompression unit configured to decompress the compressed image using the plurality of motion vectors to reconstruct the second image.
16. A system for frame rate conversion comprising:
a motion estimator configured to perform motion estimation between a first image and a second image to generate a plurality of motion vectors;
a compression unit to compress the second image using the plurality of motion vectors to generate a compressed image, wherein the first and second images correspond to different instances in time; and
motion compensated temporal interpolator after the compression unit and before an output decoder, the motion compensated temporal interpolator being configured to decompress the compressed image using the plurality of motion vectors, without using a reconstructed image, to reconstruct the second image and to perform motion compensated temporal interpolation on the first and second images to generate an interpolated image.
17. The system of claim 16 , wherein the motion estimator is configured, for each group of pixels of the second image, to obtain a corresponding group of pixels of the first image using a motion vector from the plurality of motion vectors, to subtract the corresponding group of pixels of the first image from the group of pixels of the second image or subtract the group of pixels of the second image from the corresponding group of pixels of the first image to generate residual data, and wherein the compressor unit compresses the residual data to generate compressed residual data.
18. the system of claim 16 , wherein the motion compensated temporal interpolator is configured, for each group of pixels of the second image, to decompress compressed residual data to generate reconstructed residual data, to obtain a corresponding group of pixels of the first image using the motion vector, to add the reconstructed residual data to the corresponding group of pixels of the first image or to subtract the reconstructed residual data from the corresponding group of pixels of the first image.
19. The system of claim 16 wherein the compression unit is a spatial compression unit configured to spatially compress at least one of the first and second images to generate a spatially compressed image, and the motion compensated temporal interpolator comprises a spatial decompression unit configured to spatially decompress the spatially compressed image to reconstruct the at least one of the first and second images.