Adaptation of frame selection for frame rate conversion
View Patent ↗At least a method and a system are described for providing frame rate upconversion by way of using “short side” or bidirectional interpolation in generating interpolated frames. In a representative embodiment, the method may comprise receiving a first frame of video corresponding to a first frame time and a second frame of the video corresponding to a second frame time, computing a first absolute difference between the first frame time and an interpolated frame time wherein the interpolated frame time temporally is located between the first frame time and the second frame time. The method further comprises using a first motion vector field and the first frame to compute a third frame at the interpolated frame time. The method further comprises outputting the third frame as the interpolated frame at the interpolated frame time if the first absolute difference is less than or equal to a first threshold value.
1. A method, comprising:
receiving a first frame of a video corresponding to a first frame time and a second frame of the video corresponding to a second frame time;
using a first motion vector field and the first frame to compute a third frame at an interpolated frame time, wherein the motion vector field is a function of the interpolated frame time and the first frame;
using a second motion vector field and the second frame to compute a fourth frame at the interpolated frame time, wherein the second motion vector field is a function of the interpolated frame time and the second frame;
for pixels located in a border region of an interpolated frame, combining pixel values of corresponding pixels from the third frame and the fourth frame to generate a pixels in a border region of a fifth frame;
for pixels located outside the border region:
computing a first absolute difference between the first frame time and the interpolated frame time, the interpolated frame time temporally located between the first frame time and the second frame time;
selecting corresponding pixels from the third frame for a non-border region of the fifth frame if the first absolute difference is less than a first threshold value;
computing a second absolute difference between the interpolated frame time and the second frame time if the first absolute difference is greater than or equal to the first threshold value;
selecting corresponding pixels from the fourth frame for the non-border region of the fifth frame if the second absolute difference is less than a second threshold value; and
outputting the fifth frame as the interpolated frame.
2. The method of claim 1 , wherein a frame rate is upconverted by a value equal to the number of interpolated frames plus 1.
3. The method of claim 1 , wherein a frame rate of the video is upconverted from 24 Hz to 120 Hz.
4. The method of claim 1 , wherein the first frame and the second frame correspond to successive input frames, the second frame corresponding to a previous frame, and the first frame corresponding to a next frame.
5. The method of claim 1 , wherein combining pixel values comprises computing an average of the pixel values.
6. The method of claim 5 , wherein the average comprises a weighted average of the third frame and the fourth frame based at least in part on temporal distances from the interpolated frame time, the first frame time, and the second frame time.
7. The method of claim 1 , wherein the first frame and the second frame correspond to successive input frames, the first frame corresponding to a previous frame, and the second frame corresponding to a next frame.
8. A non-transitory computer-readable medium embodying a program executable in a computing device, the program configured to at least:
receive a first frame of a video corresponding to a first frame time and a second frame of the video corresponding to a second frame time;
using a first motion vector field and the first frame, compute a third frame at an interpolated frame time, wherein the motion vector field is a function of the interpolated frame time and the first frame;
using a second motion vector field and the second frame, compute a fourth frame at the interpolated frame time, wherein the second motion vector field is a function of the interpolated frame time and the second frame;
for pixels located in a border region of an interpolated frame, combine pixel values of corresponding pixels from the third frame and the fourth frame to generate a pixels in a border region of a fifth frame;
for pixels located outside the border region:
compute a first absolute difference between the first frame time and the interpolated frame time, the interpolated frame time temporally located between the first frame time and the second frame time;
select corresponding pixels from the third frame for a non-border region of the fifth frame if the first absolute difference is less than a first threshold value;
compute a second absolute difference between the interpolated frame time and the second frame time if the first absolute difference is greater than or equal to the first threshold value;
select corresponding pixels from the fourth frame for the non-border region of the fifth frame if the second absolute difference is less than a second threshold value; and
output the fifth frame as the interpolated frame.
9. The non-transitory computer-readable medium of claim 8 , wherein a frame rate is upconverted by a value equal to the number of interpolated frames plus 1.
10. The non-transitory computer-readable medium of claim 8 , wherein a frame rate of the video is upconverted from 24 Hz to 120 Hz.
11. The non-transitory computer-readable medium of claim 8 , wherein the first frame and the second frame correspond to successive input frames, the second frame corresponding to a previous frame, and the first frame corresponding to a next frame.
12. The non-transitory computer-readable medium of claim 8 , wherein combining pixel values comprises computing an average of the pixel values.
13. The non-transitory computer-readable medium of claim 12 , wherein the average comprises a weighted average of the third frame and the fourth frame based at least in part on temporal distances from the interpolated frame time, the first frame time, and the second frame time.
14. The non-transitory computer-readable medium of claim 8 , wherein the first frame and the second frame correspond to successive input frames, the first frame corresponding to a previous frame, and the second frame corresponding to a next frame.
15. A system, comprising:
one or more processors, one or more circuits, or any combination thereof, operable for, at least:
receiving a first frame of a video corresponding to a first frame time and a second frame of the video corresponding to a second frame time;
using a first motion vector field and the first frame to compute a third frame at an interpolated frame time, wherein the motion vector field is a function of the interpolated frame time and the first frame;
using a second motion vector field and the second frame to compute a fourth frame at the interpolated frame time, wherein the second motion vector field is a function of the interpolated frame time and the second frame;
for pixels located in a border region of an interpolated frame, combining pixel values of corresponding pixels from the third frame and the fourth frame to generate a pixels in a border region of a fifth frame;
for pixels located outside the border region:
computing a first absolute difference between the first frame time and the interpolated frame time, the interpolated frame time temporally located between the first frame time and the second frame time;
selecting corresponding pixels from the third frame for a non-border region of the fifth frame if the first absolute difference is less than a first threshold value;
computing a second absolute difference between the interpolated frame time and the second frame time if the first absolute difference is greater than or equal to the first threshold value;
selecting corresponding pixels from the fourth frame for the non-border region of the fifth frame if the second absolute difference is less than a second threshold value; and
outputting the fifth frame as the interpolated frame.
16. The system of claim 15 , wherein a frame rate is upconverted by a value equal to the number of interpolated frames plus 1.
17. The system of claim 15 , wherein a frame rate of the video is upconverted from 24 Hz to 120 Hz.
18. The system of claim 15 , wherein the first frame and the second frame correspond to successive input frames, the second frame corresponding to a previous frame, and the first frame corresponding to a next frame.
19. The system of claim 15 , wherein combining pixel values comprises computing an average of the pixel values.
20. The system of claim 19 , wherein the average comprises a weighted average of the third frame and the fourth frame based at least in part on temporal distances from the interpolated frame time, the first frame time, and the second frame time.