Quality control in frame interpolation with motion analysis
View Patent ↗Techniques and tools for quality control in frame interpolation and motion analysis are described. The techniques and tools may be implemented separately or in combination in software and/or hardware devices for various applications. For example, a media playback device uses quality control in frame interpolation with motion analysis to increase the frame rate of streamed video for playback. The device selectively uses frame synthesis to increase frame rate and quality under normal circumstances, but avoids using frame synthesis when it would not provide suitable quality improvement. The device uses selective ghosting reduction and selective feathering to reduce artifacts in synthesized frames.
1. A method comprising:
receiving a pair of source video frames in a video sequence;
determining whether frame synthesis with motion analysis is on, wherein scene change detection indicates whether the frame synthesis with motion analysis is on;
if the frame synthesis with motion analysis is on, synthesizing pixel values for an intermediate video frame between the source video frames using frame interpolation with motion analysis, wherein frame interpolation with motion analysis includes global motion estimation,
otherwise, synthesizing pixel values for the intermediate video frame using a fallback mode or skipping synthesis of the intermediate video frame; and
providing the source video frames and the intermediate video frame, if any, for display.
2. The method of claim 1 wherein the fallback mode comprises simple averaging of pixel values of the source video frames.
3. The method of claim 1 wherein the fallback mode comprises using only pixel values of a selected one of the source video frames.
4. The method of claim 1 wherein one or more bits in a bitstream signal whether the frame synthesis with motion analysis is on or off.
5. The method of claim 4 wherein the one or more bits are at frame level.
6. The method of claim 4 wherein the one or more bits are at sequence level.
7. The method of claim 4 wherein the one or more bits also signal a count of intermediate video frames between the source video frames.
8. The method of claim 1 wherein quality control measurement indicates whether the frame synthesis with motion analysis is on or off.
9. The method of claim 8 wherein a playback device performs the quality control measurement.
10. The method of claim 8 wherein the quality control measurement is based at least in part on results of the global motion estimation.
11. The method of claim 8 wherein the quality control measurement is based at least in part on results of local motion estimation.
12. The method of claim 1 wherein, for the frame synthesis with motion analysis, the synthesizing includes on a pixel-by-pixel basis determining valid pixels in motion compensated versions of the source video frames.
13. The method of claim 1 wherein, for the frame synthesis with motion analysis, the synthesizing includes feathering.
14. The method of claim 1 wherein, for the frame synthesis with motion analysis, the synthesizing includes ghosting reduction.
15. The method of claim 1 wherein, for the frame synthesis with motion analysis, the synthesizing includes:
detecting an active window;
using the frame interpolation with motion analysis for pixels in the active window; and
using another technique for pixels outside the active window.
16. The method of claim 1 wherein the determining whether frame synthesis with motion analysis is on occurs before the receiving the pair of source video frames.
17. The method of claim 1 wherein the determining whether frame synthesis with motion analysis is on occurs after the receiving the pair of source video frames.
18. The method of claim 1 wherein the frame interpolation with motion analysis includes producing a global warping matrix.
19. The method of claim 18 wherein the global warping matrix is an affine matrix.
20. A computer-readable medium storing computer-executable instructions comprising code for:
receiving one or more bits in a bitstream, wherein the one or more bits signal whether frame synthesis is on or off;
if frame synthesis is on, determining whether frame synthesis with motion analysis is on, wherein scene change detection indicates whether the frame synthesis with motion analysis is on; and
if the frame synthesis with motion analysis is on, synthesizing one or more output frames between source frames using a normal mode, wherein the normal mode comprises frame interpolation with motion analysis including global motion estimation,
otherwise, synthesizing the one or more output frames using a fallback mode or skipping synthesis of the one or more output frames.
21. The computer-readable medium of claim 20 wherein the one or more bits are at frame level in the bitstream.
22. The computer-readable medium of claim 20 wherein the one or more bits are at sequence level in the bitstream.
23. The computer-readable medium of claim 20 wherein the one or more bits also signal a count of output frames between the source frames.
24. The computer-readable medium of claim 20 wherein the frame interpolation with motion analysis further includes local motion estimation to correct the global motion estimation.
25. A computer-readable medium storing computer-executable instructions comprising code for:
receiving a pair of source video frames in a video sequence;
determining whether frame synthesis with motion analysis is on, wherein scene change detection indicates whether the frame synthesis with motion analysis is on; and
synthesizing pixel values for an intermediate video frame between the source video frames using frame interpolation with motion analysis if the frame synthesis with motion analysis is on, wherein the frame interpolation with motion analysis includes global motion estimation,
otherwise, synthesizing pixel values for the intermediate video frame using a fallback mode or skipping synthesis of the intermediate video frame.
26. The computer-readable medium of claim 25 wherein the fallback mode comprises simple averaging of pixel values of the source video frames.
27. The computer-readable medium of claim 25 wherein the fallback mode comprises using only pixel values of a selected one of the source video frames.
28. The computer-readable medium of claim 25 wherein one or more bits in a bitstream signal whether the frame synthesis with motion analysis is on.
29. The computer-readable medium of claim 28 wherein the one or more bits are at frame level.
30. The computer-readable medium of claim 28 wherein the one or more bits are at sequence level.
31. The computer-readable medium of claim 28 wherein the one or more bits also signal a count of intermediate video frames between the source video frames.
32. The computer-readable medium of claim 25 wherein quality control measurement indicates whether the frame synthesis with motion analysis is on.
33. The computer-readable medium of claim 32 wherein a playback device performs the quality control measurement.
34. The computer-readable medium of claim 32 wherein the quality control measurement is based at least in part on results of the global motion estimation.
35. The computer-readable medium of claim 32 wherein the quality control measurement is based at least in part on results of local motion estimation.
36. The computer-readable medium of claim 25 wherein, for the frame synthesis with motion analysis, the synthesizing includes on a pixel-by-pixel basis determining valid pixels in motion compensated versions of the source video frames.
37. The computer-readable medium of claim 25 wherein, for the frame synthesis with motion analysis, the synthesizing includes feathering.
38. The computer-readable medium of claim 25 wherein, for the frame synthesis with motion analysis, the synthesizing includes ghosting reduction.
39. The computer-readable medium of claim 25 wherein, for the frame synthesis with motion analysis, the synthesizing includes:
detecting an active window;
using the frame interpolation with motion analysis for pixels in the active window; and
using another technique for pixels outside the active window.
40. The computer-readable medium of claim 25 wherein the frame interpolation with motion analysis includes producing a global warping matrix.
41. The computer-readable medium of claim 40 wherein the global warping matrix is an affine matrix.
42. The computer-readable medium of claim 25 wherein the frame interpolation with motion analysis further includes correcting the global motion estimation with local motion estimation.
43. A computer-readable medium storing computer-executable instructions comprising code for:
receiving a pair of source video frames in a video sequence;
determining whether frame synthesis with motion analysis is on, wherein scene change detection indicates whether the frame synthesis with motion analysis is on; and
synthesizing pixel values for an intermediate video frame between the source video frames using frame interpolation with motion analysis if the frame synthesis with motion analysis is on,
otherwise, synthesizing pixel values for the intermediate video frame using a fallback mode or skipping synthesis of the intermediate video frame.
44. The computer-readable medium of claim 43 wherein the fallback mode comprises simple averaging of pixel values of the source video frames.
45. The computer-readable medium of claim 43 wherein the fallback mode comprises using only pixel values of a selected one of the source video frames.
46. The computer-readable medium of claim 43 wherein one or more bits in a bitstream signal whether the frame synthesis with motion analysis is on.
47. The computer-readable medium of claim 43 wherein, for the frame synthesis with motion analysis, the synthesizing includes on a pixel-by-pixel basis determining valid pixels in motion compensated versions of the source video frames.
48. The computer-readable medium of claim 43 wherein, for the frame synthesis with motion analysis, the synthesizing includes feathering.
49. The computer-readable medium of claim 43 wherein, for the frame synthesis with motion analysis, the synthesizing includes ghosting reduction.
50. The computer-readable medium of claim 43 wherein, for the frame synthesis with motion analysis, the synthesizing includes:
detecting an active window;
using the frame interpolation with motion analysis for pixels in the active window; and
using another technique for pixels outside the active window.