Increasing motion smoothness using frame interpolation with motion analysis

1. A computer-readable medium storing a computer program with computer executable instructions for causing a computer system programmed thereby to perform a method of video frame synthesis, the method comprising:

estimating global motion between first video data with a first timestamp and second video data with a second timestamp;

checking quality of the global motion before the synthesis of a new video frame;

estimating local motion at an intermediate timestamp between the first and second timestamps to correct the global motion at the intermediate timestamp;

checking quality of the corrected global motion;

using the corrected global motion in motion compensation for the synthesis of the new video frame at the intermediate timestamp; and

outputting the synthesized new video frame at the intermediate timestamp to a buffer for display during playback.

2. The computer-readable medium of claim 1 wherein the global motion comprises a forward warping or a reverse warping.

3. The computer-readable medium of claim 1 wherein an affine warping matrix parameterizes the global motion.

4. The computer-readable medium of claim 1 further comprising:

before the estimating local motion, warping the first video data and the second data to the intermediate timestamp using the global motion.

5. The computer-readable medium of claim 4 wherein the warping comprises warping the first video data forward to the intermediate timestamp and warping the second video data backward to the intermediate timestamp.

6. The computer-readable medium of claim 1 wherein a streaming video client performs the estimating global motion and the estimating local motion in real time.

7. The computer-readable medium of claim 1 wherein delta vectors parameterize the local motion.

8. The computer-readable medium of claim 1 wherein the first video data are a first video frame and the second video data are a second video frame.

9. The computer-readable medium of claim 1 further comprising decomposing each of first and second video frames into plural levels, wherein the first video data are a level for the first video frame, and wherein the second video data are a corresponding level for the second video frame.

10. The computer-readable medium of claim 9 wherein the decomposing includes performing Laplacian pyramid decomposition.

11. The computer-readable medium of claim 9 wherein the estimating global motion includes iteratively refining the global motion through the plural levels.

12. The computer-readable medium of claim 9 wherein the estimating local motion includes iteratively refining the local motion through the plural levels.

13. The computer-readable medium of claim 1 wherein the estimating global motion includes using emphasis selection to improve accuracy and/or increase speed.

14. A computer-readable medium storing a computer program with computer-executable instructions for causing a computer system programmed thereby to perform a method of processing source data samples in a sequence, the method comprising:

estimating global motion between source data samples in a sequence; and

estimating local motion to correct the global motion at a position of an intermediate data sample in the sequence, wherein the intermediate data sample is between the source data samples in the sequence, wherein the estimating local motion includes estimating motion at the position of the intermediate data sample, wherein the estimating local motion further includes computing plural candidate vector sets and switching between the plural candidate vector sets, wherein the plural candidate vector sets include a first set computed after warping according to the global motion and a second set computed after warping according to the global motion and previously estimated local motion;

synthesizing a frame at the position of the intermediate data sample based at least in part on the global and local motion; and

outputting the synthesized frame at the position of the intermediate data sample to a buffer for display during playback.

15. The computer-readable medium of claim 14 wherein the estimating local motion occurs after warping of the source data samples to the position of the intermediate data sample using the global motion.

16. The computer-readable medium of claim 14 wherein the method further comprises de-emphasizing center region points in the estimating global motion.

17. The computer-readable medium of claim 14 wherein the method further comprises emphasizing points with little or no motion in the estimating global motion.

18. The computer-readable medium of claim 14 wherein the method further comprises emphasizing spatial feature points in the estimating global motion.

19. The computer-readable medium of claim 14 wherein the method further comprises checking for a scene change.

20. The computer-readable medium of claim 14 wherein the method further comprises checking for a window around an active area.

21. The computer-readable medium of claim 14 wherein the synthesizing includes selectively feathering to reduce artifacts in the synthesized frame.

22. The computer-readable medium of claim 14 wherein the synthesizing includes reducing ghosting artifacts in the synthesized frame.

23. The computer-readable medium of claim 14 wherein the source data samples are source video frames.

24. The computer-readable medium of claim 14 wherein the source data samples are levels of pyramid decompositions of source video frames.

25. The computer-readable medium of claim 14 wherein the estimating global motion includes iteratively refining the global motion at progressively finer levels of pyramid decompositions of source frames.

26. The computer-readable medium of claim 14 wherein the estimating local motion includes iteratively refining the local motion at progressively finer levels of pyramid decompositions of source frames.

27. The computer-readable medium of claim 14 wherein an affine warping matrix parameterizes the global motion, and wherein the estimating global motion includes:

initializing translation parameters for the affine warping matrix;

refining the translation parameters;

initializing affine parameters for the affine warping matrix based at least in part on the refined translation parameters; and

refining the affine parameters.

28. The computer-readable medium of claim 14 wherein the estimating local motion includes iteratively refining the local motion at progressively finer levels of pyramid decompositions of source frames.

29. The computer-readable medium of claim 14 wherein the switching occurs on a point-by-point basis.

30. A system comprising:

a pyramid decomposition module for decomposing source frames into levels;

a global motion estimation module for estimating global motion between the levels, wherein the global motion estimation module checks quality of the global motion before frame synthesis;

a local motion estimation module for correcting the global motion at an intermediate timestamp, wherein the local motion estimation module checks quality of the corrected global motion before the frame synthesis;

a frame synthesis module for synthesizing a new frame at the intermediate timestamp using the corrected global motion.

31. The system of claim 30 further comprising:

a scene change detection module.

32. The system of claim 30 further comprising:

a window detection module.