Extracting key frame candidates from video clip
View Patent ↗A method of analyzing a digital video clip to determine candidate frames for subsequent key frame selection, includes: performing a global motion estimate on the video clip that indicates translation of a scene or camera, or scaling of the scene; and forming a plurality of video segments based on the global motion estimate and labeling each segment in accordance with a predetermined series of camera motion classes. The method further includes extracting key frame candidates from the labeled segments and computing a confidence score for each candidate by using rules corresponding to each camera motion class and a rule corresponding to object motion. An important feature of the invention is that the rules corresponding to camera motion include extracting a candidate frame for a pan segment with a confidence score related to a translation amount during the pan, and not extracting candidate frames for object motion based frames for fast pan and zoom segments, and the rule corresponding to object motion includes extracting a candidate frame for a fixed segment with a confidence score related to the segment length.
1. A method of analyzing a digital video clip to determine candidate frames for subsequent key frame selection, comprising:
a. performing a global motion estimate on the video clip that indicates translation of a scene or camera, or scaling of the scene;
b. forming a plurality of video segments based on the global motion estimate and labeling each segment in accordance with a predetermined series of camera motion classes;
c. extracting key frame candidates from the labeled segments and computing a confidence score for each candidate by using rules corresponding to each camera motion class and a rule corresponding to object motion; and
d. wherein, the rules corresponding to camera motion include extracting a candidate frame for a pan segment with a confidence score related to a translation amount during the pan, and not extracting candidate frames for object motion based frames for fast pan and zoom segments, and the rule corresponding to object motion includes extracting a candidate frame for a fixed segment with a confidence score related to the segment length.
2. The method of claim 1 wherein the predetermined camera motion classes include pan (left or right, and tilt up or down), zoom (in or out), fast pan or fixed.
3. The method of claim 2 wherein the rules include a pan rule, a zoom rule, a fast pan rule and a fixed rule.
4. The method of claim 3 wherein the pan rule includes extracting a plurality of frames to cover the space of environment while reducing the spatial overlap among the frames from a pan segment.
5. The method of claim 3 wherein the pan rule includes extracting a frame located at a point when the pan motion is slowed down.
6. The method of claim 3 wherein the zoom rule includes extracting a candidate frame at an endpoint of the zoom-in or zoom-out segment.
7. The method of claim 3 wherein the fast pan rule includes not extracting a candidate frame from a fast pan segment.
8. The method of claim 3 wherein the fixed rule includes extracting a candidate frame located at a midpoint of a fixed segment.
9. A method of analyzing a digital video clip to determine key frames, comprising:
a. performing a global motion estimate on the video clip that indicates translation of a scene or camera, or scaling of the scene;
b. forming a plurality of video segments based on the global motion estimate and labeling each segment in accordance with a predetermined series of classes;
c. extracting key frame candidates from the labeled segments and computing a confidence score for each candidate by using rules associated with each label;
d. wherein, the rules for labels corresponding to camera motion include extracting a candidate frame for a pan segment with a confidence score related to a translation amount during the pan, and not extracting candidate frames for object motion based frames for fast pan and zoom segments, and the rule for labels corresponding to object motion includes extracting a candidate frame for a fixed segment with a confidence score related to the segment length; and
e. selecting key frames from the candidate frames based on the confidence score of each candidate.
10. The method of claim 9 further including ranking the selected key frames in accordance with the confidence score.
11. The method of claim 10 wherein step d includes employing the ranking and a user specified number to select the key frames.
12. The method of claim 11 wherein employing the ranking and a user specified number to select the key frames includes selecting at least one key frame from each segment if there are confidence scores above a predetermined threshold.
13. The method of claim 12 wherein employing the ranking and a user specified number to select the key frames includes selecting key frames from the remaining candidates with the highest confidence values to fill the user specified number of key frames.
14. The method of claim 9 wherein the predetermined series of classes include pan (left or right, and tilt up or down), zoom (in or out), fast pan or fixed.
15. The method of claim 14 wherein the rules include a pan rule, a zoom rule, a fast pan rule and a fixed rule.
16. The method of claim 15 wherein the pan rule includes extracting a plurality of frames to cover the space of environment while reducing the spatial overlap among the frames from a pan segment.
17. The method of claim 15 wherein the pan rule includes extracting a frame located at a point when the pan motion is slowed down.
18. The method of claim 15 wherein the zoom rule includes extracting a candidate frame at an endpoint of the zoom-in or zoom-out segment.
19. The method of claim 15 wherein the fixed rule includes extracting a candidate frame located at a midpoint of a fixed segment.