Analyzing camera captured video for key frames
View Patent ↗In order to analyze a digital video clip to determine candidate frames for subsequent key frame selection, a camera motion sensor is provided in the camera so that information is provided during image capture regarding camera motion including translation of the scene or camera, or scaling of the scene. The sensor includes an accelerometer or a lens motor sensor. A plurality of video segments is formed based on the global motion estimate and each segment is labeled in accordance with a predetermined series of camera motion classes. Thereafter, key frame candidates are extracted from the labeled segments and a confidence score is computed for each candidate by using rules corresponding to each camera motion class and a rule corresponding to object motion.
1. A method of analyzing a digital video clip captured by a camera to determine candidate frames for subsequent key frame selection, comprising:
a. providing a camera motion sensor, including an accelerometer or a lens motor sensor, in the camera so that camera motion information is provided during image capture regarding camera motion including translation of a scene or camera, or scaling of the scene;
b. forming a plurality of video segments based on the camera motion information and labeling each segment in accordance with a predetermined series of camera motion classes; and
c. extracting key frame candidates from the labeled segments and computing a confidence score for each candidate by using rules corresponding to each camera motion class and a rule corresponding to object motion.
2. The method of claim 1 wherein the predetermined camera motion classes include pan (left or right, and tilt up or down), zoom (in or out), fast pan or fixed.
3. The method of claim 2 wherein the rules include a pan rule, a zoom rule, a fast pan rule and a fixed rule.
4. The method of claim 3 wherein the pan rule includes extracting a plurality of frames to cover the space of environment while reducing the spatial overlap among the frames from a pan segment.
5. The method of claim 3 wherein the pan rule includes extracting a frame located at a point when the pan motion is slowed down.
6. The method of claim 3 wherein the zoom rule includes extracting a candidate frame at an endpoint of the zoom-in or zoom-out segment.
7. The method of claim 3 wherein the fast pan rule includes extracting candidate frames from a fast pan segment.
8. The method of claim 3 wherein the fixed rule includes extracting a candidate frame located at a midpoint of a fixed segment.
9. The method of claim 1 wherein the object motion rule includes extracting a candidate frame for a fixed segment with a confidence score related to the segment length, extracting a candidate frame for a pan segment with a confidence score related to a translation amount during the pan, and does not extract candidate frames for object motion based frames for fast pan and zoom segments.
10. A method of analyzing a digital video clip captured by a camera to determine key frames, comprising:
a. providing a camera motion sensor, including an accelerometer or a lens motor sensor, in the camera so that camera motion information is provided during image capture regarding camera motion including translation of a scene or camera, or scaling of the scene;
b. forming a plurality of video segments based on the camera motion information and labeling each segment in accordance with a predetermined series of camera motion classes; and
c. extracting key frame candidates from the labeled segments and computing a confidence score for each candidate by using rules corresponding to each camera motion class and a rule corresponding to object motion;
d. selecting key frames from the candidate frames based on the confidence score of each candidate.
11. The method of claim 10 further including ranking the selected key frames in accordance with the confidence score.
12. The method of claim 11 wherein step d includes employing the ranking and a user specified number to select the key frames.
13. The method of claim 12 wherein employing the ranking and a user specified number to select the key frames includes selecting at least one key frame from each segment if there are confidence scores above a pre-determined threshold.
14. The method of claim 13 wherein employing the ranking and a user specified number to select the key frames includes selecting key frames from the remaining candidates with the highest confidence values to fill the user specified number of key frames.
15. The method of claim 10 wherein the predetermined camera motion classes include pan (left or right and also tilt up or down), zoom (in or out), fast pan or fixed.
16. The method of claim 15 wherein the rules include a pan rule, a zoom rule, a fast pan rule and a fixed rule.
17. The method of claim 16 wherein the pan rule includes extracting a plurality of frames to cover the space of environment while reducing the spatial overlap among the frames from a pan segment.
18. The method of claim 16 wherein the pan rule includes extracting a frame located at a point when the pan motion is slowed down.
19. The method of claim 16 wherein the zoom rule includes extracting a candidate frame at an endpoint of the zoom-in or zoom-out segment.
20. The method of claim 16 wherein the fast pan rule includes extracting candidate frames from a fast pan segment.
21. The method of claim 16 wherein the fixed rule includes extracting a candidate frame located at a midpoint of a fixed segment.
22. The method of claim 10 wherein the object motion rule includes extracting a candidate frame for a fixed segment with a confidence score related to the segment length, extracting a candidate frame for a pan segment with a confidence score related to a translation amount during the pan, and does not extract candidate frames for object motion based frames for fast pan and zoom segments.