Method and apparatus for automatically summarizing video
View Patent ↗One embodiment of the present invention provides a system that automatically produces a summary of a video. During operation, the system partitions the video into scenes and then determines similarities between the scenes. Next, the system selects representative scenes from the video based on the determined similarities, and combines the selected scenes to produce the summary for the video.
1. A method for automatically producing a summary of a video, comprising:
partitioning a video into scenes using a frame-similarity matrix, each element in the frame-similarity matrix representing a distance between feature vectors of a corresponding pair of frames;
generating a scene-similarity matrix comprising a plurality of elements based on the frame-similarity matrix, each element of the scene-similarity matrix representing a measure of similarity between different scenes of the video;
determining, by a processor, an importance score for each scene based on the scene-similarity matrix, motion data associated with the scene, and audio data associated with the scene, an importance score for a scene indicating a relative importance of the scene, wherein the importance score increases responsive to the scene having a high similarity with other scenes in the video and increases responsive to the scene having a high distance from a measure of a mean cohesiveness of the video;
selecting representative scenes from the video based on the determined importance scores; and
combining selected scenes to produce the summary for the video.
2. The method of claim 1 , wherein each feature vector comprises a color histogram of a frame.
3. The method of claim 1 , wherein an element in the frame-similarity matrix comprises a distance between two feature vectors F A and F B , wherein the distance is calculated as
1
-
F
A
*
F
B
F
A
F
B
.
4. The method of claim 1 , wherein the scene-similarity matrix is generated by detecting shot boundaries using an adaptive-threshold technique which computes a distance between feature vectors for successive frames divided by a maximum distance between successive feature vectors in a preceding window of frames.
5. The method of claim 1 , wherein selecting a scene from the video comprises clustering similar scenes together and selecting at most one scene from each cluster.
6. The method of claim 1 , wherein selecting a representative scene from the video comprises using a dynamic-programming technique to select the scene.
7. The method of claim 1 , wherein selecting representative scenes from the video comprises selecting the representative scenes based on a total importance score for the selected scenes subject to a time constraint.
8. A non-transitory computer-readable storage medium storing instructions that, when executed by a computer, cause the computer to perform a method for automatically producing a summary of a video, the method comprising:
partitioning a video into scenes using a frame-similarity matrix, each element in the frame-similarity matrix representing a distance between feature vectors of a corresponding pair of frames;
generating a scene-similarity matrix comprising a plurality of elements based on the frame-similarity matrix, each element of the scene-similarity matrix representing a measure of similarity between different scenes of the video;
determining an importance score for each scene based on the scene-similarity matrix, motion data associated with the scene, and audio data associated with the scene, an importance score for a scene indicating a relative importance of the scene, wherein the importance score increases responsive to the scene having a high similarity with other scenes in the video and increases responsive to the scene having a high distance from a measure of a mean cohesiveness of the video;
selecting representative scenes from the video based on the determined importance scores; and
combining selected scenes to produce the summary for the video.
9. The computer-readable storage medium of claim 8 , wherein each feature vector comprises a color histogram of a frame.
10. The computer-readable storage medium of claim 8 , wherein an element in the frame-similarity matrix comprises a distance between two feature vectors F A and F B , wherein the distance is calculated as
1
-
F
A
*
F
B
F
A
F
B
.
11. The computer-readable storage medium of claim 8 , wherein the scene-similarity matrix is generated by detecting shot boundaries using an adaptive-threshold technique which computes a distance between feature vectors for successive frames divided by a maximum distance between successive feature vectors in a preceding window of frames.
12. The computer-readable storage medium of claim 8 , wherein selecting a scene from the video comprises clustering similar scenes together and selecting at most one scene from each cluster.
13. The computer-readable storage medium of claim 8 , wherein selecting a representative scene from the video comprises using a dynamic-programming technique to select the scene.
14. The computer-readable storage medium of claim 8 , wherein selecting representative scenes from the video comprises selecting the representative scenes based on a total importance score for the selected scenes subject to a time constraint.
15. An apparatus that automatically produces a summary of a video, comprising:
a non-transitory computer readable storage medium storing instructions executable to perform steps comprising:
partitioning a video into scenes using a frame-similarity matrix, each element in the frame-similarity matrix representing a distance between feature vectors of a corresponding pair of frames;
generating a scene-similarity matrix comprising a plurality of elements based on the frame-similarity matrix, each element of the scene-similarity matrix representing a measure of similarity between different scenes of the video;
determining an importance score for each scene based on the scene-similarity matrix, motion data associated with the scene, and audio data associated with the scene, an importance score for a scene indicating a relative importance of the scene, wherein the importance score increases responsive to the scene having a high similarity with other scenes in the video and increases responsive to the scene having a high distance from a measure of a mean cohesiveness of the video;
selecting representative scenes from the video based on the determined importance scores; and
combining selected scenes to produce the summary for the video; and
a processor configured to execute the instructions.
16. The apparatus of claim 15 , wherein each feature vector comprises a color histogram of a frame.
17. The apparatus of claim 15 , wherein an element in the frame-similarity matrix comprises a distance between two feature vectors F A and F B , wherein the distance is calculated as
1
-
F
A
*
F
B
F
A
F
B
.
18. The apparatus of claim 15 , wherein the scene-similarity matrix is generated by detecting shot boundaries using an adaptive-threshold technique which computes a distance between feature vectors for successive frames divided by a maximum distance between successive feature vectors in a preceding window of frames.
19. The apparatus of claim 15 , wherein selecting a scene from the video comprises clustering similar scenes together and selecting at most one scene from each cluster.
20. The apparatus of claim 15 , wherein selecting representative scenes from the video comprises selecting the representative scenes based on a total importance score for the selected scenes subject to a time constraint.
21. The method of claim 1 , wherein the distance from the measure of the mean cohesiveness of the video indicates how distinct the scene is from other scenes in the video.